Object Tracking and Action Sequence Recognition in Continuous Video for Social Activity Analysis

2.1 Problem formalization and overall framework

For the requirements of object tracking and action sequence recognition in continuous video images within social activity analysis scenarios, the problem is first formally defined. Let the continuous video image sequence be $\left\{I_1, I_2, \ldots, I_T\right\}$, where $I_t \in \mathrm{R}^{H \times W \times 3}, H$ and $W$ denote the height and width of the image respectively, and $T$ is the total number of frames in the video sequence; the output of the algorithm is $O_t=\left\{o_t^i\right\}$, where each target $o_t^i$ contains bounding box $\boldsymbol{b}_t^i$, identity identifier $i d_i$, and action label $a_t^i \in A$, with $A$ being the predefined set of action categories. To achieve precise collaboration between object tracking and action sequence recognition, this paper proposes an overall framework comprising four core modules, which are interrelated and work synergistically to break the limitation of serial independence between tracking and recognition modules in traditional algorithms, and construct a unified learning objective. Figure 1 shows the schematic diagram of the overall framework and collaborative optimization of the algorithm. The framework specifically includes an adaptive spatiotemporal feature extraction module, a graph contrastive multi-object tracking module, a hierarchical action sequence recognition module, and a collaborative optimization layer. Among them, the adaptive spatiotemporal feature extraction module is responsible for extracting target features with both spatiotemporal correlation and discriminability from continuous video frames, providing high-quality input for subsequent tracking and recognition tasks; the graph contrastive multi-object tracking module realizes stable association and identity maintenance of multiple targets in complex scenarios; the hierarchical action sequence recognition module completes dynamic recognition of target actions based on temporal features; the collaborative optimization layer acts as a core interaction unit, establishing a bidirectional constraint mechanism between the tracking and recognition modules, so that tracking results provide accurate target temporal information for action recognition, while action recognition results feed back into the association decision of the tracking module, realizing collaborative optimization and performance improvement of the two major tasks, and ensuring the robustness and accuracy of the algorithm in complex social activity scenarios.

Figure 1. Schematic diagram of the overall framework and collaborative optimization of the algorithm

2.2 Adaptive spatiotemporal feature extraction module

This paper adopts DCNv3 as the basic backbone network to complete basic feature modeling of video images, relying on the adaptive sampling property of deformable convolution to adapt to complex working conditions such as non-rigid deformation of targets, mutual occlusion among individuals, and diverse postures in social activity scenarios. The internal structure of the network is shown in Figure 2. The sampling calculation of deformable convolution is defined as:

$y(p)=\sum_{k=1}^K w_k \cdot x\left(p+p_k+\Delta p_k\right) \cdot \Delta m_k$    (1)

where, $p$ represents the standard sampling position, $w_k$ and $p_k$ are the fixed convolution weight and reference sampling offset respectively, the learnable position offset $\Delta p_k$ can dynamically adjust the distribution of the sampling grid according to the local shape of the target, and the modulation scalar $\Delta m_k$ adaptively weights the feature response of each sampling point. This structure can automatically focus on key regions such as deformed edges of targets and gaps in human interactions, weaken feature interference caused by occluded backgrounds, and effectively improve the discriminative stability of lowlevel features in complex scenarios.

On top of the basic features, a spatiotemporal pyramid attention structure is constructed to hierarchically enhance features from spatial and temporal dimensions. In the spatial dimension, a channel–spatial collaborative attention mechanism is introduced, using average pooling and maximum pooling to respectively mine the global mean distribution and extreme response information of the feature map; the results of the two pooling operations are concatenated along the channel dimension and passed through a convolution to model cross-channel correlations, and finally a spatial attention mask is generated via Sigmoid activation. The calculation process can be expressed as:

$M_s=\sigma\left(\operatorname{Conv}_{3 \times 3}\left(\left[\operatorname{AvgPool}\left(F_t\right) ; \operatorname{MaxPool}\left(F_t\right)\right]\right)\right)$    (2)

The generated attention mask can adaptively assign weights to each spatial position of the feature map, strengthen feature responses in the main region of the target, suppress irrelevant backgrounds and redundant noise, and realize refined selection and enhanced expression of spatial-dimensional features.

In the temporal dimension, optical flow-guided temporal attention modeling is introduced, representing the instantaneous motion trajectory of the target through the optical flow field between adjacent frames $V_{t \rightarrow t-1}$, and aligning the spatially enhanced feature $\widetilde{F}_{t-1}$ of the previous frame according to the optical flow motion vectors via deformable inter-frame alignment, achieving accurate cross-frame matching of temporal appearance features. By fusing the motion dynamics information carried by optical flow with the static appearance information extracted by convolution, spatiotemporal dependencies between consecutive frames are established, compensating for the deficiency that single spatial features cannot characterize target motion trends and action evolution patterns, enabling features to possess both spatial semantic integrity and temporal motion continuity.

After completing spatiotemporal attention enhancement, RoIAlign is used to aggregate features of individual target candidate regions, extracting separately the appearance feature representing target appearance attributes $f_t^{a p p}$ and the motion feature representing motion evolution patterns $f_t^{\text {mot}}$. The two types of features are directly concatenated along the channel dimension to construct the final spatiotemporal feature of the i-th target in the t-th frame:

$f_t^i=\left[f_t^{\text {app }} ; f_t^{\text {not }}\right] \in \mathrm{R}^D$    (3)

This feature integrates multiple information including static appearance, spatial structure, and temporal motion of the target, with regularized dimensions and sufficient discriminability, providing a stable and compatible feature input basis for subsequent graph contrastive multi-object tracking and hierarchical action sequence recognition tasks.

Figure 2. Internal structure diagram of the adaptive spatiotemporal feature extraction network

2.3 Graph contrastive multi-object tracking module

To model complex target association relationships between video frames, this paper constructs a spatiotemporal topological graph structure to accomplish multi-object tracking modeling. All detected target individuals in each frame of the continuous video are defined as the graph node set $V$, and all potential candidate matching relationships between adjacent frames are taken as edges to form the edge set E, thereby establishing the spatiotemporal association graph $g=(V, \mathrm{E})$. This modeling approach breaks through the limitation of traditional frame-by-frame independent matching, and is capable of simultaneously characterizing the spatial distribution association and temporal evolution association of targets, adapting to the complex characteristics of dense target arrangement, mutual occlusion, and close-range interaction in social activity scenarios, laying a topological structure foundation for subsequent global association cost calculation and optimal matching. The construction of the spatiotemporal topological graph and the mechanism diagram of graph contrastive learning are shown in Figure 3.

Figure 3. Construction of spatiotemporal topological graph and mechanism diagram of graph contrastive learning

Based on the spatiotemporal graph topology, a joint association cost function integrating appearance, motion, and behavioral temporal information is constructed to measure the matching similarity between nodes, expressed as:

$c_{i j}=\lambda_{\text {app }} \cdot d_{\cos }\left(f_t^i, f_{t+1}^i\right)+\lambda_{\text {mot }} \cdot\left\|p_t^i-p_{t+1}^j\right\|_2+\lambda_{\text {act }} \cdot l_{a c t}\left(a_t^i, \alpha_{t+1}^j\right)$    (4)

where the first term uses cosine distance to measure the appearance similarity of target features in adjacent frames, the second term uses the 2D coordinate Euclidean distance to represent target motion continuity, and the third term introduces the behavior consistency constraint loss $l_{\text {act}}$. Relying on the inherent temporal transition rules of social activity behaviors, a behavior transition probability matrix is constructed, and the rationality of transitions between different behavior categories is quantified according to the matrix, imposing additional penalty weights on low-probability abrupt behavior matching pairs. Through hyperparameters $\lambda_{\text {app}}, \lambda_{\text {mot}}$, and $\lambda_{\text {act}}$ balancing the contribution ratios of the three constraints, the behavioral temporal prior is naturally integrated into the tracking association process, reducing the identity switch probability in dense scenarios from the perspective of matching logic.

To further enhance the identity discrimination ability of graph node features, an inter-frame natural sample-driven graph contrastive learning training strategy is introduced, and the loss function is defined as:

$L_{\text {contrast }}=-\frac{1}{|P|} \sum_{(u, v) \in P} \log \frac{\exp \left(\operatorname{sim}\left(f_u, f_v\right) / \tau\right)}{\sum_{k \in N(u)} \exp \left(\operatorname{sim}\left(f_u, f_k\right) / \tau\right)}$     (5)

During training, graph nodes corresponding to the same target in adjacent frames are constructed as positive sample pairs, and nodes of different targets or nodes from frames with large temporal gaps are assigned to the negative sample set $N(u)$. The function sim(·) adopts cosine similarity to measure the node feature distance, and is the temperature coefficient used to scale the feature distribution interval. This training method constructs sample pairs relying on the temporal characteristics of the video itself, promoting the feature representations of the same identity target to cluster in the high-dimensional space, while enlarging the distribution gap between features of different identities, thereby improving the model’s ability to distinguish targets with similar appearances from the perspective of feature learning.

After obtaining the association cost values of all edges in the spatiotemporal graph, the Hungarian algorithm is adopted based on the cost matrix to solve the optimal bipartite matching, completing the global association assignment of target nodes between adjacent frames. With the optimization goal of minimizing the overall matching cost, the algorithm traverses all candidate matching combinations and outputs the uniquely corresponding association result, combined with the trajectory life cycle management mechanism, realizing new target trajectory initialization, existing trajectory continuous updating, and disappearance trajectory termination removal. This matching strategy can globally coordinate the association relationships of all targets between frames, avoiding the suboptimal solution problem caused by local greedy matching, and outputting continuous and stable target trajectory sequences, providing complete and reliable temporal target data support for subsequent hierarchical action sequence recognition.

2.4 Hierarchical action sequence recognition module

The hierarchical action sequence recognition module adopts a three-layer progressive structure, gradually mining target behavior features from local motion, temporal context to social interaction, to achieve accurate online recognition of behaviors in complex social activity scenarios. Figure 4 shows the three-layer progressive architecture diagram of hierarchical action sequence recognition. The bottom layer uses a temporal convolutional network to construct a local motion feature extractor, focusing on capturing target limb micro-actions. The output feature of the l-th layer is calculated as:

$h_t^{(l)}=\operatorname{ReLU}\left(\sum_{k=0}^{K-1} w_k^{(l)} \cdot h_{t-d \cdot k}^{(l-1)}+b^{(l)}\right)$   (6)

where, h^(l)_t denotes the output feature of the l-th layer at frame t, w^(l)_k and $b^{(l)}$ are the convolution kernel weights and bias term of this layer respectively, K is the convolution kernel size, and d is the dilation factor. The dilation factor adopts an exponential growth strategy, expanding progressively with the network depth, so that the convolution receptive field expands exponentially, effectively covering the complete period of human limb motion, accurately capturing local micro-action features such as waving and turning, and providing fine-grained motion foundations for subsequent temporal context modeling.

The middle layer introduces a causal self-attention mechanism to effectively aggregate temporal context features while ensuring the real-time requirement of online recognition. Its calculation expression is:

$\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}+M\right) V$   (7)

where, $Q, K$, and $V$ are the query, key, and value feature matrices respectively, $d_k$ is the feature dimension, and $M$ is the causal mask matrix. The mask matrix adopts a lower triangular structure design, with elements in the upper triangular region set to negative infinity, which approach 0 after softmax activation, thereby blocking the interference of future frame feature information on current frame recognition, ensuring that the model only utilizes current and historical frame temporal information for behavior judgment, meeting the practical demand for online real-time recognition in social activity analysis. The temporal context feature $\left\{h_t^{c t x}\right\}$ output by this mechanism can effectively integrate the temporal evolution pattern of target behavior, improving the coherence and accuracy of behavior recognition.

The top layer focuses on modeling target interaction relationships and refining behavior labels in social activity scenarios, adopting a collaborative design of interaction encoding and behavior unit aggregation. In the interaction encoding stage, a lightweight Transformer encoder is introduced to jointly encode the high-level context features of all targets in the same frame. The modeling process is:

$\tilde{h}_t^{\text {inter }}=$ TransformerEncoder $\left(\left[h_{t, 1}^{c t x}, h_{t, 2}^{c t x}, \ldots\right]\right)$   (8)

Through a simplified multi-head attention structure, this encoder efficiently captures spatial interactions and behavioral correlations among different targets at the same moment, accurately characterizing interaction patterns such as cooperation and following among individuals in group activities, adapting to the scenario characteristics of social activities. In the behavior unit aggregation stage, continuous temporal features are first divided into fixed-length behavior segments, segment-level behavior features are extracted through segment average pooling, mapped to the behavior category space via a fully connected layer, and then the boundary regression layer refines and calibrates frame-level behavior labels, correcting the boundary ambiguity problem in segment classification. This aggregation method can effectively integrate segment features and temporal context information, ensuring that the output frame-level behavior label $a_t^i$ has good temporal continuity, while providing reliable behavior constraint information for the graph contrastive multi-object tracking module, realizing the collaborative optimization of tracking and recognition.

Figure 4. Three-layer progressive architecture diagram of hierarchical action sequence recognition

2.5 Collaborative optimization loss for tracking and recognition

To achieve deep synergy between the object tracking and action sequence recognition tasks, a unified collaborative optimization loss function is constructed. Through bidirectional constraints of multiple loss components, the two tasks are promoted to support and improve each other collaboratively. The total loss function is defined as:

$L_{\text {total }}=L_{\text {track }}+\lambda_1 L_{\text {action }}+\lambda_2 L_{\text {consistency }}$    (9)

where, $L_{\text {track}}$ is used to optimize multi-object tracking performance, $L_{\text {action}}$ ensures the accuracy and continuity of action sequence recognition, and $L_{\text {consistency}}$ serves as the core constraint to realize feature space and prediction distribution alignment between the tracking and recognition modules. The hyperparameters $\lambda_1$ and $\lambda_2$ are used to balance the contribution weights of each loss component, ensuring training stability and task synergy.

The tracking loss $L_{\text {track}}$ is composed of binary cross-entropy matching loss and ID classification loss. The binary crossentropy loss optimizes the positive-negative sample discrimination accuracy of inter-frame target matching, while the ID classification loss enhances the discriminative ability of target identity features through the classification task. Their combination provides loss constraints for stable association of target trajectories. The action recognition loss $L_{\text {action}}$ adopts a combined form of cross-entropy loss and temporal smoothing regularization term $L_{\text {smooth}}$. The cross-entropy loss optimizes the classification accuracy of frame-level action labels, and the temporal smoothing regularization term penalizes unreasonable action mutations by constraining the differences in action labels between adjacent frames, ensuring the temporal coherence of the output action sequence. The consistency loss $L_{\text {consistency}}$ achieves deep synergy between tracking and recognition modules through dual constraints, and its calculation expression is:

$\frac{1}{|S|} \sum_{(t, i) \in S} \begin{gathered}L_{\text {consistency }}= \left\|f_t^i-g_t^i\right\|_2^2+\beta \cdot K L\left(p\left(a \mid f_t^i\right) / / p\left(a \mid g_t^i\right)\right)\end{gathered}$     (10)

where, $S$ is the training sample set, $f_t^i$ is the target feature extracted by the action recognition module. The first term constrains the distribution consistency of the two types of features in the high-dimensional space through Euclidean distance, realizing feature space alignment, and prompting tracking features to carry behavioral semantic information and behavioral features to contain identity discriminative information. The second term measures the difference in action category prediction distributions output by the two modules through KL divergence, forcing their behavior judgments to converge, thereby achieving complementary enhancement between tasks. is the weight coefficient of the KL divergence term, used to balance the constraint strength of feature alignment and prediction distribution alignment.

The settings of hyperparameters $\lambda_1, \lambda_2$ , and β follow the principles of task priority and training stability, and their optimal ranges are determined through multiple rounds of validation experiments: $\lambda_1$ is set within 0.8–1.2 to ensure a balance between action recognition loss and tracking loss; $\lambda_2$ is set within 0.3–0.5 so that the consistency loss can achieve synergy between the two modules without dominating the overall training process; β is set within 0.1–0.2 to avoid feature homogenization caused by excessive constraints from the KL divergence term. This loss balancing strategy effectively avoids the problem of a single-task loss dominating the training process, ensuring synchronous improvements in tracking accuracy, action recognition accuracy, and temporal coherence, and realizing deep synergy and performance optimization of the two tasks.

2.6 Training and inference pipeline

The model is trained using an end-to-end alternating optimization strategy to ensure collaborative convergence of all modules and improve overall performance. All training processes are completed in a standardized experimental environment to guarantee reproducibility. The training pipeline is divided into three stages: first, the adaptive spatiotemporal feature extraction module and the graph contrastive multi-object tracking module are jointly pre-trained, with the optimization goal of target identity association accuracy, initializing parameters related to feature extraction and tracking; next, the hierarchical action sequence recognition module is pre-trained independently, focusing on action classification accuracy and temporal coherence to complete the initialization of action recognition-related parameters; finally, all modules are integrated and jointly fine-tuned end-to-end based on the collaborative optimization loss function to achieve deep synergy between tracking and recognition tasks. During training, the input video frames are uniformly resized to a fixed resolution, the AdamW optimizer is adopted, the initial learning rate is set to 1e-4, the batch size is fixed according to hardware resources, and a cosine annealing learning rate scheduling strategy is employed, with the learning rate decaying exponentially after each iteration. An early stopping strategy is also introduced, using comprehensive performance on the validation set as the evaluation metric, and training is terminated if performance does not improve over five consecutive rounds. Data augmentation strategies include random cropping, horizontal flipping, color jittering, and Gaussian noise addition, effectively improving model generalization and avoiding overfitting.

In the inference stage, an online real-time processing pipeline is adopted, balancing processing speed and task accuracy to meet the engineering application requirements of social activity analysis. The inference pipeline proceeds sequentially as target detection, feature extraction, graph association matching, and action sequence recognition: first, target detection is performed on the input video frame to obtain candidate target regions; then, the adaptive spatiotemporal feature extraction module extracts spatiotemporal fused features for each target; based on the graph contrastive multi-object tracking module, inter-frame target association matching is completed, and identity stability is maintained through behavior consistency constraints; finally, the hierarchical action sequence recognition module performs online behavior judgment on target trajectories and outputs frame-level behavior labels. The tracking and recognition modules adopt a soft coupling approach, where action recognition results feed back into tracking association decisions through the consistency constraint term, and tracking trajectories provide complete temporal context for action recognition, realizing dynamic synergy between the two tasks. Experimental tests show that the algorithm achieves a real-time processing speed of up to 25 FPS on the NVIDIA RTX 4090 GPU hardware platform, meeting the online analysis demands of complex social activity scenarios, balancing accuracy and real-time performance, and possessing strong engineering application value.

The experimental results fully verify the effectiveness of the proposed algorithm in social activity video analysis tasks. The observed performance improvements stem from the organic integration of innovative modules and task collaboration mechanisms. The adaptive spatiotemporal feature extraction module enhances the robustness of target features under occlusion and scale variations, providing a high-quality representation foundation for subsequent tracking and recognition. The graph contrastive loss reduces identity switches significantly by pulling features of the same target closer and pushing those of different targets apart, improving long-term identity preservation in tracking. Behavior consistency cost and consistency loss achieve deep coupling between tracking and recognition tasks, enforcing feature space alignment and prediction distribution constraints, enabling mutual supervision and complementary enhancement between the two tasks, and fundamentally resolving the performance bottleneck caused by independent optimization of single tasks. Compared with existing methods, the proposed algorithm demonstrates clear advantages in social activity scenarios characterized by dense targets, frequent interactions, and severe occlusion, balancing tracking accuracy, action recognition accuracy, and inference real-time performance, which aligns with the practical demands of real-world social activity analysis.

However, certain limitations remain. In extremely complex scenarios involving extreme occlusion and highly overlapping targets, the completeness of certain target features is difficult to guarantee, leading to slight declines in tracking continuity and action recognition accuracy. For distant small targets occupying limited image areas, the ability to extract and discriminate behavioral features still requires improvement. Moreover, when handling very large-scale target crowds, the computational complexity of the interaction encoding module increases slightly, posing certain limitations for adaptation to extremely real-time scenarios. These limitations reflect the inherent challenges in modeling complex social activity video analysis tasks and highlight opportunities for optimization in feature representation, model lightweighting, and fine-grained behavior understanding.

Future research will focus on addressing these limitations and aligning with domain development trends. Firstly, integrating large-scale Vision Transformers with dynamic feature modeling mechanisms can strengthen feature extraction and association capabilities under extreme occlusion and small-target conditions. Secondly, designing lightweight interaction encoding structures and dynamic inference architectures can further reduce computational overhead while maintaining performance, improving feasibility for edge deployment. Additionally, incorporating spatiotemporal multimodal information and fine-grained behavior annotations can enhance the representation and recognition of complex social interaction behaviors, while extending cross-dataset generalization performance to promote practical deployment in intelligent surveillance, public safety, and behavior analysis applications, ultimately delivering more efficient and robust technical solutions for social activity video understanding tasks.