A Deep Learning-Based Real-Time Action Recognition and Analysis System for Martial Arts Athletes

2.1 Overall system framework

An end-to-end real-time martial arts action recognition and analysis system is developed, targeting the inherent characteristics of martial arts movements—namely, dynamically variable joint coordination, pronounced hierarchical temporal rhythms, and large instantaneous motion amplitudes—to overcome the homogenized processing limitations of fixed spatiotemporal modeling in conventional graph convolutional models. A hierarchical inference architecture is designed, incorporating spatially dynamic adaptation, temporally semantic stratification, and modality-adaptive enhancement. Continuous Red-Green-Blue (RGB) video is employed as the raw input, from which human joint coordinate information is extracted via a lightweight pose estimation network. Through temporal noise reduction and multi-dimensional motion feature reconstruction strategies, standardized skeletal spatiotemporal graph data are constructed, encompassing spatial positions, motion velocities, and skeletal geometric attributes, thereby providing a fundamental input for the fine-grained characterization of complex martial arts motion patterns. The entire architecture operates according to the progressive logic of spatial topology optimization, temporal feature mining, multimodal robustness enhancement, and lightweight inference output. Each functional module is deeply adapted to the specific motion mechanisms of martial arts, while simultaneously accommodating both feature representation precision and the real-time requirements of edge-device deployment.

To quantitatively define the complete inference pipeline of the system, a continuous input video sequence of length T is considered, and a multi-level feature mapping model from raw visual data to action recognition results is established. The overall computational process is formulated as:

$\hat{y}=F_{\text {cls}}\left(F_{\text {fuse}}\left(F_{\text {MSTM}}\left(F_{\text {DAG}}\left(F_{\text {pose}}\left(I_{1: T}\right)\right)\right)\right)\right)$        (1)

where, $I_{1: T}$ denotes the raw input video sequence comprising $T$ frames; $F_{\text {pose}}$ is designated as the pose preprocessing function, responsible for skeletal keypoint detection, temporal filtering, and high-dimensional feature construction; $F_{D A G}$ is defined as the deformable adaptive graph convolution operation, which serves to dynamically rectify joint topological structures and neighborhood receptive fields, thereby fitting the transiently varying joint coordination relationships inherent in martial arts movements; $F_{M S T M}$ is specified as the multi-scale temporal modeling operation, which performs action temporal phase segmentation and differentiated scale feature extraction to mine the phased rhythmic semantics of martial arts actions; $F_{\text {fuse}}$ is designated as the cross-modal fusion operation, which adaptively fuses RGB appearance features with skeletal structural features according to scene complexity, thereby enhancing the model's anti-interference capability; and $F_{\text {cls}}$ is defined as the lightweight classification operation, which maps highdimensional spatiotemporal features to the action category space and outputs the final recognition result. This modular architecture enables a decoupled design between feature modeling and engineering optimization, allowing flexible switching between unimodal and multimodal inference modes, and effectively balancing recognition accuracy and inference latency.

2.2 Skeletal sequence extraction via lightweight pose estimation

Video frame-level human joint perception is accomplished through a lightweight pose estimation network. To address the issues of keypoint jitter and temporal offset caused by the high motion speed and large limb deformation amplitudes characteristic of martial arts movements, skeletal data preprocessing and denoising optimization are performed. For an input video sequence of length T, two-dimensional coordinate information for 17 core human joints is extracted frame by frame, and a frame-level static skeletal topology is constructed. The skeletal graph corresponding to the t-th frame is defined as $G_t=(V, E)$, where V denotes the set of human joint nodes, and E denotes the fixed topological edge relationships established according to anatomical connection rules, serving to constrain the basic human limb structural associations. The raw detected joint coordinates exhibit significant temporal noise that disrupts the spatiotemporal continuity of continuous martial arts motion. A Kalman filtering algorithm is therefore employed to perform temporal smoothing correction of joint trajectories, suppressing high-frequency jitter through an iterative mechanism of state prediction and observation correction. The specific update equations are formulated as:

$\hat{x}_{t \mid t}=\hat{x}_{t \mid t-1}+K_t\left(z_t-H \hat{x}_{t \mid t-1}\right)$                  (2)

$P_{t \mid t}=\left(I-K_t H\right) P_{t \mid t-1}$                     (3)

where, $\hat{x}_{t \mid t-1}$ denotes the prior predicted state of the joint coordinates, $\hat{x}_{t \mid t}$ denotes the filtered posterior state, $z_t$ denotes the raw observed coordinates from pose estimation, $H$ denotes the fixed observation matrix, $K_t$ denotes the adaptive Kalman gain, and $P$ denotes the state covariance matrix, which is used to dynamically correct state estimation errors, thereby effectively ensuring the temporal stability and trajectory smoothness of skeletal keypoints under high-speed martial arts motion.

To precisely characterize the complex limb coordination and deformation properties of martial arts movements, high-dimensional node representations are constructed from the smoothed joint coordinate sequences through the integration of kinetic features and skeletal geometric features, thereby overcoming the limited motion representation capacity of raw coordinate features alone. Joint motion velocities and accelerations are respectively computed from consecutive frame joint position differences to quantify the instantaneous force generation and dynamic limb variation characteristics of martial arts movements. The computations are formulated as:

$\Delta v_i^t=v_i^t-v_i^{t-1}$             (4)

$\Delta^2 v_i^t=v_i^{t+1}-2 v_i^t+v_i^{t-1}$                  (5)

where, $\Delta v_i^t$ and $\Delta^2 v_i^t$ denote the motion velocity and acceleration, respectively, of the $i$-th joint at the $t$-th frame. Concurrently, skeletal geometric representations are constructed by defining the bone vector between adjacent joints as $b_{i j}^t=v_i^t-v_j^t$, from which bone Euclidean lengths and normalized directional features are derived through vector computation, thereby providing a complete description of limb posture deformation patterns. The raw joint coordinates, detection confidences, motion velocities, motion accelerations, bone lengths, and bone directions are integrated to form 11-dimensional node feature vectors. A standardized skeletal spatiotemporal tensor of dimension $\mathrm{R}^{T \times N \times C}$ is ultimately constructed, providing high-precision and high-robustness foundational data for subsequent dynamic graph convolution modeling and temporal feature mining.

2.3 Deformable adaptive graph convolution module

Conventional spatial graph convolution relies on fixed human anatomical topology to accomplish node information aggregation, with feature iteration across network layers depending on constant adjacency relationships for message passing, rendering it inadequate for adapting to the unique instantaneous joint coordination variation characteristics of martial arts movements. The feature update formulation of standard graph convolution can be uniformly expressed as:

$H^{(l+1)}=\sigma\left(\widetilde{D}^{-1 / 2} \widetilde{A} \widetilde{D}^{-1 / 2} H^{(l)} W^{(l)}\right)$            (6)

where, $H^{(l)}$ denotes the input node features at the $l$-th network layer, $W^{(l)}$ denotes the learnable weight matrix between layers, $\widetilde{A}$ denotes the fixed anatomical adjacency matrix incorporating selfloop structures, $\widetilde{D}$ denotes the corresponding degree matrix, and $\sigma$ denotes the nonlinear activation function. The fixed topological structure is capable only of characterizing static physiological connectivity relationships of the human body, failing to accommodate the irregular joint coordination patterns under dynamic martial arts movements such as leaping, twisting, and explosive force exertion. To address this problem, a deformable adaptive graph convolution module is designed, integrating anatomical prior structures, data-driven association learning, and dynamic receptive field sampling mechanisms to enable adaptive evolutionary modeling of skeletal spatial topology, thereby accommodating the complex and variable limb motion patterns of martial arts. Figure 1 illustrates the network architecture of the proposed deformable adaptive graph convolution module.

1.png

Figure 1. Network architecture of the deformable adaptive graph convolution module

To enhance the representational discriminability of joint motion features and provide high-precision feature support for subsequent dynamic topology learning, high-dimensional space projection transformation is performed on the raw node features. Low-dimensional base features are migrated to a high-dimensional latent space through linear mapping, fully exploiting the motion discrepancy information across different joints. The feature projection process is defined as:

$\widetilde{H}^{(l)}=H^{(l)} W_p+b_p$                 (7)

where, $W_p$ denotes the dimensional transformation weight matrix and $b_p$ denotes the bias parameter. The latent space dimension is uniformly set to twice the dimension of the original features, thereby expanding the feature representation space without introducing redundant computation, effectively enhancing the model's capacity to capture subtle limb deformations and coordination discrepancies in martial arts movements, and providing a robust feature foundation for subsequent joint similarity computation and offset prediction.

Based on the high-dimensional projected features, an attention mechanism is introduced to accomplish adaptive association modeling among joint nodes, freeing the model from the constraints of fixed anatomical connections and enabling action-specific joint coordination relationships to be learned autonomously from motion data. The similarity weight between joints is computed as:

$A_{\text {adapt}}^{(i, j)}=\operatorname{softmax}_j\left(\frac{\left(\widetilde{H}_i^{(l)} W_q\right)\left(\widetilde{H}_j^{(l)} W_k\right)^{\top}}{\sqrt{d_h}}\right)$              (8)

where, $W_q$ and $W_k$ denote the projection matrices for the query and key vectors, respectively, and $d_h$ denotes the dimension of the high-dimensional latent space. To balance modeling accuracy and computational complexity, a sparse constraint is imposed on the adaptive similarity matrix, retaining the eight most similar neighboring nodes for each node while uniformly zeroing out all other association weights. This sparse sampling strategy effectively filters out interference from irrelevant joint associations, enabling precise focus on the genuinely coupled limb nodes during martial arts motion execution, and thereby constructs a dynamic association topology that conforms to the specific motion patterns of the sport.

To accommodate the large-amplitude and instantaneous limb motion characteristics of martial arts, a deformable offset prediction mechanism is introduced into the module, enabling dynamic adaptive adjustment of the graph convolution receptive field. A lightweight multilayer perceptron is employed to predict two-dimensional offset parameters based on the real-time motion features of each joint, thereby accomplishing dynamic correction of the neighborhood sampling positions. The offset computation process is formulated as:

$\Delta_i^{(l)}=M L P_{\text {offset}}\left(\widetilde{H}_i^{(l)}\right)$               (9)

Based on the predicted offsets, bilinear interpolation sampling is performed on the adaptive topology matrix to obtain the deformable topology matrix. To integrate static structural priors with dynamic motion features, adaptive fusion of the dual topologies is achieved through learnable weights:

$A_{\text {final}}^{(l)}=\alpha^{(l)} A_{\text {anat}}+\beta^{(l)} A_{\text {deform}}^{(l)}, \alpha^{(l)}+\beta^{(l)}=1$           (10)

The fusion weights are adaptively generated by a learnable network based on global features, enabling the shallower layers of the network to prioritize anatomical topology for ensuring structural rationality, while the deeper layers emphasize dynamic topology for mining sport-specific motion features, thereby achieving differentiated topological modeling across different network layers.

Leveraging the fused dynamic topology matrix, the module accomplishes the final graph convolution feature aggregation and iterative update. Cross-node information interaction is realized after normalization of the topology matrix, with the complete convolution operation defined as:

$H^{(l+1)}=\sigma\left(\widetilde{D}_{\text {final}}^{-1 / 2} \widetilde{A}_{\text {final}}^{(l)} \widetilde{D}_{\text {final}}^{-1 / 2} H^{(l)} W^{(l)}\right)$              (11)

where, A^~(l)_final denotes the fused adjacency matrix with self-loops incorporated, and $\widetilde{D}_{\text {final}}$ denotes the corresponding normalized degree matrix. To preserve shallow-layer effective features and mitigate the gradient attenuation problem in deeper networks, a residual connection is introduced to fuse the convolution output features with the original input features. This modeling approach enables dynamic adjustment of the information propagation range and weight allocation among joints according to the force-generation rhythm, motion amplitude, and limb coordination patterns of martial arts movements, completely breaking through the representational limitations of fixed topology in traditional graph convolution and achieving fine-grained modeling of complex spatial motion features in martial arts.

2.4 Martial-arts-semantics-driven multi-scale temporal modeling and phase perception module

Spatial graph convolution accomplishes dynamic association modeling in the spatial dimension of skeletal data, effectively characterizing the instantaneous coordination patterns of martial arts limbs, while temporal modeling is responsible for characterizing the rhythmic regularities and semantic hierarchies of complete actions as they evolve over time. Conventional temporal modeling methods generally employ fixed-receptive-field convolutional kernels to uniformly process temporal sequences, rendering them incapable of adapting to the inherent phased motion characteristics of martial arts actions. Martial arts routine actions exhibit clear state divisions, with significant differences in limb motion speed, deformation range, and force-generation mechanisms across distinct motion phases. The uniform temporal feature extraction approach fails to differentiate the semantic contributions of each phase, resulting in insufficient representational capacity of the model for action boundaries and critical force-generation features. A martial-arts-semantics-driven multi-scale temporal modeling and phase perception module is constructed, achieving hierarchical fine-grained modeling of temporal features through data-driven automatic phase division, scale-adaptive temporal feature extraction, and adaptive phase weight allocation, thereby precisely matching the temporal evolution patterns specific to martial arts movements. Figure 2 illustrates the flowchart of the proposed multi-scale temporal modeling and phase perception module.

2.png

Figure 2. Flowchart of the multi-scale temporal modeling and phase perception module

The module first constructs frame-level motion representations to accomplish unsupervised automatic action phase segmentation, thereby overcoming the limitations of manually defined temporal interval division. Based on the skeletal temporal feature matrix of dimension $T \times N \times C$, the single-frame motion amplitude feature and joint coordination feature are respectively computed to quantify the temporal motion states. The computation formulas are as follows:

$m_t=\left\|X_t-X_{t-1}\right\|_2$       (12)

$c_t=\frac{1}{N} \sum_{i=1}^N\left\|v_i^t-\bar{v}^t\right\|_2, \bar{v}^t=\frac{1}{N} \sum_{i=1}^N v_i^t$         (13)

where, $m_t$ denotes the overall motion variation amplitude at the $t$-th frame, used to characterize instantaneous motion intensity; $X_t$ and $X_{t-1}$ denote the skeletal features of two consecutive frames, respectively; $c_t$ denotes the intra-frame joint motion coordination degree, used to characterize limb coordination consistency; $N$ denotes the total number of human joints; $v_i^t$ denotes the motion vector of the $i$-th joint at the $t$-th frame; and $\bar{v}^t$ denotes the average motion vector of all joints within the frame. The two types of features are fused to construct a two-dimensional temporal descriptor, which is fed into a lightweight temporal clustering model. With the minimization of interval feature residual as the optimization objective, the complete action sequence is partitioned into three stable semantic phases, corresponding respectively to the preparation, force-generation, and completion processes of martial arts actions. The ordered temporal segmentation boundaries are output, providing a foundational basis for differentiated temporal modeling.

Based on the temporal phase segmentation, multi-scale differentiated feature extraction is performed on distinct motion phases using adaptive dilation rate temporal convolution, accommodating the motion rhythm variations across phases. The convolution dilation rate is dynamically determined according to the motion intensity within each interval, enabling adaptive adjustment of the receptive field. The dilation rate computation is formulated as:

$r_s=\left|\frac{\underset{t}{\operatorname{max}} m_t}{a v g_{t \in\left[b_{s-1}, b_s\right]} m_t}\right|$          (14)  

where, $r_s$ denotes the convolution dilation rate for the $s$-th action phase; $\underset{t}{\operatorname{max}} m_t$ denotes the maximum motion amplitude across the entire sequence; $a v g_{t \in\left[b_{s-1}, b_s\right]} m_t$ denotes the average motion amplitude within the current phase; and $b_{s-1}$ and $b_s$ denote the start and end segmentation boundaries of the current temporal phase. Based on the adaptive dilation rate, the temporal feature extraction process for each phase is defined as:

$F_s(\tau)=\sum_{p=0}^{k_s-1} W_s(p) \cdot X_{b_{s-1}+\tau-r_s \cdot p}$            (15)  

where, $k_s$ denotes the phase-specific convolution kernel size, $W_s(p)$ denotes the learnable convolution weight parameters, and $\tau$ denotes the temporal position index. A small dilation rate convolution is configured for the high-dynamic force-generation phase to capture fine-grained instantaneous motion features, while a large dilation rate convolution is configured for the static transition and completion phases to mine long-range global motion trends. Simultaneously, a global temporal convolution branch with fixed parameters is established to extract unified contextual features from the complete sequence, achieving complementary fusion of local phase-specific features and global temporal features.

To further enhance the dominant role of features from critical motion phases, a phase-aware attention mechanism is introduced into the module to adaptively quantify the semantic contributions of distinct temporal phases. The computation process for attention scores and normalized weights is formulated as:

$e_s=M L P_{a t t}\left(F_s\right)$         (16)

$w_s=\frac{\exp \left(e_s\right)}{\sum_{k=1}^K \exp \left(e_k\right)}$          (17)

where, $e_s$ denotes the raw attention score for the $s$-th phase feature, obtained through mapping by a lightweight fully connected network; $w_s$ denotes the normalized phase-wise attention weight; and $K$ denotes the total number of action phases. The final temporal features are obtained through weighted fusion and feature concatenation as:

$F_{\text {temp}}=$ Concat $\left(\sum_{s=1}^K w_s \cdot F_s, F_{\text {global}}\right)$          (18)

where, $F_{\text {temp}}$ denotes the final temporal representation output by the module, and $F_{\text {global}}$ denotes the global temporal convolution features. This architecture automatically elevates the feature weights of critical phases according to the force-generation characteristics of martial arts movements, while attenuating feature interference from ineffective transition phases, thereby effectively compensating for the deficiencies of fixed temporal modeling methods. Subsequent ablation experiments further validate the incremental gains achieved by phase segmentation, multi-scale convolution, and the attention mechanism.

2.5 Lightweight deployment strategy

To enable real-time deployment of the martial arts action recognition model on edge terminals, a hierarchical lightweight optimization system is constructed from three dimensions: spatial structure optimization, temporal data reduction, and model knowledge transfer. The aforementioned dynamic graph convolution and temporal perception modules enable high-precision specialized action modeling; however, the complex spatiotemporal interactive computations incur substantial parameter counts and computational overhead, rendering them difficult to satisfy the low-latency inference requirements of mobile devices. Through the collaborative application of multi-level lightweight strategies, the computational load of the model is compressed, significantly reducing inference cost while fully preserving the semantic information of martial arts force-generation characteristics, temporal rhythms, and joint coordination, thereby achieving an optimal trade-off between recognition accuracy and computational efficiency.

A grouped graph convolution mechanism is introduced to optimize the computational complexity of spatial modeling, partitioning all joints into independent subgroups according to human motor functional characteristics, thereby circumventing redundant computations arising from global node-wise interactions. The 17 human joints are divided into three functional units—upper limbs, lower limbs, and torso—with each joint subgroup independently performing graph convolution feature extraction, and the output features from all groups are finally concatenated to accomplish spatial representation fusion. The computation process is defined as:

$H_{\text {group}}^{(l+1)}=\operatorname{Concat}\binom{G \operatorname{Conv}_1\left(H_1^{(l)}\right), G \operatorname{Conv}_2\left(H_2^{(l)}\right),}{\ldots, G \operatorname{Conv}_G\left(H_G^{(l)}\right)}$         (19)

where, G denotes the total number of joint groups, GConv$_g$ denotes the independent graph convolution operation corresponding to the g-th group, and H^(l)_g denotes the node features of the g-th group of joints at the l-th network layer. To circumvent the issue of inter-group feature isolation caused by independent group convolutions, a group shuffling mechanism is introduced to accomplish cross-group information exchange:

$H_{{shuffle }}^{(l+1)}=$Shuffle$\left(H_{{group}}^{(l+1)}\right)$         (20)

This architecture reduces the computational complexity of global graph convolution from $O\left(N^2 d\right)$ to $O\left(\sum^G{ }_{g=1} N^2{ }_g d\right)$, where $N_g$ denotes the number of joints in a single group and d denotes the feature dimension, thereby compressing the overall spatial computational cost to approximately one-third of the original while ensuring the learnability of cross-limb coordination features.

To address the non-uniform temporal motion characteristics of martial arts movements, a motion-amplitude-adaptive temporal sparse sampling strategy is designed, replacing the fixed-interval sampling approach and effectively eliminating redundant frames from quiescent phases. Based on the frame-level motion amplitude features, the global motion mean and median values across the entire sequence are statistically computed, and the temporal sampling stride is adaptively determined. The specific computation is formulated as:

$\delta=\max \left(1,\left\lfloor\frac{M}{\operatorname{median}\left(m_t\right)}\right\rfloor\right)$        (21)

where, $\delta$ denotes the adaptive sampling stride, $M$ denotes the average motion amplitude of the complete video sequence, and $\operatorname{median}\left(m_t\right)$ denotes the median value of motion amplitudes across all frames. Dense sampling with unit stride is employed for force-generation phases with motion amplitudes above the median value, preserving fine-grained temporal features of instantaneous explosive actions, while sparse sampling with large strides is applied to low-motion-amplitude transition and completion phases. The effective sequence length after sampling is statistically obtained through a binary indicator function. This strategy compresses the temporal data volume to below 70% of the original without losing critical action semantics, substantially reducing the computational overhead of temporal modeling.

To compensate for the feature representational degradation caused by structural compression and sparse sampling, a multi-constraint knowledge distillation strategy is introduced to transfer the specialized motion modeling knowledge of the high-precision large model to the lightweight student network. Figure 3 illustrates the model lightweighting and knowledge distillation framework for edge deployment. The complete network with 15 M parameters is adopted as the teacher model, and the lightweight compressed network is adopted as the student model. A composite distillation loss function is constructed, integrating hard-label classification loss, feature matching loss, and soft-label probability loss:

3.png

Figure 3. Model lightweighting and knowledge distillation framework for edge deployment

$L_{K D}=L_{C E}\left(y_S, y_{\text {true}}\right)+\lambda_1 L_{M S E}\left(f_S(X), f_T(X)\right)+\lambda_2 L_{K L}\left(p_S^\tau, p_T^\tau\right)$                (22)

where, $L_{C E}$ denotes the cross-entropy loss, which enforces hard supervision learning from groundtruth labels; $L_{\text {MSE}}$ denotes the mean squared error loss, which constrains the alignment of deep feature distributions between the student and teacher networks; $L_{K L}$ denotes the Kullback-Leibler divergence loss, which achieves optimization of the predicted probability distribution fitting; $f_S$ and $f_T$ denote the feature mapping functions of the student and teacher networks, respectively; and $p_S^\tau$ and $p_T^\tau$ denote the predicted probabilities normalized with the temperature coefficient, where the temperature parameter $\tau$ is used to smooth the probability distribution and mine fine-grained category differences. The weight coefficients $\lambda_1$ and $\lambda_2$ respectively regulate the loss contributions of feature distillation and probability distillation. Through multi-dimensional knowledge constraints, the lightweight model is enabled to acquire the teacher network's fine-grained modeling capabilities for martial arts joint coordination and temporal force generation, effectively addressing the accuracy degradation problem inherent in lightweight models.

2.6 Red-green-blue-skeletal cross-attention fusion module

Single-modality skeletal representations and RGB visual representations each possess inherent representational deficiencies in martial arts action recognition tasks. Skeletal temporal features stably characterize limb topological coordination patterns and are insensitive to illumination and viewpoint variations, yet fail to capture subtle limb morphology and texture differences in martial arts movements. RGB visual features preserve rich appearance details, but are highly susceptible to scene background interference, limb occlusion, and other factors, making precise focus on the moving subject difficult. To achieve deep complementarity between the two modalities, a bidirectional RGB-skeletal cross-attention fusion module is constructed, establishing a dual-branch adaptive information interaction mechanism to accomplish synergistic enhancement of visual appearance features and human structural features. Lightweight networks and the aforementioned spatiotemporal modeling networks are respectively employed for dual-branch feature extraction. Temporal visual features of dimension $T \times d_{R G B}$, denoted as $F_{R G B}$, are extracted frame by frame from video appearance information using MobileNetV3. Skeletal motion features of dimension $T \times d_{\text {skel}}$, denoted as $F_{\text {skeleton}}$, are output through the deformable adaptive graph convolution and temporal modeling network. Figure 4 illustrates the network architecture of the RGB-skeletal bidirectional cross-attention multimodal fusion module. Cross-modal guided enhancement is achieved through bidirectional cross-attention, with the specific interaction computations formulated as:

$\begin{gathered}F_{R G B \leftarrow \text {skeleton}}=\operatorname{Softmax}\left(\frac{F_{R G B} W_Q\left(F_{\text {skeleton}} W_K\right)^{\top}}{\sqrt{d}}\right) F_{\text {skeleton}} W_V\end{gathered}$            (23)

$\begin{gathered}F_{\text {skeleton} \leftarrow R G B}= \operatorname{Softmax}\left(\frac{F_{\text {skeleton}} W_Q^{\prime}\left(F_{R G B} W_K^{\prime}\right)^{\top}}{\sqrt{d}}\right) F_{R G B} W_V^{\prime}\end{gathered}$             (24)

where, $W_Q$, $W_K$, and $W_V$ denote the query, key, and value projection matrices for the visionmodality-guided branch, respectively; $W_Q{ }^{\prime}$, $W_K{ }^{\prime}$, and $W_V^{\prime}$ denote the corresponding projection matrices for the skeleton-modality-guided branch; and d denotes the unified feature mapping dimension. This interaction mechanism constrains the effective perceptual region of visual features using skeletal topological structures, suppressing noise interference from complex backgrounds, while simultaneously compensating for the detail deficiency in skeletal keypoint representations using RGB fine-grained appearance information, achieving bidirectional gain for both modalities.

4.png

Figure 4. Network architecture of the Red-Green-Blue (RGB)-skeletal bidirectional cross-attention multimodal fusion module

Upon completion of the cross-modal attention interaction, the enhanced features from both branches are integrated through residual fusion and dimensional concatenation strategies to construct a highly robust comprehensive action representation. The fusion computation is formulated as:

$\begin{gathered}F_{\text {fuse}}= \operatorname{Concat}\left(F_{R G B}+F_{R G B \leftarrow \text { skeleton}}, F_{\text {skeleton}}+F_{\text {skeleton} \leftarrow R G B}\right)\end{gathered}$            (25)

where, $F_{\text {fuse}}$ denotes the final multimodal fused feature. Through residual addition, the original effective features of each branch are preserved while cross-modal complementary information is incorporated, thereby maximizing the retention of both structural features and appearance details of martial arts movements. The fused features are fed into a two-layer fully connected classification structure to accomplish action category discrimination, with the normalized probability distribution output as:

$\hat{y}=\operatorname{Softmax}\left(F_{\text {fuse}} W_{\text {cls}}+b_{\text {cls}}\right)$             (26)

where, $W_{\text {cls}}$ and $b_{\text {cls}}$ denote the learnable weights and bias terms of the classification layer, respectively, and $\hat{y}$ denotes the action recognition prediction output by the model. To accommodate the real-time inference requirements of edge devices, an adaptively switchable operation mechanism is configured for this module. The multimodal fusion strategy is enabled under adverse scenarios such as occlusion and complex backgrounds to ensure recognition accuracy, while the cross-modal interaction branch is disabled under simple scenarios to reduce computational load, thereby achieving a dynamic balance between model robustness and real-time performance.