Action Recognition and Instructional Feedback in Physical Education via Mobile Interaction

2.1 System overall architecture

To meet the requirements for real-time performance, precision, and personalized instructional feedback in mobile sports classroom scenarios, this paper designs and implements an end-to-end four-layer pipeline system architecture. Each layer adopts an asynchronous parallel execution mechanism to ensure overall system performance and adaptability for mobile deployment, with end-to-end latency strictly controlled within 80ms, providing stable support for real-time interactive teaching in sports classrooms. The specific architecture is shown in Figure 1. The Image Acquisition and Preprocessing Layer is responsible for the real-time capture of sports action images via the mobile camera, synchronously performing background suppression and keyframe sampling operations. This effectively filters out complex background interference in classrooms and reduces computational load, providing high-quality input for subsequent feature extraction stages. The Enhanced Multi-scale Attention Pose (EMAP) Feature Extraction Layer, based on a lightweight multi-scale attention network, efficiently extracts human joint heatmaps and part affinity fields to achieve high-precision joint localization in mobile scenarios. The STD-GCN Action Recognition and Quality Assessment Layer models the spatio-temporal correlations and dynamic changes of joint movements through a STD-GCN to complete action category recognition and quality scoring, accurately capturing the subtle dynamic features of sports actions. The Feedback Visualization Layer transforms action recognition results and joint angle error information into intuitive visual outputs, synchronously providing action scoring, error localization, and correction prompts to meet the fine-grained requirements of instructional feedback. Each layer utilizes an asynchronous parallel execution mechanism to facilitate parallel data flow and processing, effectively shortening overall processing latency. Furthermore, the entire system adopts a lightweight design, eliminating the need for high-performance computing equipment. It can be directly deployed on ordinary mobile devices, balancing convenience and practicality. The system flexibly adapts to the variable teaching scenarios of sports classrooms, providing stable and efficient architectural support for real-time action recognition and personalized instructional feedback.

Figure 1. Overall four-layer pipeline architecture of the mobile action recognition system for sports classrooms

2.2 Mobile image acquisition and adaptive preprocessing

The mobile image acquisition module utilizes the device's built-in camera to complete the real-time capture of sports action sequences. The acquisition resolution is set to 1280×720 with an initial frame rate of 30 FPS, balancing image clarity and data transmission efficiency to ensure the complete capture of subtle dynamic changes in human joint movements. Given the scene characteristics of sports classrooms—such as variable lighting, complex backgrounds, and multi-target occlusion—traditional preprocessing methods struggle to balance background interference suppression with computational control, failing to meet the low-resource, low-latency operational requirements of mobile terminals. Therefore, an adaptive preprocessing scheme is designed. Through the synergistic effect of frequency-domain fast foreground separation and optical flow energy keyframe sampling, dual optimization of background suppression precision and mobile real-time performance is achieved, providing high-quality input data for subsequent pose feature extraction. Figure 2 illustrates the flowchart of the adaptive image preprocessing and frequency-domain foreground separation.

Figure 2. Flowchart of adaptive image preprocessing and frequency-domain foreground separation

The adaptive background suppression strategy achieves fast foreground separation based on frequency domain transformation. Its core lies in performing frequency domain decomposition of the image via Discrete Cosine Transform (DCT), utilizing the differences in frequency domain distribution between background and foreground to achieve efficient separation, avoiding the defects of large computational load and weak anti-interference ability associated with traditional spatial domain methods. Background modeling adopts an adaptive update mechanism, with the core formula as follows:

$B_t=\alpha B_{t-1}+(1-\alpha) D C T^1\left(M_{\text {low }} \odot D C T\left(I_t\right)\right)$         (1)

where, $B_t$ is the background model of the $t$-th frame, $B_{t-1}$ is the background model of the previous frame, $\alpha$ is set to 0.95 to control the background update rate, balancing background stability and dynamic adaptability; $I_t$ is the captured image of the $t$-th frame, $D C T$ and $D C T^{-1}$ represent the Discrete Cosine Transform and inverse transform respectively; $M_{\text {low}}$ is a lowpass mask retaining $10 \%$ of the low-frequency components in the image frequency domain, used for smooth updating of the background model, effectively suppressing the interference of high-frequency noise on background modeling. Foreground separation is achieved through inter-frame frequency domain difference threshold judgment, with the core formula as follows:

$F_t(x, y)=1\left(\left|D C T\left(I_t-B_t\right)\right|^2>\tau\right)$             (2)

where, $F_t(x, y)$ is the foreground mask of the t-th frame, 1 is an indicator function that outputs 1 (foreground) when the condition inside the parentheses is met, otherwise outputs 0 (background); $\tau$ adopts an adaptive threshold rule, dynamically adjusted based on the variance of frequency domain differences of the previous 5 frames, ranging from 50 to 120, ensuring precise separation of foreground humans and background environments under different lighting conditions. Compared with the traditional Gaussian Mixture Model, this strategy reduces computational complexity by 67%, increases processing speed by 3 times, enables real-time background suppression on mobile terminals, effectively eliminates stray responses caused by complex classroom backgrounds, and improves the accuracy of subsequent pose estimation.

To further reduce the computational burden on mobile terminals and improve the overall system frame rate, based on background suppression, a keyframe sampling strategy based on optical flow energy changes is designed. By screening frames containing key action information for subsequent processing, the computational overhead of redundant frames is reduced. Optical flow energy $E_k$ is used to characterize the intensity of human motion in the k-th frame image, calculated based on the Lucas-Kanade optical flow method, reflecting the motion amplitude of pixels between adjacent frames. The core of keyframe sampling is to calculate the cumulative change in optical flow energy, with the formula as follows:

$S_t=\sum_{k=t-L}^t \frac{\left|E_k-E_{k-1}\right|}{\max \left(E_k, E_{k-1}\right)}$                    (3)

where, $S_t$ is the cumulative change in optical flow energy of the t-th frame, L is set to 5, i.e., calculating the sum of optical flow energy changes over the last 5 frames, balancing sampling accuracy and real-time performance; $\max \left(E_k, E_{k-1}\right)$ is used for normalization to avoid sampling bias caused by absolute value differences in optical flow energy. When $S_t$ is greater than the set threshold T_key  = 0.35, the frame is determined as a keyframe, retained and used for subsequent pose extraction and action recognition; otherwise, it is determined as a redundant frame, used only for background model updating without subsequent complex calculations.

In the adaptive preprocessing scheme, the background suppression and keyframe sampling modules adopt an asynchronous parallel execution mechanism. While the background suppression module processes the current frame, the keyframe sampling module calculates and judges the optical flow energy of the previous frame, achieving pipeline advancement of data processing and further shortening processing latency. Actual tests show that the single-frame processing time of this preprocessing scheme on mobile terminals is only 8~12 ms, and it can effectively suppress interference from complex backgrounds and lighting changes, achieving a foreground separation accuracy of over 92%. It provides high-quality, low-redundancy input data for the subsequent EMAP pose feature extraction layer, while ensuring the overall end-to-end latency of the system is controlled within 80 ms, fully adapting to the real-time interaction requirements of mobile sports classrooms.

2.3 Lightweight multi-scale attention pose feature extraction network

To achieve high-precision and high-efficiency human pose feature extraction in mobile scenarios, a lightweight multi-scale attention pose feature extraction network is designed. Taking preprocessed RGB keyframes as input, this network outputs human joint heatmaps and part affinity fields, providing core feature support for subsequent action recognition and quality assessment. The network input is an RGB image of size 3×H×W, where H and W are the image height and width, respectively. The output includes K joint heatmaps and L part affinity fields; K and L are set according to the requirements of key joints and associated parts for sports actions, ensuring the complete capture of pose features for human motion. The network overall adopts a multi-resolution parallel convolution backbone structure, balancing pose detail capture and global semantic information extraction through feature extraction and fusion at different scales, solving the problem that single-scale feature extraction struggles to balance precision and efficiency. The specific architecture is shown in Figure 3.

The multi-resolution parallel convolution backbone consists of three parallel convolution branches. Each branch uses inputs with different downsampling ratios and extracts features at different scales via 3×3 convolution kernels. The core formulas are as follows:

$F_1=\operatorname{Conv}_{3 \times 3,32}\left(X_{\downarrow 1}\right)$                  (4)

$F_2=\operatorname{Conv}_{3 \times 3,48}\left(X_{\downarrow 2}\right)$                    (5)

$F_3=\operatorname{Conv}_{3 \times 3,64}\left(X_{\downarrow 4}\right)$                      (6)

where, $X_{\downarrow 1}, X_{\downarrow 2}$, and $X_{\downarrow 4}$ represent the input image at original resolution, downsampled by 2 times, and downsampled by 4 times, respectively; $\operatorname{Conv}_{3 \times 3,32}$ represents a convolution operation with a $3 \times 3$ kernel and 32 output channels. The feature maps $F_1, F_2$, and $F_3$ extracted by the three branches correspond to high, medium, and low scales, respectively. To achieve cross-scale feature fusion, $F_2$ and $F_3$ are upsampled to the resolution of $F_1$ via bilinear interpolation, then added element-wise with $F_1$ to obtain the fused feature map $F_1$. The core formula is:

$F_1^{\prime}=F_1+U p\left(F_2\right)+U p\left(F_3\right)$                (7)

where, ${U p}(\cdot)$ represents the upsampling operation. This fusion strategy retains the joint detail information in the highresolution branch $F_1$ while incorporating the global semantic features from the medium-low resolution branches $F_2$ and $F_3$, effectively improving the completeness and accuracy of pose feature extraction. Simultaneously, the parallel convolution design reduces network computational complexity, adapting to mobile deployment requirements.

Figure 3. Structure diagram of the enhanced Multi-Scale Attention Pose (EMAP) feature extraction network

To further enhance the representation capability of key pose features and suppress interference from background and redundant information, a channel-spatial cooperative attention module is embedded after cross-scale fusion. This module precisely strengthens key pose features through coordinated modulation in both channel and spatial dimensions. Channel attention calculation first performs global average pooling and max pooling on the channel dimension of the fused feature $\operatorname{map} F_1$, obtaining $z_{\text {avg}}$ and $z_{\text {max}}$. After processing by a twolayer lightweight fully connected network, the channel attention weight $a_c$ is generated via a sigmoid activation function. The core formula is:

$a_c=\sigma\left(M L P\left(z_{\text {avg }}\right)+M L P\left(z_{\max }\right)\right)$                   (8)

Spatial attention calculation performs average pooling and max pooling on the spatial dimension of the feature map, concatenating the obtained $s_{\text {avg }}$ and $s_{\text {max }}$, extracting features via a $7 \times 7$ convolution kernel, and generating spatial attention weight $a_s$ via sigmoid activation. The core formula is:

$a_s=\sigma\left(\operatorname{Conv}_{7 \times 7}\left(\left[s_{\text {avg}} ; s_{\max }\right]\right)\right)$                     (9)

Finally, the channel attention weights and spatial attention weights are multiplied element-wise to obtain the cooperative attention weights. These are element-wise weighted with the original fused feature map and added with a residual connection to achieve feature enhancement and gradient propagation optimization. The core formula is:

$F^{\prime}=F \otimes\left(a_c \cdot a_s\right)+F$          (10)

The network output end is designed with a dual-branch structure, used for predicting joint heatmaps and part affinity fields, respectively. Both branches share backbone features and attention-enhanced features, ensuring feature consistency and computational efficiency. The heatmap branch predicts the spatial position probability distribution of each joint, using Gaussian heatmaps as labels with a Gaussian distribution standard deviation $\sigma$= 4, fitting the spatial distribution characteristics of joint points; the part affinity field branch predicts association vectors between adjacent joints, characterizing the connection relationships between joints. To optimize network training effectiveness, a joint loss function is designed, comprehensively considering heatmap prediction error and part affinity field prediction error. The core formula is:

$L_{\text {pose}}=\sum_{k=1}^K\left\|H_k-H_k^*\right\|_2^2+\lambda_{\text {paf}} \sum_{l=1}^L\left\|L_l-L_l^*\right\|_2^2$                     (11)

where, $H_k$ and $H^*{ }_k$ are the predicted heatmap and labeled heatmap of the $k$-th joint, respectively; $L_l$ and $L^*{ }_l$ are the predicted value and labeled value of the $l$-th part affinity field, respectively; $\lambda_{\text {paf }}=0.5$ is used to balance the loss weights of the two branches. In the inference stage, non-maximum suppression is first performed on the heatmaps to screen joint point candidates with confidence above a set threshold, followed by matching the part affinity fields via the Hungarian algorithm to complete human skeleton assembly, ensuring the accuracy of joint connections and providing high-precision pose skeleton input for subsequent action recognition. The network overall uses depthwise separable convolutions to replace some standard convolutions, further reducing computational complexity. Mobile terminal tests show it fully meets real-time processing requirements.

2.4 Spatio-temporal differential graph convolutional network

Based on the human joint heatmaps and part affinity fields output by the EMAP network, human joint coordinates in continuous keyframes can be extracted to construct spatio-temporal graphs for modeling the spatial correlations and temporal dynamic features of joints. STD-GCN is dedicated to accurately capturing the dynamic variation patterns of sports actions, achieving precise recognition of action categories and quantitative assessment of action quality, while balancing the low computational requirements of mobile terminals. It ensures real-time performance through structural optimization, providing core technical support for sports classroom action assessment. Figure 4 shows the STD-GCN and the contrastive scoring siamese architecture.

Figure 4. Architecture diagram of Spatio-Temporal Differential Graph Convolutional Network (STD-GCN) and contrastive scoring siamese network

The spatio-temporal graph uses human joint points in continuous keyframes as the vertex set, with each vertex corresponding to the spatial coordinates and feature information of a joint at a specific moment. Spatial edges are defined as the connection relationships between adjacent joints, characterizing the spatial structure of the human skeleton; temporal edges are defined as the connection relationships of the same joint between adjacent keyframes, characterizing the temporal motion trajectories of the joints. To accurately capture the dynamic features of joint movements, a differential graph is constructed based on the spatio-temporal graph, introducing joint velocity and acceleration as edge features to quantify the changing trends of joint motion. Joint velocity is defined as the coordinate difference of the same joint between adjacent frames, with the core formula as follows:

$v_{t, k}=\left(x_{t, k}-x_{t-1, k}, y_{t, k}-y_{t-1, k}\right)$                  (12)

where, $v_{t, k}$ is the velocity vector of the $k$-th joint in the $t$-th frame, $x_{t, k}$ and $y_{t, k}$ are the horizontal and vertical coordinates of the $k$-th joint in the $t$-th frame, respectively. Joint acceleration is the difference in joint velocity between adjacent frames, with the core formula as follows:

$a_{t, k}=v_{t, k}-v_{t-1, k}$                (13)

where, $a_{t, k}$ is the acceleration vector of the k-th joint in the t-th frame. By integrating velocity and acceleration features into the edge representations, the differential graph can effectively distinguish the joint motion patterns of different sports actions. Especially for actions with significant dynamic features such as jumping and swinging, it can accurately capture subtle differences in joint movements, providing more discriminative feature support for subsequent action recognition and quality assessment.

The core of the graph convolution layer is the effective aggregation of features from the spatio-temporal graph and the differential graph. An extended neighborhood set is designed to include the spatial neighbors of the target joint and the temporal neighbors of adjacent frames, ensuring the full capture of spatio-temporal correlation information of joints. The extended neighborhood set $N(t, k)$ is defined as the spatial neighboring joints of the k-th joint in the t-th frame, as well as the corresponding same joint in the (t-1)-th and (t+1)-th frames, achieving collaborative aggregation of spatio-temporal features. The logic core formula for the original graph convolution operation is:

$f_{\text {out}}(t, k)=\sum_{\left(t^{\prime}, l\right) \in N(t, k)} \frac{1}{Z_{t, k}\left(t^{\prime}, l\right)} W_{r\left(t^{\prime}, l\right)}\left(f_{\text {in}}\left(t^{\prime}, l\right) \otimes m_{t^{\prime}, l}\right)$               (14)

where, f_out(t,k) is the output feature, Z_t,k(t',l) is a normalization coefficient used to balance the contribution weights of different neighbors, W_r(t',l) is the weight matrix corresponding to the neighbor relationship, f_in(t',l) is the input feature, and m_t',l is a motion modulation vector used to preliminarily adjust the feature weights of joints with different motion intensities. To further enhance the emphasis on features of vigorously moving joints, the graph convolution operation is optimized by introducing a gating coefficient g_t,l and constructing a dynamic modulation term based on joint velocity and acceleration features. The optimized core formula is:

$f_{\text {out}}(t, k)=\sum_{\left(t^{\prime}, l\right)} \frac{1}{Z} W_r\left(f_{\text {i }}\left(t^{\prime}, l\right)+g_{t, l} \cdot M L P\left(\left[v_{t^{\prime}, l} ; a_{t^{\prime}, l}\right]\right)\right)$              (15)

where, Z is a global normalization coefficient, W_r is a unified weight matrix, Multi-Layer Perceptron (MLP) is used for the nonlinear mapping of velocity and acceleration features, and g_t,l is generated via a sigmoid function to dynamically regulate the weight of the modulation term. This optimized design allows the network to adaptively emphasize features of vigorously moving joints, suppress redundant information from static or slowly moving joints, and improve the pertinence and discriminability of feature extraction.

The action classification process is based on the spatio-temporal features output by the graph convolution layer. Global feature vectors are extracted via global average pooling, input into a fully connected layer for feature mapping, and finally output action category probabilities via a softmax activation function. The classification loss adopts the cross-entropy loss function L_cls to optimize action classification accuracy. To achieve quantitative assessment of action quality, a siamese network contrastive learning scoring scheme is designed. Using the standard action features of professional athletes as a reference, the features of the action to be evaluated are compared with the standard action features, and the action quality score is calculated via feature similarity. The core scoring formula is:

Score $=100 \times \exp \left(-\gamma \cdot\left\|f_{\text {stu}}-f_{\text {pro }}\right\|_2^2\right)$               (16)

where, Score is the action quality score, $\gamma=0.15$ is used to adjust the influence of feature similarity on the score, $f_{s t u}$ is th global feature vector of the action to be evaluated, $f_{p r o}$ is th global feature vector of the standard action, and $\|\cdot\|_2^2$ is th squared Euclidean distance. To collaboratively optimiz action classification and quality assessment performance, total loss function is designed:

$L_{\text {total}}=L_{c l s}+\lambda_s L_{\text {score}}$                     (17)

where, $\lambda_s=0.2$ is used to balance the weights of classification loss and scoring loss, and $L_{\text {score}}$ is the scoring loss, calculated using mean squared error loss to measure the difference between the evaluated score and the manually labeled score During the action quality assessment process, joint angle errors are synchronously calculated. Based on join coordinates, the included angles between adjacent joints ar calculated, with the core formula as follows:

$\theta=\arccos \left(\frac{v_1 \cdot v_2}{\left\|v_1\right\| \cdot\left\|v_2\right\|}\right)$                  (18)

where, $v_1$ and $v_2$ are vectors of adjacent joints, and $\theta$ is the joint included angle. By comparing with the joint included angles of standard actions, joint angle errors are obtained, providing core data support for error localization and correction prompts in subsequent feedback visualization. STD-GCN adopts a lightweight design, reducing computational complexity through weight sharing and feature reuse. Mobile terminal tests show that the single-frame inference time is only 12 ms , and it can ensure real-time performance when working in coordination with the EMAP network, meeting the interaction requirements of sports classrooms.

2.5 Real-time feedback visualization and system implementation

The core objective of the real-time feedback visualization module is to transform action recognition results, joint angle errors, and quality scores into intuitive, interpretable visual information, providing precise and efficient instructional guidance for teachers and students. Its design balances intuitiveness and instructiveness, ensuring that non-technical professionals can quickly understand the location of action deviations. The joint angle error heatmap is the core carrier of feedback visualization, intuitively presenting the degree of angular deviation for each joint through color coding. The core formula is:

$\operatorname{Color}(k)=H S L\left(\min \left(1, \frac{\Delta \theta_k}{30^{\circ}}\right) \times 240^{\circ}, 1.0,0.5\right)$                      (19)

where, $\operatorname{Color}(k)$ is the color encoding of the $k$-th joint, and $\Delta \theta_k$ is the angle error of that joint. $\min \left(1, \Delta \theta_k / 30^{\circ}\right)$ is used to normalize the angle error to the $0 \sim 1$ interval, preventing color distortion caused by extreme errors; in the HSL color space, $240^{\circ}$ corresponds to blue and $0^{\circ}$ corresponds to red. The larger the angle error, the closer the color is to red; the smaller the error, the closer it is to blue. This allows teachers and students to quickly judge the severity of joint deviations through color. The rendering process uses semi-transparent line overlay, superimposing joint connection lines and the error heatmap onto the original action image. This neither obscures student action details nor clearly presents joint positions and deviation distributions; simultaneously, direction arrows are drawn at joints with large deviations, pointing towards the adjustment direction of the standard joint angle, and the action quality score is superimposed in real-time in the upper right corner of the image. The scoring range is 0~100 points, achieving synchronous visual output of error localization, correction guidance, and quality assessment, significantly enhancing the pertinence and practicality of instructional feedback.

To ensure the stable and real-time operation of the system on mobile terminals, multiple optimization strategies are adopted to complete engineering deployment and latency control. For model deployment, a TensorFlow Lite INT8 quantization scheme is used. The trained EMAP and STD-GCN models undergo integer quantization, converting model parameters from 32-bit floating-point numbers to 8-bit integers. Without significant loss of model accuracy, this reduces model size by 75%, lowers memory usage by 80%, and simultaneously greatly improves inference speed. The system adopts a double-buffered asynchronous inference mechanism, constructing two independent data buffers. One buffer is used for image acquisition and preprocessing, while the other is used for pose feature extraction, action recognition, and feedback visualization. The two buffers alternate work and execute in parallel, effectively eliminating waiting gaps in the data processing flow and improving overall processing efficiency. To verify deployment effectiveness, performance tests were conducted on mainstream mobile devices. On the iPhone 12, EMAP network single-frame inference took 28 ms, STD-GCN single-frame inference took 12 ms, image preprocessing and visualization rendering took 20 ms, and the system end-to-end latency was only 60 ms; on the Huawei Mate 50, EMAP took 32 ms, STD-GCN took 14 ms, and end-to-end latency was 68 ms. Both are below the preset target of 80 ms, and the frame rate remained stable above 30 FPS. Continuous 1-hour stability tests indicate that the system experiences no lag or crashes, can adapt to the demands of long-term continuous operation in sports classrooms, achieves real-time synchronization of action recognition and instructional feedback, and fully adapts to the actual application scenarios of mobile sports classrooms.