Design and Performance Evaluation of an English Oral Teaching Feedback System Integrating Visual Emotion Recognition and Pose Estimation

2.1 Problem formulation and overall system framework

This paper aims to address the problem of visual emotion recognition and teaching feedback generation for learners in English oral teaching scenarios. Let the input video sequence be $V=\left\{I_t \in \mathrm{R}^{H \times W \times 3}\right\}_{t=1}^T$, where T is the total number of frames, H and W and are the frame height and width, respectively. The system needs to synchronously extract facial expression features and body pose features from each frame $I_t$, and after multimodal fusion, output the emotion category probability distribution $y_t \in \mathrm{R}^C$, where C = 5 corresponds to five emotion states: confidence, nervousness, focus, fatigue, and neutral. Additionally, structured teaching feedback is generated based on the spatiotemporal information of the entire sequence. The core challenges of this task lie in three aspects: spatial alignment of cross-modal features, adaptive perception of modality reliability in dynamic scenarios, and effective transformation of recognition results into teaching semantics. These three challenges correspond to the design motivations of the three core modules of the system, respectively.

The overall system framework consists of three sequentially cascaded modules. Module 1 is the pose-guided facial visual attention mechanism, which takes the skeletal keypoint coordinates obtained from pose estimation as spatial priors to generate attention masks and applies them to the facial feature maps, enabling the facial feature extraction network to perceive the current body context and output pose-enhanced facial emotion features $f_{ {expr }} \in \mathrm{R}^d$, while the pose encoder outputs pose emotion feature $f_{ {pose }} \in \mathrm{R}^d$, and in this paper, d = 256. Module 2 is the dynamic confidence-weighted cross-modal adaptive fusion strategy, which estimates the real-time confidence $c_{ {expr }}$ and $c_{ {pose }}$ of the two modalities from the two dimensions of data quality and feature discriminability, respectively, and dynamically computes fusion weights through a gating mechanism to output the fused feature $f_{{fusion }} \in \mathrm{R}^d$. Module 3 is the spatiotemporal joint interpretability feedback generation mechanism oriented to oral teaching, which, based on the fused feature sequence, performs joint modeling of Grad-CAM spatial attribution and temporal analysis of emotion confidence, and generates structured teaching feedback F via semantic template mapping. The three modules are organically connected with feature alignment, adaptive fusion, and interpretable feedback as the technical main line. All modules adopt an end-to-end joint training strategy. The overall loss function $L_{ {total }}$ is composed of a weighted combination of emotion classification loss and feedback consistency loss; see Section 2.6 for details.

2.2 Visual feature extraction module

The facial feature extraction branch starts from the input frame $I_t$.  First, an improved RetinaFace detector is used to extract the facial region, obtaining a facial image $I^{(t)}{ }_{ {face }} \in \mathrm{R}^{224 \times 224 \times 3}$ cropped and resized to $224 \times 224$. The backbone network for facial feature extraction is ResNet-50 pre-trained on AffectNet, and the feature map output from its fourth residual stage is $F^{(t)}{ }_{{face }} \in \mathrm{R}^{2048 \times 7 \times 7}$. Although single-scale deep features possess strong semantic abstraction capabilities, their spatial resolution is low, making it difficult to preserve the textural details of subtle facial regions. To this end, this paper introduces a multi-scale feature aggregation strategy: the $1024 \times 14 \times 14$ feature map output from the third residual stage is concatenated with the fourth-stage feature via a skip connection, and then reduced in dimension by a $1 \times 1$ convolution, ultimately obtaining the multi-scale facial feature map $F^{\sim(t)_{f a c e} \in \mathrm{R}^{C_f \times H_f \times W_f}}$, where $C_f=256, H_f=7$, and $W_f=7$. This multi-scale feature map provides a feature basis with more complete spatial structure preservation for the subsequent pose-guided attention weighting. Synchronously, 468 facial landmark coordinates $L_{{face }}=\left\{\left(u_l, v_l, r_l\right)\right\}_{l=1}^{468}$ are extracted, where $r_l$ is the visibility confidence; these landmarks will be used for the data quality assessment in modality confidence estimation in Section 2.3. Figure 1 illustrates the network architecture of the dual-branch visual feature extraction module.

1.png

Figure 1. Network architecture diagram of the dual-branch visual feature extraction module

The pose feature extraction branch adopts the HRNet-W32 architecture, with the original frame $I_t$ s input, and outputs 17 skeletal keypoint coordinates in COCO format $P_t=\left\{p_k=\left(x_k, y_k, s_k\right)\right\}_{k=1}^{17}$, where $s_k \in[0,1]$ is the detection confidence. Not all 17 keypoints contribute equally to emotion expression, so this paper selects K' = 6 keypoints that are highly correlated with emotion expression to form the pose subset $P_{ {pose }}=\left\{p_{ {head }}, p_{ {neck }}, p_{l-{ shoulder }}, p_{r-{ shoulder }}, p_{l- { hip }}, p_{r-{ hip }}\right\}$, aiming to remove redundant nodes weakly associated with emotional semantics, such as limb extremities, while reducing the computational overhead of subsequent spatiotemporal graph encoding. To capture the spatial structural features and temporal dynamic features of the pose, a Spatiotemporal Graph Convolutional Network (ST-GCN) is employed to encode the pose sequence. The spatial graph structure is defined as $G=(V, E)$, where the node set $V$ contains K' keypoints, and the edge set $E$ is defined based on natural human skeletal connections. The graph convolution operation of the l-th layer of ST-GCN is:

$H^{(l+1)}=\sigma\left(\widetilde{D}^{-\frac{1}{2}} \widetilde{A} \widetilde{D}^{-\frac{1}{2}} H^{(l)} W^{(l)}\right)$      (1)

where, $\widetilde{A}=A+I_K$ is the adjacency matrix with added self-loops, $I_{K^{\prime}}$ is the $K^{\prime} \times K^{\prime}$ identity matrix, the introduction of self-loops allows the graph convolution to retain the node's original features while aggregating neighborhood information; $\widetilde{D}$ is the degree matrix of $\widetilde{A}$, used for symmetric normalization of the adjacency matrix to prevent nodes with large degrees from dominating the feature update during aggregation; $H^{(l)} \in \mathrm{R}^{K^{\times} d_l}$ is the node feature matrix of the l-th layer, and $d_l$ is the feature dimension of the current layer; $W^{(l)} \in \mathrm{R}^{d_l \times d_{l+1}}$ is the learnable weight matrix, and $\sigma$ is the ReLU activation function. In the temporal dimension, a $1 \times \tau$ temporal convolution kernel ($\tau$ = 5) is used to capture the pose evolution patterns between adjacent frames, enabling the network to perceive the dynamic change trend of pose within a short-time window.

After 4 layers of ST-GCN encoding, global average pooling is performed on the node dimension to aggregate the features of each node into a fixed-length global pose representation, resulting in the pose emotion feature vector $f^{(t)}{ }_{{pose }} \in \mathrm{R}^{256}$. The facial feature branch is mapped via global average pooling and a fully connected layer from the multi-scale feature map $f^{(t)}_{face}$, similarly outputting a facial emotion feature $F^{\sim(t)}{ }_{ {expr}}$ with a dimension of 256. The output dimensions of the two branches are kept consistent, providing a prerequisite for feature space alignment for subsequent cross-modal attention guidance and dynamic fusion. The alignment of facial and pose features in dimension is not merely an engineering choice, but the foundation for ensuring that the pose-guided attention mask can directly act on the spatial facial feature map and that the gating fusion mechanism can compare the contributions of the two modalities in a unified feature space.

2.3 Pose-guided facial visual attention mechanism

The quality of facial expression feature extraction directly depends on whether the spatial regions attended to by the network on the feature map are semantically relevant to emotions. Traditional facial feature extraction networks learn attention distributions in a data-driven manner, lacking explicit spatial guidance from body context. When the facial image itself suffers from uneven illumination or partial occlusion, it is difficult for the network to autonomously determine which regions are more critical for emotion judgment in the current pose context. This paper designs a pose-guided facial visual attention mechanism that transforms the structured spatial information obtained from pose estimation into an attention prior on the facial feature map, enabling the facial feature extraction process to perceive the current body pose context. This module comprises three steps: spatial coordinate mapping, heatmap generation, and attention weighting. Figure 2 illustrates the principle of the pose-guided facial visual attention mechanism.

2.png

Figure 2. Schematic diagram of the pose-guided facial visual attention mechanism

Spatial coordinate mapping addresses the cross-coordinate system alignment problem between pose keypoints and the facial feature map. The pose keypoints P^(t)_pose are mapped from the original image coordinate system to the facial feature map coordinate system. Due to the affine deformation of the facial region in the image caused by head pose changes, linear transformations struggle to accurately describe this non-rigid mapping relationship. Therefore, this paper employs Thin Plate Spline (TPS) transformation for nonlinear mapping. The TPS transformation parameters $\Theta_{T P S}$ are solved jointly using the head-related points $p_{ {head }}$ and $p_{ {neck }}$ from the pose keypoints and the four corner points of the facial detection box:

$\Theta_{T P S}=\arg \min _{\Theta} \sum_{q \in Q} \| q-\left.T P S(q ; \Theta)\right|_2 ^2+\lambda \cdot$ bending energy      (2)

where, Q is the control point set, formed by the head pose keypoints and the corner points of the facial detection box; $\lambda$ is the regularization coefficient used to control the smoothness of the transformation to prevent overfitting. After TPS mapping, the corresponding coordinate of each pose keypoint $p_k$ in the facial feature map space is $\left(\tilde{x}_k, \tilde{y}_k\right)=T P S\left(p_k ; \Theta^*{ }_{T P S}\right)$. This mapping unifies the spatial structure information of the pose from the original image coordinate system into the facial feature map space, providing a prerequisite for spatial alignment for the subsequent attention mask generation.

A 2D Gaussian heatmap is generated for each mapped keypoint, constructing a spatial attention distribution centered at the keypoint location:

$G_k(i, j)=\exp \left(-\frac{\left(i-\tilde{x}_k\right)^2+\left(j-\tilde{y}_k\right)^2}{2 \sigma_k^2}\right), i \in\left[1, H_f\right], j \in\left[1, W_f\right]$      (3)

where, the Gaussian kernel bandwidth $\sigma_k$ is adaptively adjusted based on the detection confidence of the keypoint $s_k$: $\sigma_k=\sigma_0 \cdot\left(1+\beta \cdot\left(1-s_k\right)\right) . \sigma_0$. $\sigma_0$ is the base bandwidth, set to 2.5 in this paper, and $\beta$ is the adjustment coefficient, set to 0.6. The physical rationale behind this design is that keypoints with lower detection confidence exhibit greater spatial position uncertainty; hence, a larger heatmap diffusion radius is assigned to cover a broader potential facial region. The K' heatmaps are concatenated along the channel dimension into $G \in \mathrm{R}^{K^{\prime} \times H_f \times W_f}$, which is then compressed to 1 channel via a $1 \times 1$ convolution, followed by a Sigmoid activation to obtain the attention mask:

$A_{ {pose }}=\sigma\left({Conv}_{1 \times 1}(G)\right), A_{ {pose }} \in[0,1]^{H_f \times W_f}$      (4)

This mask possesses a clear semantic interpretation in the spatial dimension: high-response regions indicate the facial areas that the feature extraction network should focus on given the current pose context. The mask is multiplied element-wise with the facial feature map, and a residual connection is introduced to preserve the original features, preventing potential deviations in the pose prior from causing the loss of effective information:

$F_{ {face }}^{ {guided }}=F_{ {face }} \odot U p S a m p l e\left(A_{ {pose }}\right)+F_{ {face }}$      (5)

where, UpSample( ) uses bilinear interpolation to upsample $A_{pose}$ to the spatial resolution of $F_{face}$. The introduction of the residual connection is based on the following consideration: while spatial guidance from the pose prior is beneficial in most cases, when pose estimation suffers severe deviations, the attention mask may mislead the facial feature network to focus on incorrect regions. The residual connection provides a data-driven pathway for the network to bypass this misleading information. F^guided_face is subsequently mapped via global average pooling and a fully connected layer to obtain the pose-guided enhanced facial emotion feature $f^{(t)}{ }_{{expr}} \in \mathrm{R}^{256}$. At this point, the facial feature extraction process incorporates structured spatial information from the pose modality, achieving fine-grained alignment of cross-modal information in the image feature space.

2.4 Dynamic confidence-weighted cross-modal adaptive fusion strategy

The Pose-Guided Facial Attention (PGFA) module achieves fine-grained alignment of cross-modal features at the spatial level, but the relative importance of the two modalities during fusion still relies on a fixed weight allocation strategy. In oral teaching scenarios, factors such as sudden illumination changes, head rotations, and body occlusions alternately affect the data quality of the facial and pose modalities. A fixed-weight fusion strategy struggles to maintain optimal performance under dynamic conditions. This paper designs a dynamic confidence-weighted cross-modal adaptive fusion strategy, which evaluates the real-time confidence of each modality from two dimensions—data quality and feature discriminability—and implements data-driven dynamic weight allocation through a gating mechanism. Figure 3 presents the architecture of the dynamic confidence-weighted cross-modal adaptive fusion module.

3.png

Figure 3. Dynamic confidence-weighted cross-modal adaptive fusion module

Data quality confidence reflects the reliability of the input signal. For the facial modality, data quality is determined jointly by three aspects: facial detection box area, head orientation, and landmark visibility:

$\begin{gathered}c_{ {expr }}^{ {data }}=\lambda_1 \cdot \frac{{area}( { face })}{H \cdot W}+\lambda_2 \cdot \cos \left(\theta_{ {yaw }}\right) \cdot \cos \left(\theta_{ {pitch }}\right)  +\lambda_3 \cdot \frac{1}{468} \sum_{l=1}^{468} r_l\end{gathered}$      (6)

where, area(face) is the area of the facial detection box; $H \cdot W$ is the total image area, their ratio reflects the proportion of the face in the frame; $\theta_{ {yaw }}$ and $\theta_{ {pitch }}$ are the head yaw and pitch angles, respectively, solved via the Perspective-n-Point (PnP) algorithm using the 468 facial landmarks. Their cosine product approaches 1 when the head is facing forward and approaches 0 when the head is turned sideways or lowered; $r_l$ is the visibility confidence of the l-th landmark. The weights $\lambda_1, \lambda_2, \lambda_3$ are set to 0.3, 0.4, and 0.3, respectively, highlighting the dominant influence of head orientation on facial expression recognition. For the pose modality, the pose data quality confidence is the weighted average of the detection confidence of the 17 skeletal keypoints:

$c_{ {pose }}^{ {data }}=\sum_{k=1}^{17} \omega_k \cdot s_k, \sum_{k=1}^{17} \omega_k=1, \omega_k>0$      (7)

where, $s_k$ is the detection confidence of the k-th keypoint, and $\omega_k$ is the weight corresponding to each keypoint. Points related to the shoulders and head are assigned higher weights to emphasize their relevance to emotion expression. The data quality confidence only reflects the reliability at the input signal level, but high data quality does not guarantee that the feature is discriminative in the current classification task. Therefore, a supplementary assessment from the feature discriminability dimension is required. The feature discriminability confidence is dynamically updated during the training process based on the ratio of between-class scatter to within-class scatter, calculated using a sliding window approach. Let the feature set of modality m in the current mini-batch be $\left\{f_m^{(n)}\right\}_{n=1}^N$, with corresponding class labels $\left\{y^{(n)}\right\}_{n=1}^N$. The between-class scatter matrix $S_m^{ {between }}$ and within-class scatter matrix $S_m^{ {within}}$ are defined as:

$\begin{gathered}S_m^{ {between }}=\sum_{c=1}^C N_c \cdot\left(\mu_c-\mu\right)\left(\mu_c-\mu\right)^{\top}, S_m^{{within }}= \sum_{c=1}^C \sum_{n: y^{(n)}=c}\left(f_m^{(n)}-\mu_c\right)\left(f_m^{(n)}-\mu_c\right)^{\top}\end{gathered}$      (8)

where, $\mu_c$  is the feature mean of class c, $\mu$ is the global feature mean, and $N_c$ is the number of samples in class c. The larger the between-class scatter and the smaller the within-class scatter, the more effective the feature is for the current class discrimination task. The feature discriminability confidence maps the scatter ratio to the [0,1] interval via the Sigmoid function:

$c_m^{ {disc }}={sigmoid}\left(\frac{{Tr}\left(S_m^{ {between }}\right)}{{Tr}\left(S_m^{ {within }}\right)+\epsilon}-\delta\right)$      (9)

where ${Tr}( )$ is the trace of the matrix, $\delta=0.5$ is the bias threshold, and $\epsilon=10^{-8}$ is a tiny constant to prevent division by zero.

The comprehensive confidence $c_m$ is the geometric mean of the data quality confidence and the feature discriminability confidence:

$c_m=\sqrt{c_m^{ {data }} \cdot c_m^{ {disc }}}, m \in\{expr,pose \}$      (10)

The geometric mean is adopted instead of the arithmetic mean because the geometric mean is more sensitive to low values in either dimension. The comprehensive confidence approaches 1 only when both confidence values are high, which aligns with the design expectation that the overall confidence should be significantly suppressed if either data quality or discriminability is lacking. The calculation of fusion weights also needs to consider the temporal evolution trend of confidence to avoid severe inter-frame oscillations. This paper introduces exponential moving average for temporal smoothing, defining the smoothed confidence $\tilde{c}_m^{(t)}$ as: 

$\tilde{c}_m^{(t)}=\alpha \cdot c_m^{(t)}+(1-\alpha) \cdot \tilde{c}_m^{(t-1)}$      (11)

The smoothing coefficient $\alpha=0.3$ and the initial value $\tilde{c}_m^{(0)}=0.5$ are chosen, making the fusion weights evolve smoothly over time. The smoothed confidence is input into the gating network to compute the dynamic fusion weights:

$g^{(t)}={Softmax}\left(W_g \cdot\left[\tilde{c}_{{expr }}^{(t)}, \tilde{c}_{{poss }}^{(t)}\right]^{\top}+b_g\right)$      (12)

where, $W_g \in \mathrm{R}^{2 \times 2}$ and $b_g \in \mathrm{R}^2$ are learnable parameters. The Softmax function ensures $g^{(t)}{ }_{ {expr }+} g^{(t)}{ }_{ {pose }}=1$ and that each component is non-negative. The fused feature is the weighted sum of the two modal features:

$f_{ {fusion }}^{(t)}=g_{ {expr }}^{(t)} \cdot f_{{expr }}^{(t)}+g_{ {pose }}^{(t)} \cdot f_{ {pose }}^{(t)}$      (13)

The advantage of this linear weighted form is that the gradient can be backpropagated end-to-end to the feature extraction networks and the gating network of both modalities. To further enhance the model's robustness when either modality fails, during the training stage, the gating output $g^{(t)}$ is replaced with a uniform distribution [0.5, 0.5] with a probability of 0.1. This random perturbation regularization forces both the pose encoder and facial encoder to possess the ability to perform classification independently even when the other modality is missing, thereby improving the robustness of modality complementarity.

2.5 Spatiotemporal joint interpretability feedback generation mechanism for oral teaching

The first two modules complete the extraction and adaptive fusion of emotion features. The fused feature f^(t)_fusion passes through a two-layer fully connected network and Softmax activation to obtain the emotion probability distribution $y^{(t)}={Softmax}\left(W_{c l s} f^{(t)}{ }_{ {fusion }}+b_{c l s}\right)$, and the final emotion category is $\hat{y}^{(t)}=\arg \max_c y^{(t)}{ }_c$. However, single-frame emotion classification results fluctuate temporally, and isolated emotion labels themselves lack teaching guidance value, making it difficult to directly serve teaching decisions. To this end, this paper designs a spatiotemporal joint interpretability feedback generation mechanism, which transforms emotion recognition results into structured feedback with spatiotemporal localization and teaching semantics. Figure 4 presents the visualization principle of the spatiotemporal joint interpretability feedback generation for oral teaching. First, a bidirectional long short-term memory network (BiLSTM) is used to model the temporal dynamics of the fused feature sequence, obtaining context-enhanced features:

$h_t^{b i}=\left[\overrightarrow{L S T M}\left(f_{{fusion }}^{(t)}\right) ; \overleftarrow{L S T M}\left(f_{ {fusion }}^{(t)}\right)\right]$      (14)

where, $\overrightarrow{L S T M}$ and $\overleftarrow{L S T M}$ model the feature sequence from the forward and backward directions, respectively. The concatenated vector $\mathrm{h}_t^{b i}$ of their outputs integrates past and future contextual information. This design enables the final emotion prediction to be based on both the current frame's fused feature and the preceding and succeeding temporal context, effectively suppressing accidental fluctuations in single-frame predictions. For temporal boundary frames, a forward or backward filling strategy is adopted to complete the context window, ensuring that predictions at any position in the sequence have sufficient temporal information support.

4.png

To explain the visual evidence relied upon by the model's decisions, this paper performs spatial attribution mapping on the emotion prediction scores. For the prediction score $y_c^{(t)}$ of the target emotion category c at time t, the gradients with respect to the k-th facial feature map channel $A^k_{face}\in \mathrm{R}^{H_f \times W_f}$ and the k'-th pose graph convolution layer feature map A^k’_pose are calculated separately. The facial Grad-CAM weight $\alpha_k^{c, t}$ is obtained by globally average pooling the gradients over all spatial positions of the feature map:

$\alpha_k^{c, t}=\frac{1}{Z_f} \sum_{i=1}^{H_f} \sum_{j=1}^{W_f} \frac{\partial y_c^{(t)}}{\partial\left(A_{ {face }}^k\right)_{i j}}$      (15)

where, $Z_f=H_f \times W_f$ is the total number of spatial positions. The facial attribution heatmap is obtained by linearly weighting the channel feature maps according to the weights and applying ReLU activation:

$L_{ {face }}^{c, t}={ReLU}\left(\sum_k \alpha_k^{c, t} A_{{face }}^k\right)$      (16)

Similarly, pose attribution is obtained by calculating gradients with respect to the node features of the graph convolution layer, yielding $L_{ {pose }}^{c, t} \in \mathrm{R}^{K^{\prime}}$, which indicates the skeletal keypoints that contribute most to the current decision. The attribution heatmap answers, in the spatial dimension, which facial regions and which pose keypoints the model focuses on. However, frame-by-frame independent attribution is susceptible to gradient noise interference; heatmaps of adjacent frames may exhibit discontinuous jumps in spatial distribution, affecting the coherence of subsequent feedback. This paper applies temporal Gaussian filter smoothing to the heatmap sequence, giving the attribution results smooth evolution characteristics in the temporal dimension.

Attribution in the temporal dimension reveals the dynamic evolution pattern of emotional states during oral expression. Define the emotion state confidence sequence as $s_c(t)=y_c^{(t)}$. To locate key moments when the emotion state changes significantly, its first-order difference and second-order difference are calculated:

$\Delta s_c(t)=s_c(t)-s_c(t-1), \Delta^2 s_c(t)=\Delta s_c(t)-\Delta s_c(t-1)$      (17)

The emotion turning point $t^*$ is located at the moment when the second-order difference takes a local maximum, i.e., the inflection point of the confidence curve, which corresponds to the critical point where the emotion state accelerates upward or downward. Further explanation requires establishing an association between emotion dynamics and teaching events. Teaching event nodes $\mathrm{E}=\left\{e_1, e_2, \ldots, e_M\right\}$ related to oral teaching tasks (including starting reading, encountering unfamiliar words, teacher questioning, ending expression, etc.) are aligned with the video timeline. For each turning point t*, calculate its temporal distance from the nearest event node $\tau=\min _{e \in \mathrm{E}}\left|t^*-t_e\right|$, where $\tau<\Delta T$ and $\Delta T=2$. If and seconds, then this turning point is attributed to the corresponding event-driven cause, giving the emotion change a semantic explanation at the teaching level.

The attribution results and event associations ultimately need to be transformed into actionable feedback for learners. This paper constructs a semantic template library T, where each template consists of a trigger condition C, feedback text R, and visual annotation $V_{is}$. The trigger condition C is a multimodal logical expression, for example, $C=\left\{A U 4\ activation \wedge shoulder\ forward \wedge \Delta s_{ {conf }}(t)<-0.2\right\}$, which integrates three aspects: facial action units, pose spatial features, and emotion temporal changes. For the input video sequence, the template library is traversed to match trigger conditions, and the template with the highest confidence is selected for semantic filling. To avoid feedback content generated in adjacent windows being too dense in time, the longest valid subsequence algorithm is used to select M' key time windows in the sequence, with at least 1 second time interval between windows. The final structured feedback is output as:

$F=\left\{\left(t_{ {start }}^{(m)}, t_{ {end }}^{(m)}, L_{{face }}^{(m)}, L_{ {pose }}^{(m)}, R^{(m)}\right)\right\}_{m=1}^{M'}$      (18)

where, t^(m)_start and t^(m)_end mark the time window corresponding to the feedback, L^(m)_face and L^(m)_pose are the corresponding spatiotemporal attribution heatmaps, and L^(m) is the natural language feedback text. This structured format covers three dimensions simultaneously: temporal localization, visual evidence, and teaching semantics, elevating the system output from isolated classification labels to teaching feedback with a complete explanation chain.

2.6 Loss function and training strategy

The overall loss function consists of three parts: emotion classification loss, confidence consistency loss, and regularization loss, combined as $L_{{total }}=L_{ {cls }}+\lambda_1 L_{{conf }}+\lambda_2 L_{ {reg }}$. The emotion classification loss adopts Focal Loss to alleviate the problem of imbalanced emotion category distribution in oral teaching scenarios, where neutral and focused samples are significantly more numerous than nervous and fatigued samples:

$L_{c l s}=-\frac{1}{T} \sum_{t=1}^T \sum_{c=1}^C \alpha_c\left(1-y_c^{(t)}\right)^\gamma \log \left(y_c^{(t)}\right)$      (19)

where, $\alpha_c$ is the class-balanced weight, determined by normalizing the inverse of each class's sample frequency, used to assign higher loss weights to minority classes; $\gamma=2$ is the focusing parameter, controlling the loss decay rate for easy-to-classify samples, so that the model focuses optimization on hard-to-classify samples; $y_c^{(t)}$ is the model's predicted probability of frame t belonging to class c. The confidence consistency loss imposes a constraint on the positive correlation between the gating fusion weights and the unimodal classification confidence:

$L_{ {conf }}=\frac{1}{T} \sum_{t=1}^T \sum_m \| g_m^{(t)}-confidence_m^{(t)} \|_2^2$      (20)

where, $g_m^{(t)}$ is the gating fusion weight of modality m at time t, and $confidence_m^{(t)}$ is the maximum Softmax probability output when only the unimodal feature of that modality is input to the classifier. The purpose of this loss term is to make the fusion weights assigned by the gating network match the classification reliability of each modality. When a modality's feature has strong discriminability at the current moment, its unimodal classification confidence is high, and the gating network should correspondingly increase the fusion weight of that modality, and vice versa. This explicit positive correlation constraint provides direct supervision from feature discriminability to the gating network's decision process, preventing the gating weights from relying solely on data quality confidence while ignoring the actual effectiveness of features in the classification task. At the same time, it provides a clear optimization direction for the gating network in the early training stage. The regularization loss $L_{{reg}}$ imposes a norm constraint $1_2$ on the weight matrix of the gating network $W_g$, with a coefficient of $1 \times 10^{-4}$, to suppress the excessive growth of gating parameters toward extreme values during training.

Training adopts a two-stage strategy to avoid coupled oscillations in the early stage of multi-module joint optimization. In the first stage, all parameters of the gating fusion module are frozen, and only the facial feature extractor and pose feature extractor are trained under supervision, with the optimization objective being $L_{c l s}$, for 20 epochs. This stage enables the feature extractors of both modalities to establish good unimodal classification capabilities in their respective feature spaces, providing reliable unimodal classification confidence $confidence_m^{(t)}$ for subsequent confidence estimation. In the second stage, all module parameters are unfrozen, and end-to-end joint fine-tuning is performed with $L_{total}$, as the optimization objective for 30 epochs. The optimizer uses Adam with an initial learning rate of $1 \times 10^{-4}$, decaying to 0.5 times the current value every 10 epochs, and the batch size is set to 32. The weight coefficients $\lambda_1$ and $\lambda_2$ are set to 0.5 and $1 \times 10^{-4}$, respectively. This combination was determined through grid search on the validation set to keep the magnitudes of each loss term similar in the early training stage. All experiments were completed under the PyTorch framework using a single NVIDIA A100 40GB GPU.

The three modules proposed in this paper form a complete chain from feature alignment and adaptive fusion to interpretable feedback in terms of technical route. However, the actual effectiveness of each module is limited by several preconditions and implementation constraints. The semantic template library in the STEFG module currently contains 120 templates, which already covers the vast majority of common emotion-behavior patterns in oral teaching scenarios. Nevertheless, it is essentially an enumerative design within a finite state space. When learners exhibit emotional expressions that are not preset in the template library—such as limb stiffness caused by excessive anxiety or rare nervous smiles—the system will degrade to a basic mode that only outputs category labels and attribution heatmaps, and cannot generate matching semantic feedback text. The root of this limitation lies in the fact that the feedback generation stage adopts a matching-based rather than generation-based strategy. A possible direction for future improvement is to introduce a large language model, taking the attribution results as multimodal context input and having the large language model generate free-text feedback suggestions. This solution can not only break through the ceiling of the template library size, but also enable the feedback text to possess stronger semantic diversity and contextual adaptability. However, its cost is that it requires additional resolution of the balance between the inference latency of the large language model and content controllability in teaching scenarios.

The effectiveness of the PGFA module is highly dependent on the detection accuracy and spatial stability of pose estimation. Although the currently adopted HRNet-W32 performs excellently on standard pose estimation benchmarks, in the field deployment of oral teaching scenarios, the positioning accuracy of shoulder and torso keypoints noticeably decreases when learners wear loose clothing. The fundamental reason for this phenomenon is that the training data of HRNet mainly consists of images of people in tight or regular clothing. There is a distribution shift between the visual appearance prior learned by the model and the contour features under loose clothing. The positional deviation of shoulder keypoints is transmitted to the facial feature map space via TPS spatial mapping, directly affecting the spatial precision of the attention mask. In extreme cases, it may even mislead the facial feature extraction network to focus on edge regions irrelevant to emotional semantics. This problem reveals an inherent contradiction in the pose-guided attention mechanism: the original intention of this mechanism is to use pose information to compensate for the deficiencies of facial features, but when the pose information itself is unreliable, the introduced spatial prior becomes a source of noise instead. A feasible path to solve this problem is to introduce pose uncertainty estimation, using the detection confidence of each keypoint not only for the adaptive adjustment of heatmap bandwidth, but further transmitting positional uncertainty to the generation process of the attention mask in the form of a probability distribution, enabling the network to automatically reduce the intensity of spatial guidance when pose estimation is unreliable.

The design of the current system takes single-person video input as the basic assumption, and this constraint limits the applicability of the method in real classroom scenarios. In teaching activities such as group discussions and peer assessments, multiple learners may appear in the same frame, and the problem of individual association confusion becomes the main obstacle to system deployment. The skeletal keypoints output by the pose estimation branch in multi-person scenarios lack explicit modeling of identity attribution. The shoulder and head keypoints of different individuals are spatially interleaved, making it difficult for the PGFA module to determine which subject's pose information should be used to guide facial attention. If directly extended to multi-person scenarios, the system framework needs to be redesigned: introducing a multi-target tracking mechanism at the perception level to achieve temporal association of individual identities; constructing an individual matching strategy at the attention guidance level based on the spatial subordination between facial detection boxes and pose keypoints; and maintaining independent emotion state sequences for each individual and generating individualized diagnostic reports at the feedback generation level. The above extensions not only involve the reconstruction of the perception front-end, but also require the dataset annotation paradigm to evolve from single-subject emotion labels to multi-subject interactive emotion annotation. Its complexity exceeds the scope of the current methodology and can be explored in depth as an independent follow-up research direction.