A Video Image Processing Approach for Classroom Interaction Detection and Dynamic Teaching Quality Evaluation in English Instruction

2.1 Overview of the framework

An end-to-end integrated framework is developed for classroom interaction frequency detection and dynamic teaching quality evaluation in English instruction. Within this framework, all modules are tightly coordinated and hierarchically organized, forming a complete technical pipeline from video input to teaching quality assessment output. Initially, input classroom video sequences are preprocessed to mitigate illumination variation and shadow interference, thereby ensuring high-quality visual data for subsequent analysis. Based on the preprocessed video frames, stable tracking of teacher and student instances is achieved through the integration of improved instance segmentation and Kalman filtering, enabling accurate extraction of individual spatial positions and persistent identity associations. This stage establishes a reliable foundation for downstream interaction feature extraction. Subsequently, a three-dimensional complementary visual feature extraction system is constructed. Interaction-related features are extracted from three perspectives—pose, facial expression, and spatial proximity—and are fused into a unified interaction intensity vector. Through this design, both fine-grained details of classroom interaction are comprehensively captured. The extracted features are then fed into a specialized spatiotemporal graph convolutional network, where classroom interaction events are precisely detected and classified. In parallel, interaction energy density is constructed by integrating pixel-level optical flow energy with joint angular velocity. Continuous quantification of interaction frequency is subsequently achieved via wavelet energy spectrum analysis. Finally, temporal misalignment caused by differences in instructional pacing across teachers is addressed using dynamic time warping. An attention-based bidirectional long short-term memory network is further employed to perform deep representation learning on interaction frequency and associated features, enabling dynamic and interpretable teaching quality evaluation. Through the coordinated optimization of all modules, full-process automation—from video image processing to teaching quality assessment—is realized. Both detection accuracy and real-time performance are maintained, ensuring effective adaptability to the complex conditions of English classroom environments. The overall framework for classroom interaction analysis and teaching quality evaluation is illustrated in Figure 1.

Figure 1. Overall framework for English classroom interaction detection and teaching quality evaluation

2.2 Improved teacher–student instance segmentation and stable tracking

To address image quality degradation caused by non-uniform illumination and shadow interference in classroom environments, a preprocessing strategy is adopted by integrating Contrast Limited Adaptive Histogram Equalization (CLAHE) with Gaussian mixture model-based grayscale modeling. Through CLAHE, local grayscale values are adaptively equalized, thereby mitigating detail loss in dark regions and local overexposure induced by backlighting or side lighting conditions. On this basis, pixel distributions in shadow regions are modeled using a Gaussian mixture model, enabling accurate detection and removal of shadow areas. As a result, standardized video frames with uniform illumination and clear object boundaries are generated, providing stable and reliable input for subsequent teacher–student instance segmentation. In addition, the preprocessing pipeline is designed to be lightweight, ensuring that real-time performance of the overall framework is preserved.

The original YOLACT instance segmentation network is further refined to accommodate the characteristics of classroom scenarios. A ResNet-50 backbone is employed, within which a top-down multi-scale feature pyramid structure is constructed to facilitate cross-level fusion of low-level detailed features and high-level semantic features. This design significantly enhances the capability of the network to extract features from small-scale student targets in long-distance classroom settings. Furthermore, anchor parameters are re-optimized based on the spatial scale distribution of teachers and students under fixed classroom viewpoints. Specifically, anchor sizes and aspect ratios are adjusted, with an increased proportion of medium- and small-scale anchors to better match student targets, while redundant large-scale anchors are reduced. Through these modifications, both the recall rate for small-object segmentation and the precision of segmentation boundaries are improved, effectively addressing the limitations of the conventional YOLACT model in densely populated classroom environments, where small targets are prone to missed segmentation and inaccurate localization.

To mitigate instance tracking drift and identity switching caused by occlusion and illumination disturbances, an adaptive weighted fusion tracking strategy is developed by integrating motion prediction based on Kalman filtering with improved instance segmentation outputs. Let the bounding box predicted by the segmentation network be denoted as B_s, and the motion-predicted bounding box from Kalman filtering be denoted as B_p. The fused final bounding box B_f is computed as:

$B_f=\omega B_s+(1-\omega) B_p$   (1)

The adaptive weight ω is jointly determined by the segmentation confidence and the motion prediction residual, and is defined as:

$\omega=\frac{c_s}{c_s+\eta e_p}$   (2)

where, c_s represents the instance segmentation confidence, e_p denotes the inter-frame prediction position residual, and η is a motion stability adjustment coefficient. In addition, a temporal smoothing mechanism across consecutive frames is incorporated, in which a sliding average filter is applied to the fused bounding box coordinates. This process further reduces bounding box jitter during occlusion periods. Through this fusion-based tracking mechanism, stable and continuous identity association of teacher–student instances can be maintained under complex occlusion conditions. Experimental results demonstrate that the proposed approach achieves an 18% improvement in tracking accuracy compared to conventional methods, effectively addressing drift issues in long-duration video sequences.

2.3 Three-dimensional complementary visual feature extraction

Seventeen humans skeletal keypoints are extracted using the lightweight high-resolution network, and three categories of geometrically quantified pose features are specifically designed to characterize interaction behaviors in English classroom settings. Through this design, interaction intent is represented in a continuous numerical form. The hand-raising height is computed using a normalization strategy relative to the torso height, defined as:

$h=\frac{y_{\text {wrist }}-y_{\text {hip }}}{y_{\text {head }}-y_{\text {hip }}}$   (3)

The limb pointing angle is determined based on the spatial vector formed by shoulder and wrist keypoints, while torso orientation is calculated as the angle between the torso midline and the vertical axis of the image plane. All geometric features are normalized to the range [0, 1]. Finally, these features are integrated into an 8-dimensional normalized pose feature vector, which enables accurate representation of typical classroom interaction behaviors, such as student hand-raising and teacher gestural guidance.

To address the issue of low facial resolution in long-distance classroom scenarios, an ESPCN-based super-resolution model is employed to reconstruct cropped facial regions to a resolution of 128 × 128. Subsequently, 59-dimensional facial texture features are extracted using local binary patterns. The final two convolutional layers and the fully connected structure of EfficientNet-B0 are fine-tuned, and expression regression is performed using mean squared error loss. As a result, two-dimensional facial expression features—smile intensity and facial attention level—are obtained. For spatial proximity feature modeling, adaptive intersection-over-union thresholds are defined based on classroom interaction patterns. Specifically, a threshold of 0.1 is assigned for teacher–student interactions, while a threshold of 0.05 is applied for student–student interactions. In addition, the Euclidean distance between the centroids of pairwise instances is computed. By integrating spatial overlap relationships and inter-instance distances, a 17-dimensional spatial interaction feature vector is constructed, enabling quantitative characterization of spatial interaction intensity among individuals.

To ensure feature consistency, pose, facial expression, and spatial proximity features are individually normalized using Z-score standardization to eliminate dimensional disparities. These normalized features are then concatenated sequentially to form a unified 27-dimensional interaction intensity feature vector. This three-dimensional complementary feature representation overcomes the limitations of single-modality visual features by jointly modeling interaction cues from three perspectives: bodily action intent, emotional and attentional states, and inter-individual spatial relationships. Experimental validation demonstrates that the proposed feature representation improves discriminative capability by 25% compared to conventional methods, thereby providing highly informative and comprehensive input features for subsequent interaction event detection.

2.4 Design of the spatiotemporal graph convolutional interaction network

To accurately model cross-individual interaction relationships between teachers and students in English classroom settings, a specialized spatiotemporal graph convolutional interaction network is developed. The core innovation lies in the construction of a graph structure that integrates individual skeletal features with cross-individual interaction relationships. The topology of the proposed network is illustrated in Figure 2. Graph nodes are defined by the human skeletal keypoints of all instances within the classroom. Each instance consists of 17 skeletal keypoints. Under a typical classroom configuration comprising one teacher and sixteen students, the total number of graph nodes is 17 × 17. The initial feature vector of each node is constructed by integrating keypoint coordinates, optical flow velocity, and instance identity encoding, thereby providing a comprehensive representation of both motion states and identity information. The graph structure incorporates two types of edges. First, skeletal spatial edges are established based on the physiological connections of the human skeleton, enabling the modeling of intra-individual body motion characteristics. Second, cross-individual interaction edges are introduced to explicitly represent teacher–student interactions. These edges connect the teacher’s wrist and torso center to the head and torso center of each student, allowing direct modeling of interaction relationships across individuals. The adjacency matrix is constructed based on a dual constraint mechanism involving spatial distance and interaction relevance. Specifically, if two nodes satisfy a predefined spatial distance threshold or are connected via the defined interaction edges, the corresponding adjacency matrix element is set to 1; otherwise, it is set to 0. Through this design, interaction relationships are precisely quantified.

Figure 2. Topological structure and feature update mechanism of the Spatiotemporal Graph Convolutional Interaction Network (ST-GCIN)

The spatiotemporal graph convolutional interaction network adopts a four-layer spatiotemporal convolutional architecture to achieve deep integration of spatial interaction features and temporal sequence features. The spatial graph convolution layer updates feature according to the following formulation:

$H^{(l+1)}=\sigma\left(\widetilde{D}^{-1 / 2} \widetilde{A} \widetilde{D}^{-1 / 2} H^{(l)} W^{(l)}\right)$   (4)

where, $H^{(l)}$ denotes the node feature matrix at the $l$-th layer; $\widetilde{A}$ represents the normalized adjacency matrix; $\widetilde{D}$ is the corresponding degree matrix of $\widetilde{A} ; W^{(l)}$ is the learnable weight matrix; and σ denotes the rectified linear unit activation function. In the temporal dimension, a one-dimensional convolutional layer is employed to extract temporal features, with a kernel size of 3 and a stride of 1. This configuration is designed to capture short-term temporal dependencies characteristic of classroom interaction events, thereby enabling effective fusion of interaction features across consecutive frames. Each spatiotemporal convolutional layer is followed by batch normalization and rectified linear unit activation to mitigate gradient vanishing and improve training stability and convergence speed.

To enhance detection accuracy, address class imbalance, and ensure real-time performance, both the training strategy and model architecture are further optimized. An online hard example mining strategy is adopted, in which training samples are ranked according to loss values, and the top 20% of high-loss samples are selected for focused training. This approach strengthens the network’s ability to recognize complex interaction events. The cross-entropy loss function is improved by introducing class weighting coefficients, which are dynamically assigned based on the proportion of samples in each interaction category, thereby alleviating the issue of missed detection in minority classes. For model lightweighting, depthwise separable convolution is utilized to replace conventional convolutional layers, reducing both the number of parameters and computational complexity. As a result, real-time processing performance is maintained at 32 frames per second. Experimental results demonstrate that the spatiotemporal graph convolutional interaction network model achieves a mean average precision of 0.87, enabling accurate detection of diverse classroom interaction events. This approach effectively overcomes the limitation of conventional graph convolutional networks, which primarily focus on intra-individual skeletal features and fail to capture cross-individual interaction relationships.

2.5 Continuous interaction frequency quantification based on wavelet energy spectrum

To overcome the limitations of conventional interaction frequency quantification methods, which rely on event counting and fail to capture subtle non-verbal interactions while lacking temporal continuity, an interaction energy density model is constructed by integrating pixel-level optical flow energy with joint angular velocity. This formulation enables fine-grained characterization of interaction intensity. Figure 3 illustrates the wavelet decomposition–based analysis of classroom interaction energy spectrum and frequency quantification. Dense motion information between consecutive video frames is extracted using the Farnebäck optical flow algorithm, with a window size of 15 and a pyramid level of 3, thereby achieving a balance between computational accuracy and real-time performance. The frame-level energy E_t is obtained by summing the magnitudes of optical flow vectors. In parallel, skeletal keypoint coordinates extracted via a lightweight high-resolution network are utilized to compute joint angular velocities ω. The average absolute value of joint angular velocities is used as an indicator of joint motion intensity. The interaction energy density D_t is derived through weighted fusion, expressed as:

$D_t=\alpha \cdot E_t+\beta \cdot \frac{1}{J} \sum_{j=1}^J\left|\omega_j\right|$   (5)

where, J = 17 represents the total number of skeletal keypoints. The weighting coefficients are set to α = 0.6 and β = 0.4, determined via five-fold cross-validation. This configuration ensures a balanced contribution between global frame-level interaction and local joint-level motion, thereby achieving optimal linear correlation with manually annotated interaction intensity.

Figure 3. Classroom interaction energy spectrum analysis and frequency quantification based on wavelet decomposition

A four-level discrete wavelet decomposition is performed on the interaction energy density sequence using the Daubechies-4 wavelet. This process yields four levels of detail components (d₁-d₄) and one approximation component. The high-frequency detail components at levels 1–2 correspond to rapid and subtle non-verbal interactions, such as hand-raising and eye contact, whereas the low-frequency detail components at levels 3–4 correspond to slower and larger-scale interactions, including group discussions and teacher instruction. The approximation component captures the overall trend of interaction energy. The interaction energy spectrum is then computed by calculating the proportion of energy in each detail component, defined as the ratio of the squared magnitude of each component to the total energy of all detail components. Through this formulation, energy contributions from different types of interactions are effectively distinguished.

To achieve continuous quantification and precise representation of interaction frequency, a composite frequency indicator F is constructed by integrating event counting with the wavelet energy spectrum. The formulation is given as:

$F=\frac{N_{\text {event }}}{T_{\text {total }}} \cdot \gamma+\frac{\sum_{l=1}^2\left|d_l\right|^2}{\sum_{l=1}^4\left|d_l\right|^2}(1-\gamma)$   (6)

where, $N_{\text {event }} / T_{\text {total }}$ denotes the interaction event frequency per unit time, and $\sum_{l=1}^2\left\|d_l\right\|^2 / \sum_{l=1}^4\left\|d_l\right\|^2$ represents the proportion of high-frequency interaction energy. The weighting coefficient γ=0.7 is determined through grid search optimization, ensuring a balance between the accuracy of event counting and the sensitivity to subtle interactions. A sliding window approach with a duration of 1 minute is employed for frequency computation, with a step size of 10 seconds. This configuration ensures continuity of the frequency sequence while aligning with the temporal scale of classroom interactions, thereby reducing the influence of transient fluctuations on the quantification results. Experimental results demonstrate that the proposed quantification method achieves a correlation coefficient of 0.92 with manually annotated interaction frequency. Compared with conventional counting-based methods, sensitivity is improved by 23%, enabling effective detection of diverse classroom interactions, particularly subtle non-verbal interactions.

2.6 Interpretable dynamic teaching quality evaluation model

To achieve dynamic and interpretable quantification of teaching quality, while addressing evaluation bias caused by variations in instructional pacing across different teachers, an evaluation-oriented feature engineering framework is first constructed. Four core features are selected to form a 4-dimensional segment-level feature vector: interaction frequency, average interaction intensity, spatial distribution entropy, and average facial expression metrics. Specifically, interaction frequency is defined as the mean value of the continuous indicator derived from wavelet energy spectrum–based quantification. The average interaction intensity is computed as the temporal mean of the interaction energy density sequence. Spatial distribution entropy is calculated based on the centroid coordinates of teacher–student instances, providing a measure of the uniformity of interaction distribution. The average facial expression metric is obtained by averaging student-level smile intensity and attention level across the entire classroom. A single 45-minute English classroom session is partitioned into 45 temporal segments at one-minute intervals. For each segment, a 4-dimensional feature vector is extracted, resulting in a 45×4 temporal feature matrix. This structured representation serves as the input for subsequent temporal alignment and deep learning–based modeling.

To address temporal misalignment caused by variations in instructional pacing, dynamic time warping is introduced to align the evaluated temporal sequence with a standardized teaching template. Based on expert knowledge in the educational domain, a standard instructional workflow template is constructed, consisting of five stages: introduction, presentation, practice, consolidation, and summary. The feature distributions and temporal proportions of each stage are determined through expert annotation. An optimal temporal mapping between the evaluated feature sequence and the standard template is obtained using a dynamic programming algorithm, in which the cumulative distance between the two sequences is minimized. After alignment, linear interpolation is applied, and the sequence is normalized into a unified 45 × 4 feature matrix. Through this process, evaluation errors induced by differences in teaching pace are effectively eliminated. On this basis, an attention-based bidirectional long short-term memory network is designed for teaching quality prediction. The network is configured with 128 hidden units, accommodating the 4-dimensional feature input and 45-step temporal sequence, thereby balancing model fitting capability and the risk of overfitting. The attention weights are computed as follows:

$\alpha_t=\operatorname{softmax}\left(v^T \tanh \left(Wh_t+b\right)\right)$   (7)

where, h_t denotes the hidden state of the long short-term memory network at time step t; W and v are learnable weight matrices; and b is the bias term. The attention mechanism dynamically assigns weights to different temporal segments, enabling the model to focus on critical instructional stages such as presentation and practice. The output layer consists of a fully connected layer, and a regularization strategy with Dropout = 0.5 is applied to suppress overfitting. The complete interpretable teaching quality evaluation framework is illustrated in Figure 4.

Figure 4. Interpretable teaching quality evaluation model based on dynamic time warping temporal alignment and attention heatmap

To further enhance interpretability and training stability, a composite loss function is constructed by combining mean squared error loss with Kullback–Leibler divergence regularization. The mean squared error loss minimizes the discrepancy between predicted scores and manually annotated scores, while the Kullback–Leibler divergence regularization constrains the distribution of attention weights, encouraging the model to focus on pedagogically meaningful stages and thereby improving interpretability. Model training is performed using the Adam optimizer, with a learning rate of 0.0001 and a batch size of 8. The loss function was optimized to convergence through gradient descent. Model interpretability is validated through attention heatmap visualization, which provides an intuitive representation of attention weight distribution across different instructional stages and clarifies the underlying mechanism of the evaluation results. Experimental results indicate that the proposed evaluation model achieves a mean absolute error of 0.31 and a Pearson correlation coefficient of 0.87 with manually annotated scores. The approach effectively mitigates evaluation bias caused by instructional pacing differences and significantly enhances model interpretability, thereby satisfying the reliability requirements of educational evaluation scenarios.

2.7 System integration strategy

An offline cascaded execution mechanism is adopted for the proposed classroom interaction frequency detection and dynamic teaching quality evaluation system in English instruction. All functional modules are sequentially connected and collaboratively optimized according to the data processing pipeline, enabling a dynamic balance between real-time performance and accuracy, and ensuring adaptability to real-world classroom environments. The video preprocessing module is first applied to remove illumination variations and shadow interference from the input video, generating standardized video frames. These frames are directly transmitted to the teacher–student instance segmentation and tracking module, where instance bounding boxes, identity information, and skeletal keypoint coordinates are simultaneously produced. These outputs are then forwarded to the three-dimensional complementary visual feature extraction module, providing precise spatial localization and motion information for feature extraction. The extracted 27-dimensional interaction intensity vectors are subsequently fed into both the spatiotemporal graph convolutional interaction network and the wavelet energy spectrum–based quantification module. The spatiotemporal graph convolutional interaction network performs interaction event detection and outputs event counting results, which are further utilized by the frequency quantification module. In parallel, the wavelet-based module integrates event counts with energy spectrum features to generate a continuous interaction frequency sequence. These outputs collectively form the input features for the teaching quality evaluation module.

Within the evaluation module, temporal alignment is performed using dynamic time warping, followed by teaching quality prediction via an attention-based bidirectional long short-term memory network. Lightweight data interaction protocols are employed between modules to minimize data redundancy and transmission latency. In addition, cross-module parameter co-optimization is implemented, enabling coordinated adjustment of instance tracking stability, feature discriminability, and network inference efficiency. Through this integrated design, both the independent accuracy of each module and the overall system-level efficiency are ensured. The system maintains a processing speed of 32 frames per second while satisfying the performance requirements of all core tasks, thereby demonstrating strong potential for practical deployment in real classroom environments.

The primary advantage of the proposed framework for classroom interaction frequency detection and dynamic teaching quality evaluation in English instruction lies in the construction of an end-to-end optimized solution that directly addresses key technical limitations in existing studies. Through the coordinated integration of multiple innovations, substantial improvements in accuracy, real-time performance, and scenario adaptability are achieved. The improved teacher–student instance segmentation and tracking strategy, which incorporates feature pyramid enhancement and Kalman filtering–based adaptive fusion, effectively mitigates tracking drift caused by illumination variation and occlusion in classroom environments. When this component is removed, the F1-score and the mean average precision decrease by 8.2% and 7.5%, respectively, indicating that stability at the image processing level is significantly enhanced and provides a reliable foundation for subsequent feature extraction and interaction event detection. The three-dimensional complementary visual feature extraction system overcomes the limitations of single-modality representations by jointly modeling pose, facial expression, and spatial relationships. When this component is removed, the F1-score and the mean average precision for interaction event detection decrease to 0.76 and 0.73, respectively, demonstrating that comprehensive feature representation is essential for capturing subtle non-verbal interactions that are often missed by conventional methods.

The spatiotemporal graph convolutional interaction network further extends traditional graph convolutional approaches by introducing cross-individual relational edges, thereby enabling direct modeling of teacher–student interaction relationships. Combined with spatiotemporal feature fusion, an F1-score of 0.89 is achieved for interaction event detection, representing a substantial improvement over conventional graph-based methods that are limited to intra-individual skeletal modeling. The wavelet energy spectrum–based quantification method enables continuous representation of interaction frequency and effectively captures subtle non-verbal interaction patterns, achieving a correlation coefficient of 0.92. This approach addresses the inherent limitation of traditional counting-based methods, which lack temporal continuity. The interpretable teaching quality evaluation model integrates dynamic time warping for temporal alignment with an attention-based mechanism, thereby effectively reducing evaluation bias caused by differences in instructional pacing. A mean absolute error of 0.31 is achieved, while interpretability is enhanced through attention heatmap visualization. This design overcomes the “black-box” limitation of existing evaluation models. Compared with existing approaches, the principal contribution of the proposed framework lies in the deep integration of video image processing techniques with the specific characteristics of English classroom environments. Through dedicated network design, comprehensive feature representation, and innovative quantification strategies, a highly targeted technical pathway is established, facilitating the precise application of image processing techniques in the domain of intelligent education.

Despite the demonstrated effectiveness, several limitations remain and should be critically examined. From the dataset perspective, the current dataset is limited to junior high school English classroom scenarios and has not been extended to other educational stages, such as primary or senior high school, nor to other subject domains. This limitation arises primarily from the substantial variation in interaction patterns across educational levels. Classroom interactions in primary education are typically more diverse and less structured, whereas those in senior high school are more logically organized and task-oriented. These differences increase the difficulty of data acquisition and annotation, thereby constraining dataset generalizability. With respect to tracking performance, although robustness is achieved under moderate conditions, further improvement is required in extreme occlusion scenarios, such as dense multi-person overlap or prolonged occlusion. This limitation is mainly attributed to the loss of skeletal keypoints under severe occlusion, which reduces the accuracy of both segmentation bounding boxes and motion prediction, thereby hindering stable identity association. In terms of model efficiency, the degree of lightweight design remains insufficient. Although the spatiotemporal graph convolutional interaction network and the attention-based bidirectional long short-term memory achieve real-time performance on personal computer platforms, deployment on resource-constrained edge devices remains challenging due to the relatively large number of model parameters. This issue arises from the prioritization of detection accuracy and interpretability during model design, where lightweight optimization was not treated as a primary objective. These limitations reflect the trade-offs between performance optimization and scenario complexity, and they provide clear directions for future research.

Building upon the current framework and considering emerging trends in video image processing, future work will focus on improving generalizability, robustness, and practical applicability. To address dataset limitations, the dataset will be expanded to include multiple educational stages (primary and senior high school) and multiple subjects (e.g., English and mathematics), thereby constructing a more generalizable classroom interaction dataset. In addition, data augmentation techniques will be incorporated to enhance scenario adaptability. To address extreme occlusion and model efficiency challenges, hybrid architectures combining Transformer models with graph convolutional networks will be explored. The global attention mechanism of Transformers is expected to improve feature matching under occlusion conditions. Meanwhile, lightweight neural network design strategies, including knowledge distillation and model pruning, will be employed to reduce model complexity and enable real-time deployment on edge devices. To overcome the limitations of single-camera viewpoints, multi-camera fusion techniques will be introduced. Through multi-view image stitching and feature fusion, full classroom coverage can be achieved, thereby reducing missed detections caused by blind spots. Furthermore, natural language processing techniques will be integrated to incorporate speech-based features, such as teacher–student dialogue content and interaction tone. By constructing a multimodal interaction feature representation, further improvements in interaction event detection accuracy are anticipated. Overall, future developments are expected to advance intelligent educational assessment toward multimodal integration, lightweight deployment, and enhanced generalization capability, thereby providing more comprehensive technical support for classroom teaching evaluation.