Joint Optimization of Multi-Object Detection and Fine-Grained Image Representation Learning in English Education Scenarios

As the digital transformation of education continues to advance [1-3], intelligent teaching systems have progressively shifted from experience-driven subjective pedagogical evaluation toward data-driven intelligent analysis of learning conditions. Classroom perception technologies grounded in computer vision [4-6] enable non-intrusive, real-time acquisition and quantitative analysis of teacher-student behaviors and instructional states, constituting a core technological foundation for supporting smart English teaching reform and enabling precise pedagogical interventions. English language classrooms [7-9] are characterized by dense target distributions, multidimensional behavioral granularity, and subtle inter-class differences—typical scenario features that demand not only accurate spatial localization of multiple object categories—including teachers, students, and instructional media—but also effective discrimination among highly similar learning and interactive behaviors, thereby achieving fine-grained behavioral state recognition [10]. From the perspective of visual cognitive mechanisms, multi-object detection relies on global semantic and spatial structural information for target localization, whereas fine-grained recognition depends on local texture and edge details for precise categorization; the feature representations of these two tasks exhibit inherent complementarity [11-13]. Existing approaches, however, predominantly adopt task cascading or simple backbone parameter sharing, which fails to achieve bidirectional deep interaction and adaptive synergy at the feature level. This limitation not only constrains the upper bound of localization accuracy in object detection but also impedes the full exploitation of the discriminative potential inherent in fine-grained representations. The construction of a multi-task joint optimization framework tailored to classroom scenarios [14-16] can address the practical challenges of fine-grained behavioral analysis in smart English education, enhance the intelligence of instructional state assessment, and simultaneously advance the theoretical foundations of feature alignment [17, 18], representation manifold construction, and joint optimization within the field of multi-task image understanding—thereby offering significant practical value and academic research significance.

Although existing fusion methods for multi-object detection and fine-grained recognition have achieved basic task coupling, fundamental deficiencies persist in feature interaction, representational constraints, and training optimization, rendering them inadequate for the complex visual task demands of English classrooms—characterized by high granularity, strong similarity, and multi-scale variability. In terms of feature interaction mechanisms [19, 20], mainstream approaches predominantly employ unidirectional feature injection and fixed-channel concatenation as the primary fusion forms, resulting in statically fixed feature interaction patterns that cannot adaptively adjust the fusion ratio between semantic and detailed features according to target scale size or texture information richness. Large-scale instructional targets fail to obtain sufficient detailed features to support fine-grained discrimination, whereas small-scale local behavioral targets are susceptible to fine-grained noise interference, which degrades localization stability. Consequently, the bidirectional dynamically adaptive synergistic interaction capability is absent. In fine-grained representation learning, traditional contrastive learning methods construct sample-wise constraint relationships only at a single semantic hierarchy [21], neglecting the inherent hierarchical subordinate properties of visual semantics. Such approaches fail to align with the multi-level semantic logic of classroom scenes—spanning from global scene context, to target subjects, and down to fine-grained behaviors—readily leading to blurred feature boundaries among similar behavioral subcategories. Concurrently, the fine-grained representation training process lacks upper-level semantic constraint guidance, rendering it prone to semantic drift, which results in insufficient self-consistency between classification outcomes and global scene semantics. At the multi-task optimization level [22], existing joint training paradigms commonly adopt fixed-weight loss fusion strategies, where parameter updates of the shared backbone network are susceptible to interference from adversarial gradients originating from the two tasks, inducing training oscillation and task performance degradation. Current gradient correction methods can only passively resolve local gradient conflicts, fail to form a closed-loop optimization mechanism coupled with the forward feature learning process, and do not conform to the coarse-to-fine progressive learning patterns inherent in visual cognition—thus fundamentally incapable of resolving the optimization imbalance problem in multi-task training.

To address the aforementioned technical deficiencies and scenario-specific challenges, an end-to-end collaborative localization-representation network is proposed, establishing a complete multi-task synergistic closed loop that encompasses feature interaction, representation shaping, and gradient optimization. A dynamic semantic-detail collaborative modulation module, a hierarchical semantic alignment contrastive learning module, and a curriculum-based gradient harmonization strategy are sequentially designed to tackle the core issues of multi-scale feature adaptive synergy, the absence of fine-grained semantic constraints, and multi-task gradient conflicts, respectively. Each innovative module is deeply coupled to form an integrated technical framework, which is comprehensively validated on a self-constructed English classroom dataset and general fine-grained datasets. The synchronized improvement of both detection and fine-grained recognition performance is effectively achieved, offering a novel technical paradigm and research direction for multi-task visual synergistic learning.

The overall organizational structure of this work is clear and coherent, with each chapter's research content progressing in a logical sequence. Chapter 2 elaborates on the overall architecture of the proposed collaborative localization-representation network, providing a complete derivation of the technical principles and implementation details for each core innovative module. Chapter 3 validates the effectiveness of the proposed method through multiple sets of comparative and ablation experiments, comprehensively analyzing model performance, representational capacity, and training stability. Chapter 4 objectively dissects the intrinsic mechanisms, existing limitations, and future extension directions of the method. Chapter 5 synthesizes the overall research findings and summarizes the core research conclusions and innovative contributions.

To comprehensively validate the effectiveness, robustness, and generalization capability of the proposed collaborative localization-representation network, systematic experimental validation was conducted from multiple dimensions, including experimental configuration, benchmark comparisons, module ablation studies, representation quality analysis, optimization mechanism verification, scenario robustness assessment, and computational efficiency evaluation. Through quantitative metric comparisons, visual analysis, and statistical validation, the performance gain mechanisms of each core module and the technical advantages of the overall framework were progressively elucidated. All experiments were conducted under identical hardware environments and hyperparameter configurations, ensuring the objectivity and comparability of the experimental results.

3.1 Experimental setup

The experiments were conducted using a self-constructed English classroom fine-grained behavior dataset as the primary evaluation dataset. The dataset was collected from real-world routine English teaching scenarios in primary and secondary schools, encompassing classroom images under varying illumination conditions, occupant densities, and instructional modes, with a total of 12,800 high-resolution images at a unified resolution of 1,920 × 1,080. A rigorous three-level semantic label system was constructed for the dataset, comprising 3 scene-level labels, 4 object-level labels, and 8 fine-grained behavior attribute labels, comprehensively covering mainstream teaching states and teacher-student behaviors in English classrooms. Professional image annotation tools were employed for dual annotation of bounding boxes and behavior attributes, with strict validation of annotation precision and semantic hierarchy self-consistency. The dataset was randomly partitioned into training, validation, and test sets at a ratio of 8:1:1, containing 10,240, 1,280, and 1,280 images, respectively. For generalization capability validation, two general fine-grained public datasets were selected—the CUB-200-2011 bird fine-grained dataset and the Stanford Cars vehicle fine-grained dataset—for cross-domain performance testing outside the educational context, verifying the model's general fine-grained representational capacity.

All experiments were implemented on the Ubuntu 20.04 operating system with the PyTorch 1.10 deep learning framework, with hardware configuration consisting of a single RTX 3090 graphics processing unit. Model training was performed using the AdamW optimizer, with the initial learning rate set to 1 × 10⁻³, the weight decay coefficient set to 5 × 10⁻⁴, the batch size set to 16, and the total number of training epochs set to 120. A cosine annealing learning rate decay strategy was adopted during training, complemented by data augmentation techniques, including random horizontal flipping, random cropping, color perturbation, and Gaussian blurring, to enhance model adaptability to diverse scenarios.

3.2 Benchmark method comparison experiments

In this section, the proposed collaborative localization-representation network and all baseline models were uniformly evaluated on the self-constructed English classroom dataset, with core performance metrics for multi-object detection and fine-grained classification compared simultaneously to quantitatively verify the overall performance advantages of the proposed framework. The experimental results are presented in Table 1.

Table 1. Overall performance comparison of baseline models and the proposed framework

From the experimental results, it is observed that the proposed collaborative localization-representation network significantly outperforms all baseline models across all evaluation metrics. Compared with Fusion-YOLO, the best-performing multi-task baseline model, the proposed method achieves improvements of 4.72% in mAP@0.5, 4.37% in mAP@0.5:0.95, and 3.52% in recall, demonstrating that the bidirectional feature synergy mechanism effectively enhances the localization accuracy of multi-scale classroom targets. For fine-grained discrimination tasks, the proposed framework achieves improvements of 4.52% in Top-1 accuracy and 4.89% in Macro F1 score, with even more pronounced advantages.

Conventional single-task models are capable of performing only detection or classification tasks in isolation, without achieving dual-task joint optimization. General-purpose multi-task models, which rely solely on simple feature sharing and fixed-weight optimization, are inadequate for fine-grained behavioral scenarios with high similarity in English classrooms. Through dynamic feature synergy, hierarchical semantic constraints, and gradient harmonization optimization, the proposed method breaks through the performance bottlenecks of both detection accuracy and fine-grained recognition accuracy, achieving simultaneous gains across the two core tasks. For confusable behaviors such as head-down writing versus head-down reading, and hand-raising for questioning versus hand-raising for signaling, the classification error rate of the proposed collaborative localization-representation network is reduced by an average of 12.3% compared with the baseline models, fully validating the capability of hierarchical contrastive representation learning in capturing subtle behavioral differences.

3.3 Module ablation experiments

To verify the independent contributions and synergistic gains of the three core modules—the dynamic semantic-detail collaborative modulation module, the hierarchical semantic alignment contrastive representation learning module, and the curriculum-based gradient harmonization module—controlled-variable ablation experiments were conducted based on a unified baseline model, with the effectiveness of each module and its subcomponents validated progressively. The overall module ablation experiments were performed using the basic multi-task shared backbone model as the baseline, with each innovative module incrementally added. The experimental results are presented in Figure 4.

tu_pian_4.png

tu_pian_5.png

Figure 5. The ablation experimental results of the submodules of the dynamic semantic-detail collaborative modulation module

The experimental results demonstrate that each of the three modules independently yields positive performance gains for the model, and their combined usage exhibits significant synergistic effects. The dynamic semantic-detail collaborative modulation module contributes most significantly to detection tasks, effectively addressing the multi-scale target feature adaptation problem and substantially improving target localization accuracy. The hierarchical semantic alignment contrastive representation learning module focuses on optimizing fine-grained representational capability, notably reducing classification errors for similar behaviors. The curriculum-based gradient harmonization module, through resolving multi-task optimization conflicts, simultaneously enhances the training stability and final accuracy of both tasks. The performance gain achieved by the integration of the three modules is substantially higher than the sum of individual module gains, demonstrating that forward feature synergy, hierarchical representation shaping, and backward gradient optimization form a complete closed-loop optimization system, with strong coupled complementarity among the modules.

Ablation validation was conducted on the core components of the dynamic semantic-detail collaborative modulation module, comparing model performance under unidirectional feature interaction, fixed fusion coefficients, and ordinary convolution-based alignment. The results are presented in Figure 5.

From the data, it is observed that the bidirectional feature interaction mechanism effectively improves the performance of both tasks compared with unidirectional information injection, demonstrating the necessity of cross-branch bidirectional synergy. After the dynamic routing coefficient replaces the fixed weight, the model adaptively adapts to classroom targets of varying scales and texture characteristics, further alleviating the problems of small-target noise interference and large-target detail deficiency. The deformable feature alignment operation completely resolves the spatial misalignment defect inherent in conventional convolution-based features, achieving pixel-level precise feature fusion, which constitutes the core key to the module's performance gains.

To verify the roles of hierarchical contrastive learning, cross-level consistency constraints, and dynamic hard negative sample mining, submodule ablation experiments were conducted, with the results presented in Table 2.

Table 2. Ablation experimental results of the submodules of the hierarchical semantic alignment contrastive representation learning module

Experimental results indicate that traditional single-level contrastive learning fails to adapt to the semantic hierarchical structure of the classroom, with fine-grained representations suffering from severe semantic drift and high inter-class confusion. After the introduction of hierarchical contrastive learning, the model's multi-granularity semantic representational capability is significantly enhanced, and the discriminative ability for fine-grained behaviors is strengthened. The cross-level consistency constraint effectively resolves the semantic conflicts inherent in multi-level learning, ensuring the self-consistency of the representational system. The dynamic hard negative mining mechanism specifically optimizes the feature distances of confusable samples, substantially reducing the proportion of inter-class misclassifications, and constitutes the core component for improving fine-grained recognition accuracy.

Ablation analysis was conducted on the three core components of the curriculum-based gradient harmonization strategy—gradient projection, dynamic weighting, and relaxation factor—with the results presented in Table 3.

Table 3. Ablation experimental results of the curriculum-based gradient harmonization strategy

A gradient cosine similarity value closer to 0 indicates weaker task gradient conflicts. Experimental results demonstrate that under the fixed-weighting training mode, gradient adversariality is strongest, with severe training oscillation. Dynamic weight scheduling preliminarily alleviates the optimization imbalance problem, while the gradient projection mechanism substantially resolves gradient conflicts in the shared backbone parameters. After the introduction of the temporal relaxation factor, the model follows the curriculum learning patterns of localization-first and refinement-second, completely resolving the problems of training oscillation and performance degradation, while maximizing the synergistic performance of the dual tasks alongside training stability.

3.4 Visualization experimental validation

To validate the joint processing capability of the collaborative localization-representation network for multi-object localization and fine-grained behavioral representation in real English classroom images, a visualization experiment on method implementation effectiveness is designed. Figure 6 presents the actual operational process of the model from five perspectives: original classroom images, results of the baseline method, collaborative detection results of the collaborative localization-representation network, local magnification of hard samples, and hierarchical semantic output. It is observed that the original English classroom scenes simultaneously contain multiple object categories—including teachers, students, screens, laptops, and textbooks—with student behaviors densely distributed in space, and states such as head-down reading, head-down writing, listening, and interacting exhibiting pronounced visual similarity. Although the baseline method is capable of producing basic bounding box annotations, it still suffers from localization instability and semantic confusion for distant students, small-scale learning media, and similar learning behaviors. In contrast, the collaborative localization-representation network more completely identifies teachers, students, screens, and learning tools within the same scene, and further outputs fine-grained behavioral labels such as reading, writing, and interacting. In conjunction with the quantitative results, the collaborative localization-representation network achieves an mAP@0.5 of 91.84%, an mAP@0.5:0.95 of 60.75%, a recall of 92.48%, and Top-1 accuracy and macro F1 scores of 92.67% and 91.92%, respectively, representing improvements of 4.72, 4.37, 3.52, 4.52, and 4.89 percentage points over the optimal multi-task baseline, Fusion-YOLO. These results demonstrate that the proposed method does not simply superimpose detection and classification tasks, but rather, through dynamic semantic-detail collaborative modulation, hierarchical semantic constraints, and gradient harmonization optimization, endows the detection branch with more stable spatial localization capability and the representation branch with stronger fine-grained discriminative ability, thereby effectively supporting intelligent visual analysis in complex English classroom scenarios.

tu_pian_6.png

Figure 6. Visualization of collaborative detection and fine-grained representation implementation by the proposed collaborative localization-representation network in English classroom scenarios

To further examine the fine-grained discriminative capability of the collaborative localization-representation network for confusable behaviors in English classrooms, localized visualization comparative experiments are conducted on three sets of typical hard samples: head-down reading versus head-down writing, hand-raising for questioning versus hand-raising for signaling, and seated listening versus passive observation. As shown in Figure 7, the prediction confidence of the baseline method on the six localized samples is predominantly concentrated in the range of 0.55 to 0.63, with an average of approximately 0.60, accompanied by errors such as misclassifying writing as reading, hand-raising for signaling as questioning, and passive observation as listening. The collaborative localization-representation network, on the same samples, elevates the prediction confidence to the range of 0.90 to 0.96, with an average of approximately 0.93, and all six samples are assigned behavior categories consistent with the localized visual evidence. This phenomenon is consistent with the results of the dedicated representation experiments: the error rate for head-down reading versus head-down writing is reduced to 3.15%, for hand-raising questioning versus hand-raising signaling to 2.84%, and for seated listening versus passive observation to 2.31%. Concurrently, the collaborative localization-representation network achieves an average intra-class distance of 0.213 and an average inter-class distance of 0.894, with a feature discriminability of 4.197, significantly outperforming Fusion-YOLO (1.437) and Simple Framework for Contrastive Learning of Visual Representations (SimCLR) (1.741). Thus, the proposed method is able to shift the discriminative basis from coarse human silhouettes to critical details such as hand gestures, head orientation, learning media contact relationships, and classroom interaction states, substantially compressing intra-class variations while enlarging inter-class feature margins. This demonstrates that the proposed joint optimization framework effectively addresses the problems of blurred fine-grained behavior boundaries and semantic drift in English education scenarios.

tu_pian_7.png

Figure 7. Visualization of fine-grained representation enhancement for confusable classroom behaviors

3.5 Specialized validation of fine-grained representation capability

To intuitively and quantitatively validate the fine-grained representation advantages of the proposed method, specialized validation experiments were conducted in this section from three dimensions: confusion statistics, visualization of quantitative feature metrics, and feature distance distribution analysis. From the classification results for confusable behaviors presented in Figure 8, it is observed that existing mainstream models exhibit extremely high discrimination errors for subtle behavioral differences in the classroom, with frequent behavioral misclassifications. The proposed collaborative localization-representation network, through hierarchical semantic constraints and hard sample mining mechanisms, substantially reduces the confusion errors across various similar behaviors. For the most easily confusable behaviors—head-down actions and hand-raising actions—the recognition error rate reduction exceeds 10%, fully demonstrating the module's precise capability in capturing subtle texture and edge differences.

To quantify the quality of feature clustering, intra-class feature distances and inter-class feature distances were computed for different models, with the compactness and discriminability results presented in Table 4. A smaller intra-class distance indicates more compact feature clustering, while a larger inter-class distance indicates stronger class discriminability.

tu_pian_8.png

Figure 8. Classification error rate comparison for confusable behavior categories (%)

Table 4. Quantitative metrics of feature space distribution

The quantitative results demonstrate that, compared with the mainstream baseline models, the proposed method achieves significantly reduced intra-class feature distances and substantially increased inter-class feature distances, with feature discriminability improved by nearly 2.4 times. Combined with the t-distributed Stochastic Neighbor Embedding (t-SNE) visualization results, it is observed that the collaborative localization-representation network produces feature clusters with clear boundaries, free from class overlap or discrete noise, validating the regularization and shaping effect of the hierarchical semantic manifold on the feature space and fundamentally enhancing the discriminative capability of fine-grained representations.

3.6 Validation experiments on optimization mechanism effectiveness

The optimization mechanism of the curriculum-based gradient harmonization strategy was validated. The superiority of the proposed optimization strategy was demonstrated through statistical analysis of gradient conflict intensity throughout the entire training process, loss convergence stability, and multi-seed robustness.

Table 5. Training process gradient and convergence stability statistics

From the training statistics presented in Table 5, it is observed that conventional multi-task training suffers from significant gradient conflict problems, with severe loss oscillation and slow model convergence. Existing gradient correction methods are only capable of passively eliminating partial gradient conflicts, without adapting to the dynamic training progression. Through gradient orthogonal projection and curriculum-based weight scheduling, the proposed curriculum-based gradient harmonization strategy drives the gradient cosine similarity toward zero, essentially eliminating multi-task gradient conflicts. The loss oscillation amplitude is substantially reduced, and the model convergence speed is significantly accelerated, with training efficiency and stability comprehensively improved.

To verify the training robustness of the model, five different random seeds were employed for model initialization, with the mean and variance of the final performance computed. The results are presented in Table 6.

Table 6. Performance stability validation across multiple random seeds (Top-1 accuracy, %)

The repeated experiments across multiple groups demonstrate that the conventional model, affected by gradient conflicts, exhibits substantial performance fluctuation under different initialization conditions, with poor robustness. The proposed model achieves a performance variance of only 0.08, demonstrating extremely strong training stability and initialization robustness, effectively avoiding the performance uncertainty problem inherent in multi-task joint training.

3.7 Robustness and generalization experiments

To simulate complex interference scenarios in real English classrooms, three test subsets were constructed—illumination variation, motion blur, and object occlusion—with the performance degradation rates of different models statistically computed. The results are presented in Table 7.

Table 7. Performance degradation rate comparison under complex interference scenarios (%)

The experimental results demonstrate that various interference factors in real classroom settings significantly degrade the detection and recognition accuracy of models, with baseline models exhibiting weak anti-interference capability. Through multi-scale feature adaptive synergy and highly discriminative fine-grained representations, the proposed method effectively resists scene noise such as illumination variation, blur, and occlusion. The overall performance degradation rate is substantially lower than that of mainstream baseline models, demonstrating excellent adaptability to complex scenarios and aligning with the practical deployment requirements of real-world smart classroom applications.

To validate the general representational capability of the proposed method, generalization tests were conducted on public fine-grained datasets, with the results presented in Figure 9.

tu_pian_9.png

Figure 9. Generalization performance comparison on public datasets (Top-1 accuracy, %)

On public fine-grained datasets, the collaborative localization-representation network consistently outperforms mainstream fine-grained recognition models, demonstrating that the proposed hierarchical semantic alignment and multi-task joint optimization mechanisms are not limited to English education scenarios but possess general-purpose fine-grained image understanding capability, widely adaptable to various fine-grained visual recognition tasks.

To validate the deployment feasibility of the model, the parameter count, computational cost, and inference speed of different models were statistically computed, with the results presented in Table 8.

Table 8. Model computational efficiency comparison

Compared with the baseline models, the proposed collaborative localization-representation network achieves substantial improvements in accuracy with only modest increases in parameter count and computational cost, and the per-image inference speed meets the application requirements of real-time classroom analysis. The overall computational overhead of the model is controllable, balancing recognition accuracy and deployment efficiency, and demonstrating strong potential for practical engineering deployment.

Through comprehensive multi-dimensional comparative experiments and ablation studies, the effectiveness of the proposed framework and each of its core modules was systematically validated. The dynamic feature synergy module effectively addresses the challenge of multi-scale target feature adaptation; the hierarchical semantic contrastive learning significantly enhances the discriminability of fine-grained confusable samples; and the curriculum-based gradient harmonization strategy thoroughly resolves multi-task optimization conflicts. The model exhibits excellent robustness in complex classroom scenarios, along with strong cross-domain generalization capability and deployment feasibility. The experimental results collectively demonstrate that the proposed multi-task joint optimization framework effectively achieves bidirectional gains in object detection and fine-grained image representation, providing reliable technical support for visual analysis in smart education classrooms.