Image Understanding and Information Extraction for English Education Scenarios via Multi-Scale Visual Feature Enhancement and Cross-Modal Semantic Constraints

2.1 Overall framework overview

An end-to-end integrated image understanding and information extraction framework tailored for English education scenarios is constructed, systematically incorporating the core functionalities of multi-scale visual feature enhancement, cross-modal semantic alignment, and multi-task collaborative optimization. The overall inference process of the framework follows a progressive optimization logic. Input English teaching images are first processed via an improved backbone network for multi-level foundational visual feature extraction. The acquired multi-scale features are subsequently fed into an adaptive gating fusion module for dynamic enhancement, thereby accommodating the inherent characteristic of substantial target-scale variation within teaching imagery. The scale-optimized visual features are then mapped into a unified semantic embedding space through a hierarchical cross-modal constraint mechanism, where dual-layer semantic alignment between global scenes and localized knowledge points is achieved by leveraging the domain prior knowledge embedded in the English education knowledge graph. Finally, teaching scene classification and knowledge-point recognition tasks are performed in parallel via the collaborative optimization module.

The proposed framework departs from the conventional paradigm of serial modular stacking in traditional vision models, enabling full-module feature sharing and gradient interplay through a unified semantic embedding space. During model training, the semantic alignment loss and multi-task supervision signals are back-propagated throughout the entire process to the feature extraction and feature fusion modules, establishing a closed-loop optimization mechanism characterized by positive reinforcement of visual features and corrective feedback from semantic constraints. Concurrently, the structured prior knowledge of the educational knowledge graph permeates the entire model pipeline. This prior knowledge can guide the dynamic weight allocation of multi-scale features, reinforcing the representational capacity of critical knowledge-point regions, while also regulating the output logic of dual tasks and constraining the model's semantic reasoning process. Ultimately, a deep integration of domain knowledge and visual perception is achieved, ensuring the specialized parsing performance of the model within English education scenarios.

2.2 Scene-adaptive dynamic multi-scale gating fusion module

To effectively accommodate the extreme scale distribution disparities inherent in English education imagery, a scene-adaptive dynamic multi-scale gating fusion module is designed, in which the fusion weights of visual features across different hierarchical levels are dynamically regulated by the global semantic information of the scene, thereby achieving refined enhancement of multi-scale features. Foundational feature extraction is performed based on the ConvNeXt-V2 network, with targeted optimizations applied to the network architecture in accordance with the visual characteristics of educational imagery. The network architecture of the scene-adaptive dynamic multi-scale gating fusion module is illustrated in Figure 1. Three-channel color images with an input size of $H \times W \times 3$ are fed into the model, from which feature maps with four distinct receptive fields—corresponding to downsampling factors of 4×, 8×, 16×, and 32×—are obtained via the backbone network's downsampling operations. These feature maps correspond respectively to fine-grained textural details and coarse-grained global semantic information within the imagery. To strengthen the model's capacity for capturing fine-grained knowledge points such as minute textual characters and word symbols, the convolutional kernel size in the network's lower-level convolutional modules is adjusted to 5 × 5, thereby preserving detail perception capability while reducing computational overhead. Concurrently, the global attention structures in the higher network layers are removed and replaced by a pooled global embedding generation mechanism, further improving inference efficiency while maintaining the quality of global semantic representation. Finally, feature maps from all hierarchical levels are uniformly mapped to a 256-dimensional channel space via 1 × 1 convolution operators, eliminating dimensional discrepancies and providing a unified representational foundation for cross-scale feature fusion.

1.png

Figure 1. Network architecture of the scene-adaptive dynamic multi-scale gating fusion module

To achieve scene-adaptive dynamic regulation of feature fusion weights, a scene semantic embedding representation is constructed based on high-level global features, with which the adaptive weighting of multi-scale features is guided by global contextual information. Global average pooling is first performed on the highest-level feature map to aggregate global semantic information across the entire image, followed by linear transformation to generate a fixed-dimensional scene semantic embedding vector, computed as follows:

$z=\operatorname{Linear}\left(G A P\left(F_4\right)\right)$                   (1)

where, $z$ denotes the global scene semantic embedding vector, GAP denotes the global average pooling operation, and $F_4$ denotes the highest-level semantic feature map. Based on this embedding vector, two parallel lightweight multilayer perceptron branches are constructed, with which the fusion weights and distribution modulation parameters corresponding to each hierarchical level are respectively generated:

$\alpha_i, \beta_i=\operatorname{Sigmoid}\left(M L P_i(z)\right), i \in\{1,2,3,4\}$                           (2)

where, $\alpha_i$ denotes the fusion weight coefficient for the $i$-th level features, used to regulate the overall contribution proportion of features at different scales, and $\beta_i$ denotes the feature distribution modulation bias, used to correct feature offset errors across different teaching scenarios. The parameter values are adaptively adjusted by the model according to the scene attributes of the input image: low-level detail feature weights are strengthened for text-dense scenes, whereas high-level semantic feature representations are enhanced for full-scene illustration scenes. Moreover, only minimal inference overhead is introduced by the lightweight branch structure, achieving a balance between accuracy and efficiency.

A coarse-to-fine progressive feature fusion strategy is adopted in this work, replacing the simple element-wise summation paradigm of traditional feature pyramids, thereby effectively avoiding semantic drift during multi-scale fusion. A stable coarse-grained scene representation base is first constructed based on high-level semantic features. The highest-level features are upsampled via bilinear interpolation to match the dimensions of the second-highest-level features, followed by residual enhancement to obtain the base scene features:

$F_{\text {coarse }}=F_3+\operatorname{Conv}\left(\operatorname{Up}\left(F_4\right)\right)$                     (3)

where, $F_{\text {coarse}}$ denotes the coarse-grained fused features, and Up( ) denotes the bilinear interpolation upsampling operation. Based on the coarse-grained features, all hierarchical features are uniformly upsampled to the smallest feature resolution, and weighted fusion is performed in conjunction with dynamic gating parameters. Finally, the enhanced multi-scale features are obtained via residual connection, which fuses the detailed features with the global base features:

$F_{\text {enh}}=\operatorname{Conv}_{1 \times 1}\left(\sum_{i=1}^4\left(\alpha_i \cdot U p_i\left(F_i\right)+\beta_i\right)\right)+F_{\text {coarse}}$                    (4)

where, $F_{e n h}$ denotes the final enhanced features. The summation operation achieves adaptive fusion of multi-scale features. The modulation biases correct the distribution deviations of each hierarchical level in real time. In addition, the 1 × 1 convolution is used to integrate the fused feature information and compress the feature dimensions.

Through this progressive fusion architecture—in which a stable global semantic base is first constructed, followed by the injection of local detailed features—the interference of low-level image noise with global scene semantics is effectively suppressed, thereby resolving the representation challenge of simultaneously accommodating large-scale scenes and minute knowledge-point targets in educational imagery. Furthermore, the residual connection design stabilizes gradient propagation during model training and reduces the convergence difficulty of the dynamic gating mechanism, enabling the model to adaptively accommodate the scale feature distributions of various English teaching scenarios and significantly enhancing the feature representation capacity for fine-grained knowledge points.

2.3 Hierarchical instructional semantic-guided cross-modal contrastive constraint module

To eliminate the cross-modal heterogeneous bias between visual representations and pedagogical semantics, a dual-layer cross-modal contrastive constraint system is constructed based on a structured English education knowledge graph, through which precise convergence of visual features and domain semantics is achieved. The schematic diagram of the hierarchical instructional semantic-guided cross-modal contrastive constraint is illustrated in Figure 2. A hierarchical knowledge graph structure is established, with a multi-level entity system defined according to instructional application logic. This system encompasses three entity types—teaching scenes, thematic units, and fine-grained knowledge points—and defines the inclusion, prerequisite association, and scene-adaptation relationships among entities. Based on the English curriculum standards for basic education and textbook statistical data, the association probability distributions between scenes and knowledge points are quantitatively derived, providing structured prior support for the model's semantic reasoning. For standardized textual semantic representation, an education-domain pre-trained encoder, edu-BERT, is employed to encode the graph entities and output fixed-dimensional textual semantic vectors. To unify the representation spaces of visual and textual modalities, a structurally symmetric dual-projection mapping network is designed. Both projection modules adopt a two-layer fully connected structure with the rectified linear unit activation function, uniformly outputting 512-dimensional feature vectors consistent with the text encoder's output dimension. The unified cross-modal projection process can be defined as:

$f_v=\Phi_v\left(G A P\left(F_{\text {enh}}\right)\right), f_t=\Phi_t(edu-BERT(caption))$             (5)

where, $\Phi_v$ and $\Phi_t$ denote the visual projection network and the textual projection network, respectively; $F_{\text {enh }}$ denotes the enhanced visual features input to the module; and $f_v$ and $f_t$ denote the visual representation and textual semantic representation in the joint space, respectively. Strict alignment of modal dimensions ensures the stability and effectiveness of cross-modal similarity computation.

2.png

Figure 2. Schematic diagram of the hierarchical instructional semantic-guided cross-modal contrastive constraint

Global scene-level contrastive loss is employed to achieve macro alignment between holistic image content and teaching scene semantics, constraining the model to perceive the instructional attributes of images from a global perspective. Cross-modal contrastive learning constraints are constructed based on the normalized temperature-scaled information noise-contrastive estimation loss function. By differentiating positive and negative samples within each batch, matched image-text sample pairs are pulled closer in the embedding space, while feature intervals between heterogeneous samples are enlarged. The specific computation is formulated as:

$L_{\text {global}}=-\log \frac{\exp \left(\cos \left(f_v, f_t^{+}\right) / \tau\right)}{\sum_{j=1}^B \exp \left(\cos \left(f_v, f_t^{(j)}\right) / \tau\right)}$                   (6)

where, $\tau$ denotes the temperature hyperparameter, used to adjust the dispersion of feature similarity distributions and enhance the discriminative capability of contrastive learning; $f_t^{+}$denotes the positive sample text representation that precisely matches the input image; $f_t^{(j)}$ denotes the text representation of any negative sample within the batch; and $B$ denotes the total batch size. Through this loss function, the learning direction of visual representations is regulated at the global semantic level, effectively avoiding the issues of ambiguous scene classification and global semantic matching misalignment commonly observed in general-purpose models, thereby achieving precise adaptation between global image content and teaching scenes.

To overcome the limitation that global constraints cannot achieve fine-grained knowledge-point localization, a local semantic alignment strategy based on cross-attention mechanism is introduced, with which semantic matching at the pixel-level visual region is accomplished through knowledge-point text anchors, enabling fine-grained semantic association without reliance on manually annotated bounding boxes. The knowledge-point text vectors encoded by the knowledge graph are adopted as query vectors, while the enhanced visual features serve as key-value vectors. Semantic association weights are computed via cross-modal cross-attention, generating knowledge-point-aware visual response heatmaps. The specific computation formula is as follows:

Attn=Softmax $\left(\frac{\left(t_v W_Q\right)\left(F_{e n h} W_K\right)^T}{\sqrt{d_k}}\right)\left(F_{e n h} W_V\right)$                  (7)

where, $t_v$ denotes the text embedding vector of the knowledgepoint entity, $W_Q, W_K$, and $W_V$ denote learnable modalityadaptive weight matrices, and $d_k$ denotes the feature dimension of the query vectors. The magnitude of attention weights directly corresponds to the semantic association strength between image regions and target knowledge points. Based on this, a local contrastive loss function $L_{\text {local}}$ is constructed, which drives the model to enhance feature representations of highly associated visual regions while attenuating interference from irrelevant background areas. Through this mechanism, pixel-level semantic alignment of knowledge points is achieved, effectively resolving the technical challenge of ambiguous fine-grained knowledgepoint localization.

The global and local dual-layer contrastive losses form a collaborative optimization system, establishing a full-dimensional semantic constraint mechanism spanning from macroscopic scenes to microscopic knowledge points. During model back-propagation, gradient information from the local semantic alignment loss is directly transmitted to the preceding multi-scale fusion module, dynamically adjusting the gating weights and feature distribution parameters of the scene-adaptive dynamic multi-scale gating fusion module. Through this collaborative optimization mechanism, the representational capacity of visual features at corresponding scales is adaptively strengthened according to the semantic localization results of knowledge points, establishing a bidirectional closed loop in which visual feature enhancement empowers semantic alignment, and semantic constraints recursively optimize feature extraction. This mechanism thoroughly bridges the modal gap between visual perception and education-specific pedagogical semantics, significantly enhancing the model's fine-grained semantic understanding capability in English education scenarios.

2.4 Dual-task collaborative optimization module based on semantic topology priors

A parallel dual-task inference structure is constructed based on the optimized joint semantic features, through which teaching scene classification and knowledge-point recognition are simultaneously accomplished, breaking the limitation of single-task independent optimization in traditional educational vision models. Two task-specific output branches with identical architectural configurations are deployed at the top of the model, corresponding respectively to the scene classification task and the knowledge-point recognition task. Each branch is composed of a global average pooling layer and a two-layer fully connected network, capable of mapping the unified semantic features to task-specific probability spaces. The output dimension of the scene classification branch is matched to the preset total number of scene categories, while the output dimension of the knowledge-point recognition branch corresponds to the total number of knowledge points annotated in the dataset, ultimately yielding scene category probability distributions and knowledge-point label probability distributions. Through the parallel dual-branch structure, the representational advantages of the preceding feature enhancement and semantic alignment modules are fully shared, avoiding redundant feature extraction during multi-task training and maintaining the model's lightweight characteristics while preserving inference accuracy. The flowchart of the dual-task collaborative optimization based on semantic topology priors is illustrated in Figure 3.

To constrain the model prediction process through the inherent logic of the teaching domain, a bidirectional collaborative distillation loss function is constructed in conjunction with the topological associations of the English education knowledge graph, through which mutual correction between the scene task and the knowledge-point task is achieved. The conditional probability distributions between scenes and knowledge points are quantitatively derived from the knowledge graph, characterizing the occurrence patterns of various knowledge points across different teaching scenarios. These distributions are normalized via the Softmax function to obtain standardized prior probability distributions. Domain prior information is incorporated into the task supervision process, and the fitting error between dual-task outputs and prior knowledge is measured via Kullback–Leibler divergence. The collaborative distillation loss is computed as follows:

$L_{c o}=K L\left(\operatorname{Softmax}\left(\log P_k+\log P_{\text {prior }}\right) \| P_s\right)$                       (8)

where, $P_k$ denotes the model's knowledge-point prediction probability distribution, $P_s$ denotes the scene classification prediction probability distribution, and $P_{\text {prior}}$ denotes the scene-knowledge-point prior probability distribution derived from the knowledge graph. A bidirectional constraint mechanism is established through this loss function. On one hand, the long-tail errors in knowledge-point recognition can be corrected using scene classification results and domain priors, compensating for recognition biases in less frequent knowledge points. On the other hand, the plausibility of scene classification results can be reversely validated through the distribution characteristics of knowledge points, ensuring that the dual-task outputs conform to the inherent logic of English instruction.

3.png

Figure 3. Flowchart of dual-task collaborative optimization based on semantic topology priors

To address the pervasive issues of feature sharing conflicts and representational homogenization in multi-task learning, a feature consistency regularization strategy integrating independence measurement and adversarial constraints is designed, achieving the optimization objective of preserving commonality while maintaining distinction among multi-task features. The task-specific features from the dual-task branches are extracted, and a regularization loss function is constructed by combining the Hilbert-Schmidt independence criterion with Kullback–Leibler divergence-based adversarial constraints:

$L_{\text {cons}}=H S I C\left(H_s, H_k\right)-\lambda \cdot K L\left(P_s \| P_k\right)$                     (9)

where, $H_s$ and $H_k$ denote the task-specific features of the scene task and the knowledge-point task, respectively; HSIC( ) is used to measure the correlation between the two sets of features; and $\lambda$ denotes the adversarial constraint balancing coefficient. Maximization of the Hilbert-Schmidt independence criterion drives the dual tasks to share common educational semantic features, reducing redundant learning overhead. Concurrently, a gradient reversal mechanism is introduced into the Kullback–Leibler divergence constraint term, through which the feature values are kept constant during forward propagation, while the gradients are reversely weighted during back-propagation. In an adversarial learning manner, the dual-task branches are forced to mine differentiated feature information, effectively preventing performance degradation caused by multi-task training. This mechanism requires no additional network parameters and achieves task feature balancing solely through gradient modulation, offering high training efficiency.

To integrate multi-dimensional supervision signals and achieve full-chain collaborative optimization, a holistic loss function incorporating multiple constraints is constructed, through which the parameter updates of all modules are uniformly regulated, forming a multi-level closed-loop optimization system. The complete expression of the holistic loss function is:

$L_{\text {total}}=L_{\text {cls}}^s+L_{\text {cls }}^k+\gamma_1\left(L_{\text {global}}+L_{\text {local}}\right)+\gamma_2 L_{\text {co }}+\gamma_3 L_{\text {cons}}$                      (10)

where, $L_{\text {cls}}^s$ and $L_{\text {cls}}^k$ denote the cross-entropy classification losses for scene classification and knowledge-point recognition, respectively, which are responsible for ensuring the discriminative performance of the foundational tasks; $\gamma_1$, $\gamma_2$, and $\gamma_3$ denote hyperparameters used to balance the optimization weights of each loss term. A progressive weight adjustment strategy is adopted to align with the model training dynamics: smaller semantic constraint weights are set in the early training stages, allowing the model to prioritize learning basic visual discriminative features, while the weights of cross-modal contrastive losses are gradually increased in the middle and late training stages to achieve refined convergence in the semantic space. The collaborative distillation and regularization loss weights are maintained at relatively low values to prevent excessive intervention of domain constraints and multi-task adversarial mechanisms in foundational task learning. All hyperparameters are determined through grid search on the validation set, ensuring training stability and experimental result reproducibility.

Through the complete dual-task collaborative optimization mechanism, deep coupling among visual features, cross-modal semantics, and educational tasks is achieved. The front-end feature enhancement and semantic alignment modules provide high-quality representational support for dual-task inference, while the back-end multi-task supervision signals and domain prior constraints recursively optimize the feature extraction and semantic alignment processes. Through the full-chain gradient back-propagation and bidirectional empowerment mechanism, the disconnection between perceptual tasks and instructional objectives in traditional educational vision models is thoroughly resolved. The image understanding logic of the model is thereby rendered highly consistent with the pedagogical cognitive principles of English instruction, significantly enhancing the model's adaptability and parsing accuracy in real-world smart education scenarios.

In conjunction with the quantitative experimental data and visualization results presented above, the intrinsic operational mechanisms and module collaborative logic of the proposed integrated framework are further analyzed in this chapter, with the core principles underlying the model's performance improvements elucidated. The dynamic multi-scale gating fusion mechanism designed in this work abandons the traditional fixed-weight feature stacking paradigm, relying on global scene semantic information to drive adaptive weighted regulation of multi-scale features, whereby detail features or global semantic features can be dynamically strengthened according to the visual distribution characteristics of English teaching scenarios. In conjunction with the coarse-to-fine progressive fusion paradigm, the model is able to construct a stable scene semantic base using high-level features, thereby avoiding semantic drift caused by low-level texture noise and fundamentally resolving the challenges of large scale span and weak small-target representation in teaching imagery. On this basis, the hierarchical cross-modal contrastive constraint system establishes a dual-layer semantic convergence mechanism from global scenes to local knowledge points, using the domain knowledge graph as a semantic anchor to guide visual features toward fine-grained alignment. Moreover, the semantic supervision signals can recursively optimize the front-end feature fusion process, forming a closed-loop empowerment relationship between feature enhancement and semantic matching. The dual-task collaborative optimization strategy, based on the inherent topological regularities of teaching scenarios, unifies the semantic logic of both tasks through collaborative distillation constraints. Combined with the feature regularization mechanism, this strategy balances task feature sharing and differentiated learning, effectively resolving the industry pain points of disconnection between perceptual tasks and educational objectives in traditional vision models, as well as feature conflicts in multi-task learning.

The proposed framework exhibits excellent recognition accuracy, scale robustness, and generalization capability in conventional English teaching scenarios; however, clear performance boundaries and application limitations remain. For extremely small-scale characters with very low pixel occupancy and severely distorted handwritten English content, effective visual feature information is scarce and morphological irregularities are present, making it difficult for the model to achieve precise feature matching and knowledge-point recognition, with obvious bottlenecks existing in fine-grained parsing performance. Meanwhile, the semantic reasoning capability of the model is highly dependent on the coverage of the structured educational knowledge graph. The current knowledge base, with its limited entity count and topological associations, only covers conventional teaching scenarios and core knowledge points, rendering it incapable of adapting to extended teaching content and niche teaching scenarios. Consequently, the breadth of the model's semantic understanding is constrained by the completeness of the domain priors. Furthermore, under extremely complex acquisition conditions such as large-area occlusion and high-intensity imaging noise, the fundamental visual features of the images are severely degraded, with slight degradation observed in the model's cross-modal semantic alignment accuracy and task recognition performance.

In view of the limitations of the current research, future work can be continuously deepened in three directions: scenario expansion, capability upgrade, and deployment optimization. Subsequently, the educational knowledge system can be extended to cover multiple grade levels and multiple languages, with the topological association relationships between scenes and knowledge points refined, thereby breaking through the application constraints of a single English scenario and constructing a more generalizable educational visual semantic parsing framework. Furthermore, the open-ended semantic generation capability of large language models can be integrated to break through the inherent paradigm of fixed-label classification, achieving open-ended knowledge-point mining, semantic summarization, and structured information extraction from teaching imagery, with the higher-order semantic understanding capability of the model further enhanced. Finally, lightweight optimization of the overall framework can be performed for terminal deployment requirements, with computational overhead reduced through model distillation, structural pruning, and quantization inference, thereby adapting to the real-time inference requirements of various teaching terminals and mobile devices and promoting the large-scale deployment and engineering application of intelligent educational image understanding technology in smart teaching scenarios.