Modeling and Deep Optimization of an English Learning Assistance System via Multimodal Fusion of Image Captioning and Visual Semantic Embedding

2.1 Overall system framework

The English learning assistance system integrating image captioning and visual semantic embedding, constructed in this paper, centers on high-precision image processing and deep multimodal semantic fusion. The overall architecture is shown in Figure 1, clearly presenting the logical connections and input-output closed loop of the five core modules. The system takes raw images and English learning corpora as initial inputs; first, it completes image feature extraction through a multi-granularity cross-layer fusion visual encoder, outputting high-quality visual features that combine local texture details and global semantic structures, providing core support for subsequent multimodal alignment. The extracted visual features are fed into the hierarchical visual semantic embedding module to complete fine-grained alignment with English word-level and phrase-level semantics, generating unified-dimensional visual-semantic embedding features. Subsequently, a visual-semantic gated attention generator generates precise image captions adapted to English learning scenarios based on these embedding features. To further enhance the accuracy of captions and learning adaptability, an error-driven adversarial optimization module is introduced to iteratively optimize the generator, combining priors of English learning error distribution to ensure the rationality and practicality of the descriptions. Finally, a cross-modal adaptive curriculum learning module dynamically adjusts training sample weights, and simultaneously generates visual-semantic alignment heatmaps, transforming image processing results into intuitive learning feedback to achieve deep integration of image processing technology and English learning assistance. The collaborative effort of all modules forms an end-to-end intelligent assistance system, ensuring the precision of visual feature extraction, the fineness of semantic alignment, and the adaptability of caption generation.

Figure 1. Overall architecture of the English learning assistance system integrating image captioning and visual semantic embedding

2.2 Multi-granularity visual feature extraction and cross-layer fusion

The multi-scale representation capability of visual features is the foundation for achieving fine-grained cross-modal understanding. To balance spatial detail preservation and high-level semantic modeling, this section adopts a dual-branch collaborative encoding structure to complete image feature parsing, and the network structure diagram is shown in Figure 2. Defining the standardized input image as $\text{I}{{\text{R}}^{\text{3 }\!\!\times\!\!\text{ }{{\text{H}}_{\text{0}}}\text{ }\!\!\times\!\!\text{ }{{\text{W}}_{\text{0}}}}}$, relying on a lightweight convolutional encoding network to complete hierarchical feature iterative extraction, and passing through multi-level downsampling and feature transformation, four groups of feature maps with different scales ${{\text{F}}_{\text{1}}}\text{,}{{\text{F}}_{\text{2}}}\text{,}{{\text{F}}_{\text{3}}}\text{,}{{\text{F}}_{\text{4}}}$ are sequentially output. Shallow network features retain rich edge textures and spatial details, while deep features complete high-level semantic abstraction and target semantic condensation. The hierarchical output of multi-stage features covers visual expression requirements of different granularities, providing diverse visual representations for subsequent cross-modal semantic matching. Although convolutional encoding has efficient computational advantages in local spatial feature capture, its inherent local receptive field constraint makes it difficult to model long-range scene associations. To compensate for global semantic modeling capabilities, this paper adds a global semantic enhancement branch, inputting the middle-level feature ${{\text{F}}_{\text{3}}}$ into the Swin Transformer structure. This level of feature balances computational overhead and modeling performance by retaining basic spatial structure while fusing primary semantic information. Relying on the shifted window self-attention mechanism, the module can establish long-distance dependencies between image regions and mine global contextual information such as scene layout and target interaction, ultimately generating global semantic features ${{\text{F}}_{\text{swin}}}$, thereby constructing a dual-path feature expression system parallelizing local details and global semantics. To achieve the organic integration of the two heterogeneous features, this paper constructs an adaptive gated fusion module to dynamically complete the weighted aggregation of deep convolutional features and global semantic features. First, an upsampling operation ${{\text{U}}_{\text{P}}}\text{( }\!\!~\!\!\text{ )}$ is performed on the deep feature ${{\text{F}}_{\text{4}}}$ to complete feature scale unification and matching. Global average pooling operators are adopted to compress the spatial dimension and reduce redundant information interference. The pooling operation can be defined as:

$\text{AvgPool(F)=}\frac{\text{1}}{\text{H }\!\!\times\!\!\text{ W}}\mathop{\sum }_{\text{i=1}}^{\text{H}}\mathop{\sum }_{\text{j=1}}^{\text{W}}\text{F(i,j)}$                  (1)

Figure 2. Network structure diagram of multi-granularity visual feature extraction and adaptive cross-layer fusion

The two sets of pooled feature vectors are concatenated and fed into a multilayer perceptron (MLP). After learning feature correlations through nonlinear transformation, continuous fusion weight $\text{ }\!\!\gamma\!\!\text{ }$ is generated via a Sigmoid activation function. The overall fusion process follows the calculation form below:

${{\text{F}}_{\text{v}}}\text{= }\!\!\gamma\!\!\text{ }\cdot \text{Up(}{{\text{F}}_{\text{4}}}\text{)+(1- }\!\!\gamma\!\!\text{ )}\cdot {{\text{F}}_{\text{swin}}}\text{,} \text{ }\!\!\gamma\!\!\text{ = }\!\!\sigma\!\!\text{ }\left( \text{MLP}\left( \left[ \text{AvgPool(}{{\text{F}}_{\text{4}}}\text{);AvgPool(}{{\text{F}}_{\text{swin}}}\text{)} \right] \right) \right)$                    (2)

The weight coefficients can be autonomously updated iteratively based on image content complexity and target density, dynamically adjusting the contribution ratio of the two types of features to avoid the problem of singular representation caused by fixed fusion weights.

The aggregated visual feature ${{\text{F}}_{\text{v}}}$, outputted by adaptive fusion, integrates the fine-grained perception advantages of convolutional structures with the global modeling capabilities of Transformers, realizing the unified representation of texture details, target features, and scene semantics. The fusion strategy weakens the feature distribution differences brought by different network architectures and strengthens the robustness and integrity of visual representation. This optimized visual feature can adapt to hierarchical semantic parsing tasks; it can support the precise description of local objects while satisfying the complete semantic expression of the overall scene, providing stable and reliable front-end visual support for subsequent visual semantic embedding alignment and English text generation tasks.

2.3 Hierarchical visual semantic embedding and contrastive learning

Figure 3 shows the interaction diagram of hierarchical visual semantic embedding and visual-semantic gated attention mechanism. Among them, the core of visual semantic embedding is to establish precise mapping between visual features and language semantics; however, there exist hierarchical semantic requirements of words, phrases, and scenes in English learning scenarios, and single-granularity embedding makes it difficult to achieve precise alignment of multi-dimensional semantics. To this end, this paper constructs a hierarchical visual semantic embedding method, synchronously generating dual-granularity embedding features at the word and phrase levels, realizing hierarchical matching between visual information and English semantics, and providing refined multimodal support for subsequent scenario-based description generation. Both dual-granularity embedding spaces adopt a unified dimension d, where the word-level embedding matrix ${{\text{E}}_{\text{word}}}{{\text{R}}^{{{\text{N}}_{\text{w}}}\text{ }\!\!\times\!\!\text{ d}}}$corresponds to the semantics of basic English vocabulary, and the phrase-level embedding matrix ${{\text{E}}_{\text{phrase}}}{{\text{R}}^{{{\text{N}}_{\text{p}}}\text{ }\!\!\times\!\!\text{ d}}}$ corresponds to the semantics of common English phrases, ensuring the interoperability of semantic features at different levels.

Figure 3. Interaction diagram of hierarchical visual semantic embedding and visual-semantic gated attention mechanism

Word-level embedding is constructed based on an English learning core corpus, performing semantic encoding on vocabulary through a pre-trained language model to retain contextual association information and realize fine-grained semantic representation. Phrase-level embedding combines visual region features and linguistic phrase semantics; first, object regions are extracted from the multi-granularity visual feature ${{\text{F}}_{\text{v}}}$ via a Region Proposal Network to screen out semantically meaningful image regions, and then a Spatial Pyramid Pooling operation is performed on each region to capture regional feature information at different scales. Spatial Pyramid Pooling can be expressed as:

$\text{SPP(F)=} \text{ }\!\![\!\!\text{ MaxPool(F,}{{\text{k}}_{\text{1}}}\text{);MaxPool(F,}{{\text{k}}_{\text{2}}}\text{);MaxPool(F,}{{\text{k}}_{\text{3}}}\text{) }\!\!]\!\!\text{ }$                 (3)

where, ${{\text{k}}_{\text{1}}}\text{,}{{\text{k}}_{\text{2}}}\text{,}{{\text{k}}_{\text{3}}}$ are pooling windows of different scales. Fixed-dimensional regional feature vectors are obtained through feature concatenation, which are subsequently fused with corresponding English phrase semantic encodings to generate phrase-level visual semantic embeddings, realizing precise correspondence between visual regions and phrase semantics.

To improve the alignment precision of the dual-granularity embedding space, this paper designs a hybrid similarity measurement function, combining the advantages of cosine similarity and structural Wasserstein distance, balancing the capture of feature vector correlation and distribution differences. Cosine similarity is used to measure the linear correlation degree between visual features and semantic features, while structural Wasserstein distance quantifies the distribution difference of the two types of features in the embedding space, effectively compensating for the defect that traditional similarity measures cannot capture distributional structural differences. The similarity calculation is as follows:

$\text{s(v,c)= }\!\!\lambda\!\!\text{ }\cdot \text{cos(v,c)+(1- }\!\!\lambda\!\!\text{ )}\cdot \text{exp}\left( \text{-}\frac{{{\text{W}}_{\text{2}}}\text{(}{{\text{p}}_{\text{v}}}\text{,}{{\text{p}}_{\text{c}}}\text{)}}{{{\text{ }\!\!\tau\!\!\text{ }}_{\text{w}}}} \right)$                  (4)

where, $\text{ }\!\!\lambda\!\!\text{  } \in \!\![\!\!\text{ 0,1 }\!\!]\!\!\text{ }$ is the adaptive weight, ${{\text{ }\!\!\tau\!\!\text{ }}_{\text{w}}}$ is the temperature coefficient, ${{\text{p}}_{\text{v}}}$ and ${{\text{p}}_{\text{c}}}$ represent the distribution probabilities of visual features and semantic features respectively, and ${{\text{W}}_{\text{2}}}$() is the second-order Wasserstein distance. Its simplified calculation form is:

${{\text{W}}_{\text{2}}}\text{(}{{\text{p}}_{\text{v}}}\text{,}{{\text{p}}_{\text{c}}}\text{)=}\underset{\text{ }\!\!\gamma\!\!\text{ }\in \text{ }\!\!\Pi\!\!\text{ (}{{\text{p}}_{\text{v}}}\text{,}{{\text{p}}_{\text{c}}}\text{)}}{\mathop{\text{inf}}}\,\sqrt{\int \|\text{x-y}{{\|}^{\text{2}}}\text{d }\!\!\gamma\!\!\text{ (x,y)}}$                (5)

$\text{ }\!\!\Pi\!\!\text{ (}{{\text{p}}_{\text{v}}}\text{,}{{\text{p}}_{\text{c}}}\text{)}$ is the set of joint distributions of ${{\text{p}}_{\text{v}}}$ and ${{\text{p}}_{\text{c}}}$. A contrastive learning strategy is adopted to jointly optimize the dual-granularity embedding space, constructing visual-semantic positive and negative sample pairs to guide the alignment of embedding features. Positive sample pairs consist of matched visual features and semantic features, while negative sample pairs are non-matched feature combinations. Matching feature pairs are clustered and non-matching feature pairs are separated by minimizing the contrastive loss. The loss function is defined as:

${{\text{L}}_{\text{HVSE}}}\text{=-}\frac{\text{1}}{\text{B}}\mathop{\sum }_{\text{i=1}}^{\text{B}}\left[ \begin{matrix} \text{log}\frac{\text{exp(s(}{{\text{v}}_{\text{i}}}\text{,c}_{\text{i}}^{\text{+}}\text{)/ }\!\!\tau\!\!\text{ )}}{\mathop{\sum }_{\text{j=1}}^{\text{B}}\text{exp(s(}{{\text{v}}_{\text{i}}}\text{,}{{\text{c}}_{\text{j}}}\text{)/ }\!\!\tau\!\!\text{ )}}  \\ \text{+log}\frac{\text{exp(s(}{{\text{c}}_{\text{i}}}\text{,v}_{\text{i}}^{\text{+}}\text{)/ }\!\!\tau\!\!\text{ )}}{\mathop{\sum }_{\text{j=1}}^{\text{B}}\text{exp(s(}{{\text{c}}_{\text{i}}}\text{,}{{\text{v}}_{\text{j}}}\text{)/ }\!\!\tau\!\!\text{ )}}  \\ \end{matrix} \right]$                      (6)

where, B is the batch size, $\text{ }\!\!\tau\!\!\text{ }$ is the temperature coefficient of the contrastive loss, $\text{v}_{\text{i}}^{\text{+}}$ and $\text{c}_{\text{i}}^{\text{+}}$ are the positive samples of ${{\text{v}}_{\text{j}}}$ and ${{\text{c}}_{\text{i}}}$, respectively. During the contrastive learning process, word-level and phrase-level embedding spaces are optimized synchronously to ensure that both fine-grained vocabulary semantics and mid-grained phrase semantics align precisely with visual features, providing a high-quality multimodal feature basis for subsequent visual-semantic gated attention generation and adapting to the dual requirements of vocabulary application and phrase expression in English learning.

2.4 Visual-semantic gated attention caption generator

The core of image captioning lies in generating coherent, accurate text expressions adapted to English learning scenarios based on visual features and semantic information. This paper constructs a caption generation framework based on a Transformer decoder, focusing on achieving dynamic fusion of visual information and semantic information through a gated attention mechanism to adapt to caption generation tasks under different image complexities and English learning requirements. The decoder takes the features output by the hierarchical visual semantic embedding and the historical generation sequence as input, and through multiple rounds of self-attention and cross-attention interactions, gradually generates English descriptive texts that conform to grammatical norms and fit image content. The visual-semantic gated attention mechanism is responsible for dynamically balancing the weight distribution between visual attention and semantic attention.

In each step of the decoder generation process, visual attention weights and semantic attention weights are first calculated based on the current hidden state ${{\text{h}}_{\text{t}}}$. The visual attention branch uses the multi-granularity fused visual feature ${{\text{F}}_{\text{v}}}$ as key-value pairs, generating visual query, key, and value matrices through linear transformation. The query matrix is obtained by transforming the decoder hidden state ${{\text{h}}_{\text{t}}}$ via the visual query weight $\text{W}_{\text{Q}}^{\text{v}}$, and the key matrix is obtained by transforming ${{\text{F}}_{\text{v}}}$ via the visual key weight $\text{W}_{\text{K}}^{\text{v}}$. Finally, the visual attention weight $\text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{v}}$ is calculated from via scaled dot-product attention. The semantic attention branch uses phrase-level semantic embedding ${{\text{E}}_{\text{pharse}}}$ as key-value pairs, adopts the same scaled dot-product attention structure, and completes the transformation through semantic query weight $\text{W}_{\text{Q}}^{\text{s}}$ and semantic key weight to calculate the semantic attention weight $\text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{s}}$. The specific calculation process is as follows:

$\text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{v}}\text{=Softmax}\left( \frac{\text{(}{{\text{h}}_{\text{t}}}\text{W}_{\text{Q}}^{\text{v}}\text{)(}{{\text{F}}_{\text{v}}}\text{W}_{\text{K}}^{\text{v}}{{\text{)}}^{\top }}}{\sqrt{\text{d}}} \right)\text{,} \text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{s}}\text{=Softmax}\left( \frac{\text{(}{{\text{h}}_{\text{t}}}\text{W}_{\text{Q}}^{\text{s}}\text{)(}{{\text{E}}_{\text{pharse}}}\text{W}_{\text{K}}^{\text{s}}{{\text{)}}^{\top }}}{\sqrt{\text{d}}} \right)$                   (7)

where, d is the feature embedding dimension, and the scaling factor $\sqrt{\text{d}}$ is used to mitigate the Softmax gradient vanishing problem caused by excessively large attention scores. $\text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{v}}$ and $\text{ }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{s}}$ reflect the degree of attention to visual regions and semantic concepts at the current generation step, respectively. Based on the attention weights, the visual context vector $\text{c}_{\text{t}}^{\text{v}}\text{= }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{v}}\text{(}{{\text{F}}_{\text{v}}}\text{W}_{\text{V}}^{\text{v}}\text{)}$ and the semantic context vector $\text{c}_{\text{t}}^{\text{s}}\text{= }\!\!\alpha\!\!\text{ }_{\text{t}}^{\text{s}}\text{(}{{\text{E}}_{\text{phrase}}}\text{W}_{\text{V}}^{\text{s}}\text{)}$ are further calculated, where $\text{W}_{\text{V}}^{\text{v}}$ and $\text{W}_{\text{V}}^{\text{s}}$ are the value weight matrices for visual and semantic attention, respectively.

To achieve adaptive fusion of visual and semantic context information, a dynamic gating mechanism is designed to generate a fusion scalar ${{\text{g}}_{\text{t}}}$. This scalar is obtained by applying a linear transformation and Sigmoid activation to the current decoder hidden state ${{\text{h}}_{\text{t}}}$, capable of dynamically adjusting the fusion ratio of the two context vectors based on image clarity, semantic complexity, and contextual information of the generated sequence. The gating fusion process and scalar generation formula are as follows:

${{\text{g}}_{\text{t}}}\text{= }\!\!\sigma\!\!\text{ }\left( {{\text{h}}_{\text{t}}}{{\text{W}}_{\text{g}}}\text{+}{{\text{b}}_{\text{g}}} \right)\text{,}\quad {{\text{c}}_{\text{t}}}\text{=}{{\text{g}}_{\text{t}}}\odot \text{c}_{\text{t}}^{\text{v}}\text{+}\left( \text{1-}{{\text{g}}_{\text{t}}} \right)\odot \text{c}_{\text{t}}^{\text{s}}$                   (8)

where, ${{\text{W}}_{\text{g}}}$ and ${{\text{b}}_{\text{g}}}$ are the weight matrix and bias term of the gating mechanism, respectively, and $\text{ }\!\!\sigma\!\!\text{ }$ is the Sigmoid activation function, ensuring ${{\text{g}}_{\text{t}}}\text{ }\!\![\!\!\text{ 0,1 }\!\!]\!\!\text{ }$. When image details are rich and semantics are simple, ${{\text{g}}_{\text{t}}}$ approaches 1, and the model focuses on generating fine-grained descriptions relying on visual context; when the image is blurry and semantics are complex, ${{\text{g}}_{\text{t}}}$ approaches 0, and the model focuses on relying on semantic context to ensure the coherence and accuracy of descriptions. This dynamic regulation characteristic can effectively adapt to the description requirements of scenarios with different difficulty levels in English learning.

After concatenating the fused context vector ${{\text{c}}_{\text{t}}}$ with the current decoder hidden state ${{\text{h}}_{\text{t}}}$, it is input into a feed-forward neural network to complete nonlinear transformation, and the probability distribution $\text{p(}{{\text{y}}_{\text{t}}}\text{ }\!\!|\!\!\text{ }{{\text{y}}_{\text{t}}}\text{,}{{\text{F}}_{\text{v}}}\text{,}{{\text{E}}_{\text{phrase}}}\text{)}$ of the next word is obtained after Softmax activation. To ensure the accuracy and grammatical normativity of the generated descriptions, a cross-entropy loss function is used to supervise the training of the generator. The loss function is defined as:

${{\text{L}}_{\text{CE}}}\text{=-}\mathop{\sum }_{\text{t=1}}^{\text{T}}\text{logp(y}_{\text{t}}^{\text{*}}\text{ }\!\!|\!\!\text{ }{{\text{y}}_{\text{t}}}\text{,}{{\text{F}}_{\text{v}}}\text{,}{{\text{E}}_{\text{phrase}}}\text{)}$                          (9)

where, T is the length of the generated sequence, $\text{y}_{\text{t}}^{\text{*}}$ and is the ground truth label at step t. The cross-entropy loss can effectively minimize the difference between the generated distribution and the real distribution, guiding the model to learn English grammar rules and contextual expression habits. Combined with the dynamic regulation capability of the gated attention mechanism, the generated descriptions not only fit the image content but also conform to the cognitive laws of English learning, laying the foundation for subsequent error-driven adversarial optimization.

2.5 Adversarial optimization for learning errors

To ensure that generated descriptions not only fit the image content but also adapt to the personalized needs of English learning scenarios, it is necessary to guide the model to avoid common error patterns of learners while ensuring generation accuracy. This paper introduces a prior distribution of English learning errors and constructs an adversarial optimization framework for learning errors. Through game training between a generator and a discriminator, learning-friendly description generation is achieved. Figure 4 shows the flowchart of adversarial optimization for learning errors and adaptive curriculum learning. The error distribution prior is statistically obtained based on a large-scale English learning corpus, covering typical error types such as vocabulary misuse, grammatical errors, and semantic deviations, denoted as ${{\text{P}}_{\text{err}}}\text{(e)}$. Error vectors are obtained by sampling from this distribution to inject into the generator, guiding the model to learn expression patterns that avoid errors. Error vector sampling satisfies $\text{z}\sim {{\text{P}}_{\text{err}}}\text{(e)}$, where e is the error type feature vector. The sampling process is implemented via a reparameterization trick to ensure gradients are backpropagatable.

Figure 4. Flowchart of adversarial optimization for learning errors and adaptive curriculum learning

An error pattern discriminator ${{\text{D}}_{\phi }}$ is constructed, taking image features ${{\text{F}}_{\text{v}}}$ and the semantic embedding of the generated description as input, and outputs the probability that the description is learning-friendly. The discriminator is constructed using an MLP, and its output can be expressed as ${{\text{D}}_{\phi }}\text{(I,S)= }\!\!\sigma\!\!\text{ (MLP( }\!\![\!\!\text{ }{{\text{F}}_{\text{v}}}\text{;}{{\text{E}}_{\text{S}}}\text{ }\!\!]\!\!\text{ ))}$, where ${{\text{E}}_{\text{S}}}$ is the semantic embedding of the description text and $\text{ }\!\!\sigma\!\!\text{ }$ is the Sigmoid activation function. Adversarial training adopts a minimax game strategy: the generator ${{\text{G}}_{\text{ }\!\!\theta\!\!\text{ }}}$ generates descriptions by injecting error vectors, attempting to mislead the discriminator into judging them as learning-friendly; the discriminator learns the differences between real learning-friendly descriptions and error-induced generated descriptions to improve discrimination accuracy. The adversarial loss function is defined as:

${{\text{L}}_{\text{Adv}}}\text{=}{{\text{E}}_{\text{I,}{{\text{S}}_{\text{real}}}}}\text{ }\!\![\!\!\text{ log}{{\text{D}}_{\phi }}\text{(I,}{{\text{S}}_{\text{real}}}\text{) }\!\!]\!\!\text{ } \text{+}{{\text{E}}_{\text{I,z}\sim {{\text{P}}_{\text{ex}}}}}\text{ }\!\![\!\!\text{ log(1-}{{\text{D}}_{\phi }}\text{(I,}{{\text{G}}_{\text{ }\!\!\theta\!\!\text{ }}}\text{(}{{\text{F}}_{\text{v}}}\text{,z))) }\!\!]\!\!\text{ }$                         (10)

where ${{\text{S}}_{\text{real}}}$ is the real learning-friendly description. The expectation terms correspond to the discrimination loss of real samples and generated samples, respectively, guiding the generator to produce descriptions that better fit learning needs.

To prevent the generated descriptions from becoming detached from the image content during adversarial training, a visual-textual consistency constraint is introduced. The CLIP model is utilized to extract features of the image and the generated description, and the semantic consistency of the two is measured by calculating the Euclidean distance. The constraint formula is:

${{\text{L}}_{\text{cons}}}\text{=}\|\text{CLIP(I)-CLIP(}{{\text{S}}_{\text{gen}}}\text{)}\|_{\text{2}}^{\text{2}}$                       (11)

where, ${{\text{S}}_{\text{gen}}}$ is the description text output by the generator, $\text{CLIP(I)}$ and $\text{CLIP(}{{\text{S}}_{\text{gen}}}\text{)}$ are the CLIP feature vectors of the image and the description, respectively. The cross-entropy loss, hierarchical visual semantic embedding loss, adversarial loss, and consistency constraint are integrated to construct the total model loss function:

${{\text{L}}_{\text{total}}}\text{=}{{\text{L}}_{\text{CE}}}\text{+}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{1}}}{{\text{L}}_{\text{HVSE}}}\text{+}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{2}}}{{\text{L}}_{\text{Adv}}}\text{+}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{3}}}{{\text{L}}_{\text{cons}}}$                    (12)

where, ${{\text{ }\!\!\lambda\!\!\text{ }}_{\text{1}}}\text{,}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{2}}}\text{,}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{3}}}$ are adaptive weight coefficients, dynamically adjusted based on validation set performance, satisfying ${{\text{ }\!\!\lambda\!\!\text{ }}_{\text{1}}}\text{+}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{2}}}\text{+}{{\text{ }\!\!\lambda\!\!\text{ }}_{\text{3}}}\text{=1}$ to ensure the synergistic optimization of each loss term. This achieves precise adaptation of generated descriptions to English learning needs while guaranteeing image-description consistency. The model adopts an alternating training strategy, updating generator and discriminator parameters alternately until the loss function converges, ultimately obtaining a description generation model that possesses both accuracy and learning adaptability.