Multimodal Feature Fusion and Visual Attractiveness Prediction for Advertising Images in Digital Marketing

2.1 Overall framework overview

To address the key problems of insufficient fusion between visual features and textual semantic features of advertising images and lack of interpretability in digital marketing scenarios, and to achieve high-precision visual attractiveness prediction and interpretable heatmap generation, this paper proposes a DBAFN. The network adopts a three-stage cascaded architecture. First, multimodal feature extraction is performed on advertising images and their corresponding textual descriptions to obtain image feature maps and textual semantic vectors adapted to subsequent fusion tasks. Then, through a text–image spatial affine modulation mechanism, textual semantics are explicitly projected into the image feature space and modulation tensors are generated, providing precise semantic guidance for DBAFN. Next, a global–local dual-branch attention architecture is used to capture the global visual style and local semantic salient regions of advertising images, respectively, and an adaptive gating module is employed to complete pixel-level fusion of the two, obtaining fused feature maps with both global feature integrity and local semantic saliency. Finally, the regression task of attractiveness score is completed based on the fused feature maps, and interpretable heatmaps are generated through a lightweight decoder. Meanwhile, a multi-task joint loss function is adopted to achieve end-to-end training of the model, ensuring dual optimization of prediction accuracy and interpretability. The innovation core of this framework lies in explicit spatial modulation of textual semantics, adaptive pixel-level fusion of dual-branch attention, and collaborative improvement of interpretability and prediction accuracy. A semantic-driven visual attractiveness prediction paradigm for advertising images is constructed from the perspective of image processing, effectively breaking through the limitations of existing methods in multimodal fusion depth and interpretability. The overall architecture is shown in Figure 1.

Figure 1. Overall architecture of the dual-branch attention fusion network (DBAFN)

2.2 Multimodal feature extraction

2.2.1 Image feature extraction

The core of image feature extraction is to obtain feature representations with both multi-scale semantic information and dimensional adaptability, providing high-quality inputs for subsequent text–image spatial affine modulation and DBAFN. The innovative design focuses on the effective fusion of multi-scale features and precise dimensionality reduction, in order to solve the problems of insufficient semantic representation of single-scale features and dimensional mismatch with subsequent fusion modules. Swin Transformer is selected as the backbone network. The input RGB advertising image normalized to a resolution of 384×384 is fed into the network, and the feature maps of the third, fourth, and fifth stages are extracted, which are $F_3 \in \mathrm{R}^{48 \times 48 \times 192}, F_4 \in \mathrm{R}^{24 \times 24 \times 384}, F_5 \in \mathrm{R}^{12 \times 12 \times 768}$, respectively. These correspond  to spatial resolutions of 1/8, 1/16, and 1/32 of the input image, covering multi-scale information from local details to global semantics. To fully utilize the complementarity of multi-scale features, bilinear interpolation is applied to F₃ and F₄ for upsampling, with upsampling factors set to 4 and 2, respectively. Their spatial resolutions are uniformly increased to 12×12, consistent with F₅, obtaining the upsampled feature maps $F_3 \in \mathrm{R}^{12 \times 12 \times 192}$ and $F_4 \in \mathrm{R}^{12 \times 12 \times 384}$. Subsequently, $\mathrm{F}_3^{\prime}$ and $\mathrm{F}_4^{\prime}$ are concatenated with $\mathrm{F}_5$ along the channel dimension. The concatenation process can be expressed as:

$F_{\text {cat}}=\operatorname{Concat}\left(F_3^{\prime}, F_4^{\prime}, F_5\right)$    (1)

where, Concat (·) denotes the channel-wise concatenation operation. The final concatenated feature is $F_{c a t} \in \mathrm{R}^{12 \times 12 \times 1344}$. To reduce computational complexity and unify feature dimensions to match the tensor dimensional requirements of subsequent spatial affine modulation, a 1×1 convolution layer is designed to perform channel reduction on F_cat. The convolution layer is configured with 512 convolution kernels, stride 1, padding 0, and Rectified Linear Unit (ReLU) activation function. The weights are initialized using He normal initialization, and the bias is initialized to 0, ensuring stability and convergence speed of feature extraction. After dimensionality reduction, the final image feature map $F_I \in \mathrm{R}^{12 \times 12 \times 512}$ is obtained. This design compensates for the semantic deficiency of single-scale features through multi-scale feature fusion, and simultaneously achieves dimensional matching with the subsequent text–image spatial affine modulation module through precise dimensionality reduction, laying a solid foundation for channel-wise dynamic modulation of image features by textual semantics.

2.2.2 Text feature extraction

To accurately capture the overall semantics and key entity information in advertising copy, text feature extraction is further performed to provide high-quality semantic input for subsequent text–image spatial affine modulation. The innovative design focuses on strengthening key entity semantics and fusing multi-dimensional textual features, addressing the problems of ambiguous key semantics in traditional text extraction and insufficient adaptability to image modulation requirements. The Robustly Optimized BERT Pretraining Approach (RoBERTa)-base model is used to encode the preprocessed advertising copy, obtaining the sequence vector $T_{\text {seq}} \in \mathrm{R}^{64 \times 768}$. To enhance key semantic information that plays a decisive role in advertising attractiveness, a named entity recognition module is introduced to perform entity annotation on the copy, and the annotation categories include four types: price, discount, brand, and product. After inputting T_seq into the module, the entity category probability of each token is obtained through the Softmax function. Tokens with probability greater than 0.8 are selected as key entity tokens, and the arithmetic mean of their corresponding semantic embedding vectors is taken to obtain the entity vector $T_{\text {ent}}$. The calculation process can be expressed as:

$T_{e n t}=\frac{1}{K} \sum_{i=1}^K T_{\mathrm{seq}}[i]$ (2)

where, $K$ is the number of selected key entity tokens. If no token satisfies the condition, $T_{\text {ent}}$ is set to an all-zero vector. To take into account both the overall semantics of the copy and the key entity information, global average pooling is performed on $T_{\text {seq}}$ to obtain the sequence pooled vector $T_{\text {pool }} \in \mathrm{R}^{768}$, calculated as:

$T_{\text {pool }}=\frac{1}{64} \sum_{i=1}^{64} T_{\text {seq }}[i]$  (3)

where, $T_{\text {pool }}$ and $T_{\text {ent }}$ are concatenated along the channel dimension to obtain $T_{\text {concat }} \in \mathrm{R}^{1536}$, which is then fed into a fully connected layer for dimensionality reduction and feature fusion. The fully connected layer has an input dimension of 1536 and an output dimension of 768, uses a ReLU activation function and a dropout rate of 0.3, with weights initialized using Xavier normal initialization and bias initialized to 0, effectively avoiding gradient vanishing or explosion during training. The final output text semantic vector is $T \in \mathrm{R}^{768}$. This design strengthens key entity semantics and fuses multi-dimensional features, enabling the text vector to contain both the overall meaning of the copy and highlighted core attractiveness-related semantics. Meanwhile, precise dimensionality reduction achieves compatibility with the subsequent spatial affine modulation module, providing accurate and efficient semantic support for explicit projection of textual semantics into the image feature space.

2.2.3 Text–image spatial affine modulation

To address the core problem of disconnection and insufficient alignment between textual semantics and image features in existing multimodal fusion methods, a text–image spatial affine modulation mechanism is designed. By explicitly projecting the textual semantic vector into the image feature space, a modulation tensor with the same size as the image feature map is generated, realizing channel-wise dynamic recalibration of image features by textual semantics and providing precise semantic guidance for subsequent DBAFN. The schematic diagram of the mechanism is shown in Figure 2.

Figure 2. Schematic diagram of the text–image spatial affine modulation mechanism

The mechanism adopts two independent learnable linear layers, both taking the text semantic vector $T \in \mathrm{R}^{768}$ as input, to predict the scaling factor $\gamma$ and the shifting factor $\beta$, respectively. The input dimension of both linear layers is 768 and the output dimension is 512 , consistent with the channel number of the image feature map $F_I$, ensuring matching modulation dimensions. The weights are initialized using Xavier normal initialization, and the bias is initialized to 0 . They are updated independently during training to ensure modulation flexibility. To realize precise modulation of each spatial position of the image feature by textual semantics, a broadcasting mechanism is used to replicate and expand $\gamma \in \mathrm{R}^{512}$ and $\beta \in \mathrm{R}^{512}$ along the spatial dimension, generating modulation tensors $M_\gamma \in \mathrm{R}^{12 \times 12 \times 512}$ and $M_\beta \in \mathrm{R}^{12 \times 12 \times 512}$, without additional interpolation or cropping operations. This ensures complete consistency with the size of $F_I$ and effectively preserves the spatial structural integrity of image features. Finally, channel-wise recalibration of image features is completed through element-wise operations. The core calculation formula is as follows: 

$F_I^{\prime}=M_\gamma \odot F_I+M_\beta$       (4)

where, $\odot$ denotes element-wise multiplication, and $F^{\prime}{ }_I \in \mathrm{R}^{12 \times 12 \times 512}$ is the image feature map after text modulation. This modulation process realizes explicit mapping of textual semantics into the image space through text-driven channel scaling and shifting, making the channel responses of image features highly correlated with core textual semantics. This enables the subsequent attention mechanism to accurately capture visual regions corresponding to textual semantics, fundamentally solving the problem of disconnection and insufficient alignment between semantic and visual features in traditional multimodal fusion, and providing semantically enhanced high-quality image feature inputs for DBAFN.

2.3 Dual-branch attention fusion network

2.3.1 Global attention branch

Figure 3 shows the structure diagram of the adaptive gated DBAFN. The purpose of setting the global attention branch is to realize deep interaction between global image visual features and textual semantics, accurately capture key visual attributes such as global composition style, color distribution, and overall layout of advertising images, and solve the limitation that traditional global attention branches lack semantic guidance and only focus on the image itself while ignoring textual semantic association, thereby providing high-quality global feature support for dual-branch fusion. First, adaptive average pooling is applied to the text-modulated image feature map $F^{\prime}{ }_I$. The pooling kernel size is consistent with the spatial size of $F^{\prime}{ }_I$, and the spatial dimension is compressed to $1 \times 1$, obtaining the global feature vector $f_{\text {gap}} \in \mathrm{R}^{512}$ containing global semantic information of the image. This design maximally preserves global visual attributes while avoiding spatial information redundancy. To strengthen the guiding effect of textual semantics on global visual features, $f_{\text {gap}}$ is concatenated with the text semantic vector $T$ along the channel dimension to obtain $f_{\text {concat}} \in \mathrm{R}^{1280}$, which is then fed into a multi-head self-attention module to realize deep interaction between global image features and textual semantics. The module sets 4 attention heads, each head with hidden dimension 128, total hidden dimension 512, and dropout rate 0.3, and adopts scaled dot-product attention. The core formulas are as follows:

$M H S A(Q, K, V)$=$Concat (head_1, head_2, head_3, head_4) W_O$  (5)

$\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$    (6)

where, $Q, K$, and $V$ are generated from $f_{\text {concat}}$ through three independent linear layers, with input dimension 1280 and output dimension 512; $d_k=128$ is the dimension of each attention head, which can effectively alleviate gradient vanishing during training; $W_O$ is the output projection matrix with dimension $512 \times 512$. The output of the multi-head selfattention module is processed by layer normalization along the channel dimension to obtain the global semantic vector $f_{\text {global}} \in \mathrm{R}^{512}$. This vector not only covers global visual style information of the image, but also integrates global guidance of textual semantics, realizing deep collaboration between global visual features and textual semantics, laying the foundation for subsequent adaptive gated fusion, and effectively improving semantic relevance and representation capability of global features.

Figure 3. Structure diagram of the adaptive gated dual-branch attention fusion network (DBAFN)

2.3.2 Local branch

The core innovation of the local branch lies in adopting a cross-modal dot-product attention mechanism to achieve precise localization of local semantic regions under textual semantic guidance, solving the limitations of traditional local branches such as lack of semantic association in localization, susceptibility to background interference, and inability to accurately capture core attractiveness regions of advertisements, and focusing on strengthening local feature representation highly related to textual semantics. First, channel mapping is performed on the text-modulated image feature map $F_I^{\prime}$, and two independent $1 \times 1$ convolution layers are used to generate the key matrix $K$ and value matrix $V$. Both convolution layers are configured with 512 convolution kernels, stride 1 , padding 0 , and use ReLU activation function and dropout rate of 0.3 , effectively avoiding overfitting and optimizing feature representation. The generated $K$ and $V$ are both $\mathrm{R}^{12 \times 12 \times 512}$, which are used to capture feature information at each spatial position of the image and store feature response values, respectively. To adapt to dot-product attention computation, $K$ and $V$ are reshaped into two-dimensional matrices of $R^{144 \times 512}$, where 144 is the total number of spatial pixels of the image feature map. The generation of the query vector $q$ incorporates textual semantic guidance. Global average pooling is applied to $F_{I^{\prime}}$ to obtain the pooled vector $f_{\text {pool}} \in \mathrm{R}^{512}$, which is then passed through a linear layer to output $q \in \mathrm{R}^{512}$. The linear layer has input and output dimensions of 512, uses ReLU activation function and dropout rate of 0.3, enabling $q$ to integrate global information of the textmodulated image and ensuring that attention localization is highly correlated with textual semantics. The cross-modal attention weights are computed through the dot-product attention mechanism. The core formula is as follows:

$\alpha=\operatorname{softmax}\left(\frac{q^T K}{\sqrt{C}}\right)$ (7)

where, $C=512$ is the channel number. The scaling operation avoids excessively large dot-product results leading to Softmax saturation. The normalized attention weight vector $\alpha \in \mathrm{R}^{144}$, and its element values directly reflect the correlation between each spatial position and textual semantics. Larger values indicate stronger correlation. $\alpha$ is reshaped into an attention map of $\mathrm{R}^{12 \times 12}$, visually presenting the correspondence between image regions and textual semantics. The reshaped $V_{\text {reshape }} \in \mathrm{R}^{12 \times 12 \times 512}$ is multiplied element-wise with $A$ to obtain the local branch output feature map $F_{\text {local }} \in \mathrm{R}^{12 \times 12 \times 512}$, namely:

$F_{\text {local }}=A \odot V_{\text {reshape }}$ (8)

This design effectively strengthens local region features related to text and suppresses irrelevant background interference through text-guided attention weight allocation, achieving precise localization of core attractiveness regions of advertisements and providing local feature support focused on textual semantics for subsequent dual-branch fusion.

2.3.3 Adaptive gated fusion module

The adaptive gated fusion module is introduced to realize pixel-level adaptive fusion of global semantic features and local semantic features, dynamically balancing their contribution weights at different spatial positions, solving the limitation of traditional fusion methods that cannot fully utilize the complementarity of global and local features and require manually set fixed weights, and ensuring that the fused features both preserve the integrity of global visual style and highlight the saliency of local semantic regions. First, the global semantic vector $f_{\text {global }} \in \mathrm{R}^{512}$ is spatially expanded. A fully connected layer is used to transform it into a threedimensional tensor with the same size as the local feature map $F_{\text {local}}$, i.e., $f_{\text {global,expand}} \in \mathrm{R}^{12 \times 12 \times 512}$. The fully connected layer has an input dimension of 512 and an output dimension of 73728, uses a ReLU activation function and a dropout rate of 0.3 , effectively achieving spatial dimensional alignment between global features and local features and providing a basis for pixel-level fusion. Then, $f_{\text {global,expand}}$ and $F_{\text {local }}$ are concatenated along the channel dimension to obtain $F_{\text {fusion,concat }} \in \mathrm{R}^{12 \times 12 \times 1024}$, which is fed into two cascaded $3 \times 3$ convolution layers to generate the spatial gating weight map. The first convolution layer has 512 convolution kernels, stride 1 , and padding 1 to maintain the output size consistent with the input, using a ReLU activation function and a dropout rate of 0.3 . The second convolution layer has 1 convolution kernel, stride 1 , padding 1 , and no activation function, with output dimension $\mathrm{R}^{12 \times 12 \times 1}$. The output of the second convolution layer is normalized by a Sigmoid function to obtain the spatial gating weight map $G \in \mathrm{R}^{12 \times 12 \times 1}$, whose value range is $[0,1]$, dynamically reflecting the dependence of each spatial position on local features and global features. Finally, pixel-level weighted fusion is performed to obtain the fused feature map $F_{\text {fuse }} \in \mathrm{R}^{12 \times 12 \times 512}$ with both global and local advantages. The core calculation formula is as follows:

$F_{\text {fuse }}=G \bigodot F_{\text {local }}+(1-G) \odot f_{\text {global, expand }}$  (9)

This fusion mechanism requires no manual intervention and adaptively matches feature requirements at different spatial positions through the gating weight map. In background regions of advertising images, the weight tends to 0, emphasizing retention of global visual style and color layout information; in core regions such as product subjects and price labels, the weight tends to 1, strengthening the expression of local semantic features. Thus, precise complementary fusion of global and local features is achieved, significantly improving the semantic representation capability and discrimination performance of fused features, and providing high-quality feature inputs for subsequent attractiveness prediction and interpretable heatmap generation.

2.4 Attractiveness prediction and interpretable heatmap generation

2.4.1 Visual attractiveness score prediction

The core innovation of visual attractiveness score prediction lies in achieving high-precision score regression based on fused feature maps that contain both global visual style and local semantic features. Meanwhile, through targeted network design, the predicted scores are ensured to be consistent with the range of human subjective evaluation, and the gradient instability problem during training is solved, providing reliable guarantees for prediction accuracy. First, global average pooling is applied to the fused feature map $F_{\text {fuse }} \in \mathrm{R}^{12 \times 12 \times 512}$. The pooling kernel size is consistent with the spatial size of the feature map, compressing the spatial dimension to $1 \times 1$ and obtaining the fused feature vector $f_{\text {fuse,pool}} \in \mathrm{R}^{512}$. This vector fully integrates global visual attractiveness information and local semantic attractiveness information of advertising images, effectively avoiding prediction bias caused by single feature dimensions. Then, the feature vector is fed into two cascaded fully connected layers to complete attractiveness score regression. The first fully connected layer has input dimension 512 and output dimension 256, uses a ReLU activation function and a dropout rate of 0.3, achieving feature dimensionality reduction and redundancy suppression, and optimizing feature representation. The second fully connected layer has input dimension 256 and output dimension 1, with no activation function, directly outputting the regression value. To make the predicted score conform to the [0,1] range of human subjective evaluation, the output of the second fully connected layer is fed into a Sigmoid function to obtain the final visual attractiveness score s. The core calculation formula is as follows:

$s=\operatorname{Sigmoid}\left(F C 2\left(F C 1\left(f_{\text {fuse,pool}}\right)\right)\right)$    (10)

The weights of both fully connected layers are initialized using Xavier normal initialization, and the bias is initialized to 0, ensuring parameter stability at the early stage of training. During training, a gradient clipping strategy is adopted, with clipping threshold set to 1.0, effectively avoiding gradient explosion and ensuring convergence stability of the model. This design achieves high-precision prediction of visual attractiveness scores through full utilization of fused features, precise adaptation of score range, and optimization of training strategy. At the same time, it collaborates with the subsequent interpretable heatmap generation module to form an integrated “prediction–interpretation” technical path, which is different from existing prediction methods that can only output scores.

2.4.2 Interpretable heatmap generation

In order to achieve accurate generation of high-resolution heatmaps, a lightweight decoder is further designed in this paper, and a dual-constraint mechanism is introduced to improve the localization accuracy and sparsity of heatmaps, solving the limitations of existing interpretability methods such as large localization deviation, unclear key regions, and inability to accurately match advertising scenario requirements, and realizing visualization explanation of attractiveness prediction results to provide intuitive support for advertising design optimization. The detailed flowchart is shown in Figure 4. The decoder adopts a lightweight structure combining transposed convolution and standard convolution, containing four convolution layers in total, which gradually increase the spatial resolution of the fused feature map to be consistent with the original input image size of 384×384, while effectively preserving feature semantic representation ability. This structure performs two upsampling operations through transposed convolution layers and feature optimization and dimensionality reduction through standard convolution layers. The transposed convolution layers use appropriate kernel size, stride, and padding parameters to ensure no obvious feature distortion during upsampling. The standard convolution layers suppress overfitting and enhance key feature representation through dimensional adjustment and dropout regularization. The final original heatmap output by the decoder is normalized by the Softmax function to obtain the attractiveness heatmap, whose pixel value range is [0,1], and the value magnitude directly reflects the contribution degree of the corresponding image region to visual attractiveness. To further improve the localization accuracy and sparsity of the heatmap, and to adapt to the sparse characteristics of key regions of advertising attractiveness, a dual-constraint mechanism is introduced. The first is KL divergence loss, which uses the real eye-tracking saliency map as supervision signal to force the predicted heatmap to be precisely aligned with real eye-tracking data, improving localization accuracy. The second is L1 sparsity regularization, which is applied on the output feature map of convolution layer 1 of the decoder. By calculating the sum of absolute values of all elements in this feature map, the model is encouraged to activate only a small number of key regions and suppress irrelevant background interference. The core calculation formula of the sparsity regularization loss is as follows:

$L_{\text {sparse }}=\sum_{i=1}^{24} \sum_{j=1}^{24} \sum_{k=1}^{128}\left|F_{\text {conv } 1}(i, j, k)\right|$  (11)

where, $F_{\text {conv}1}(i, j, k)$ represents the element value at the corresponding position and channel of the output feature map of convolution layer 1. This design, through the synergy of the efficient upsampling of the lightweight decoder and the dualconstraint mechanism, enables the generated heatmap to not only accurately locate the core attractive regions in advertising images, but also has good sparsity and interpretability. It not only differs from the computational inefficiency problem caused by existing complex decoders, but also solves the defects of traditional heatmaps such as unclear localization and serious background interference, achieving dual optimization of interpretability and efficiency.

Figure 4. Flowchart of interpretable heatmap generation mechanism

2.5 Multi-task joint loss function

To achieve end-to-end model training and balance visual attractiveness prediction accuracy, heatmap localization accuracy, fused feature discriminability, and heatmap sparsity, a multi-task joint loss function is designed in this paper. Regression loss, KL divergence loss, sparse regularization loss, and contrastive learning loss are organically integrated. By reasonably setting hyperparameters to balance the weight of each loss term, overall optimization of model performance is achieved. The core innovation lies in breaking the limitation of a single loss and constructing a multi-objective collaborative optimization loss system, solving the problem that existing loss functions cannot balance multi-dimensional performance requirements. The total loss function is defined as follows:

$L_{\text {total }}=L_{\text {reg }}+\lambda_1 L_{K L}+\lambda_2 L_{\text {sparse }}+\lambda_3 L_{\text {contrast }}$     (12)

where, $\lambda_1, \lambda_2, \lambda_3$ are hyperparameters used to adjust the contribution weights of each loss term. After multiple rounds of experiments, the optimal values are 0.5, 0.01, and 0.1 respectively. This setting can achieve dynamic balance among loss terms, avoid a single loss term dominating the training process, and ensure simultaneous improvement of multi-dimensional performance of the model.

Regression loss adopts smooth L1 loss, which is used to optimize the regression accuracy of attractiveness scores and solve the problems that ordinary L1 loss is sensitive to outliers and L2 loss is prone to gradient explosion. The calculation formula is as follows:

$L_{\text {reg }}=\frac{1}{N} \sum_{i=1}^N s \operatorname{mooth}_{L 1}\left(s_i-s_{i, g t}\right)$ (13)

$\operatorname{smooth}_{L 1}(x)=\left\{\begin{array}{c}\frac{1}{2} x^2,|x| \leq \delta \\ \delta\left(|x|-\frac{1}{2} \delta\right),|x|>\delta\end{array}\right.$  (14)

where, $N$ is the batch size, set to $32 ; s_i$ and $s_{i, g t}$ are the predicted and ground truth attractiveness scores of the $i$-th sample, respectively. The ground truth scores are averaged from 3 professional evaluators and normalized to $[0,1] ; \delta$ is set to 0.1 to ensure stability of loss computation. KL divergence loss is used to optimize heatmap localization accuracy and measure the distribution difference between predicted heatmaps and real eye-tracking saliency maps. The formula is:

$L_{\mathrm{KL}}=\frac{1}{N} \sum_{i=1}^N \sum_{x=1}^{384} \sum_{y=1}^{384} H_{\mathrm{gt}, i}(x, y) \log \left(\frac{H_{\mathrm{gt}, i}(x, y)+\epsilon}{H_{\mathrm{pred}, i}(x, y)+\epsilon}\right)$ (15)

where, $\epsilon=1 e-8$, used to avoid zero values in logarithm operations and ensure validity of loss computation. Sparse regularization loss is used to enhance heatmap sparsity and force the model to focus on a small number of key attractive regions. Its formula has been given in detail in Section 2.4.2. Here the weight $\lambda_2=0.01$, which can achieve sparsity constraint without excessively suppressing feature representation of the model.

The core innovation of contrastive learning loss is to guide the model to learn feature differences of advertising images with different attractiveness levels by constructing anchor–variant pairs, and enhance the discriminability of fused features, solving the problem of insufficient discrimination ability of existing models for different attractiveness levels. Specifically, two variant samples are generated for each training sample. The high-attractiveness variant is generated by optimizing visual elements to increase attractiveness, and the low-attractiveness variant is generated by destroying visual elements to decrease attractiveness, ensuring that variants only change attractiveness-related features while other features remain consistent. Fused feature vectors of anchor, high-attractiveness variant, and low-attractiveness variant are extracted respectively, and Information Noise-Contrastive Estimation (InfoNCE) loss is used as contrastive learning loss to pull closer the feature distance between anchor and high-attractiveness variant, and push away the feature distance between anchor and low-attractiveness variant. The formula is as follows:

$\begin{gathered}L_{\text {contrast }}=\frac{1}{N} \sum_{i=1}^N- \log \left(\frac{\exp \left(\operatorname{sim}\left(f_{\text {anchor}, i}, f_{\text {pos}, i}\right) / \tau\right)}{\exp \left(\operatorname{sim}\left(f_{\text {anchor}, i}, f_{\text {pos}, i}\right) / \tau\right)+\exp \left(\operatorname{sim}\left(f_{\text {anchor}, i}, f_{\text {neg}, i}\right) / \tau\right)}\right)\end{gathered}$          (16)

where, $\operatorname{sim}(a, b)=a \cdot b /\|a\|\|b\|$ is cosine similarity, and $\tau=0.1$ is the temperature coefficient used to adjust similarity distribution, avoid gradient vanishing, and effectively enhance discriminability of fused features.

To adapt to the multi-task joint loss function, ensure stable convergence of the model, and improve generalization ability, a targeted training strategy is designed. The model uses Adaptive Moment Estimation with Weight Decay (AdamW) optimizer, initial learning rate is set to 1e-4, weight decay coefficient is 1e-5 to suppress overfitting, momentum parameters are $\beta_1=0.9$ and $\beta_2=0.999, \varepsilon=1 e-8$; batch size is 32, training epochs are 50. A learning rate scheduling strategy is adopted, where the learning rate is reduced to 1/10 of the original at the 30th and 40th epochs respectively. An early stopping strategy is introduced, using validation mean absolute error (MAE) as evaluation metric; if validation MAE does not decrease for 5 consecutive epochs, training is stopped and the best model parameters are saved. During training, data augmentation strategies such as random flipping, random cropping, and color jittering are used to further improve model generalization ability. This training strategy is deeply adapted to the multi-task joint loss function, and through parameter optimization and regularization measures, effectively avoids overfitting and gradient instability problems, ensuring that the model achieves optimal performance in multiple dimensions.

This paper fully verified the superiority of the proposed DBAFN model in advertising image visual attractiveness prediction and interpretability generation tasks through five groups of systematic experiments. The core of the in-depth analysis of experimental results lies in revealing the technical value of each innovative module and the essential logic of performance improvement. The core advantages of the proposed method come from the targeted design of multimodal fusion and attention mechanism. Compared with existing methods, the text–image spatial affine modulation mechanism has broken the limitation of semantic and visual feature decoupling in traditional multimodal fusion. By explicitly projecting the text semantic vector into the image feature space, it achieved deep alignment between text and visual information, making the channel response of image features highly correlated with the core semantics of text, which is also the core reason for the excellent performance of the proposed method in benchmark performance and cross-dataset generalization experiments. The collaborative effect of the DBAFN architecture and the adaptive gating module effectively solved the problem that a single attention branch cannot balance global and local features. The global branch captured the overall visual style and layout of advertisements, the local branch precisely located core semantic regions related to text, and the adaptive gate dynamically balanced the contributions of both, so that the fused features have both global completeness and local saliency. Further analysis of different advertising scenarios shows that e-commerce advertisements in clothing and food categories achieved the best performance due to simple visual elements and clear textual semantics; home appliance advertisements have slightly higher MAE due to complex product structures and increased difficulty in local localization; long-text advertisements suffer from insufficient semantic extraction due to length constraints, leading to reduced prediction accuracy. For this problem, segmentation encoding and semantic aggregation strategies can be used to improve adaptability in long-text scenarios.

Objectively speaking, the proposed method still has certain limitations, which are also key directions for future research. First, in the text preprocessing stage, the document length is fixed. Although this ensures model training efficiency and input dimensional consistency, it does not fully capture the semantics of long-text advertisements and cannot fully extract deep semantic information in complex marketing copy, affecting multimodal fusion accuracy. Second, the heatmap generation still has localization deviation in complex background advertisements. When advertisements contain multiple visual interference elements, the collaborative constraint effect of sparse regularization and KL divergence loss is weakened, and the localization accuracy of key regions needs further improvement. In addition, the model includes dual-branch attention and multi-task loss computation, resulting in slightly higher computational cost than lightweight comparison methods, making it difficult to directly deploy on resource-constrained scenarios such as mobile devices. To address the above limitations, future work will improve from three aspects: introducing a dynamic text length adaptation mechanism to adapt to advertising copy of different lengths through semantic segmentation and adaptive padding strategies; optimizing the decoder structure by adding attention-guided upsampling modules to enhance feature response of key regions under complex backgrounds and improve heatmap localization accuracy; and adopting model pruning and quantization techniques to compress network parameters, reduce computational cost, and achieve lightweight model deployment.