Artistic Image Style Recognition Based on Multimodal Visual Features

2.1 Overall framework of the method

The proposed method for artistic image style recognition adopts an end-to-end deep learning framework. It consists of three core components: multimodal feature extraction branches, cross-modal attention fusion module, and classification optimization module. These modules work collaboratively to realize the complete recognition process from image input to style category output. The overall framework of the method is shown in Figure 1. The input artistic image is first fed into four parallel feature extraction branches, which extract global semantic, local texture, color distribution, and frequency-domain features, providing comprehensive and complementary style representations for the subsequent fusion process. Then, these multimodal features are input into the cross-modal attention fusion module, which deeply explores the intrinsic correlations between modalities and adaptively learns the weights of each modality to generate discriminative fused features. Finally, the fused features are sent into the classification optimization module, mapped by the classification head into the probability distribution of each art style, and constrained by a joint loss function during model training to further optimize feature discriminability and model generalization capability.

Figure 1. Overall framework of the proposed artistic image style recognition method

2.2 Design of multimodal feature completeness

To comprehensively characterize the multidimensional attributes of artistic styles and overcome the limitations of single-modal and incomplete feature representation in existing methods [19], this paper introduces frequency-domain features based on discrete wavelet transform into artistic image style recognition for the first time, constructing a four-dimensional complementary modality representation consisting of global semantic, local texture, color distribution, and frequency-domain features, achieving a comprehensive description of artistic styles from macro to micro, from spatial to frequency domain. Figure 2 shows the structure of the frequency-domain feature extraction module based on discrete wavelet transform. The core innovation of this design is to supplement spatial features with frequency-domain features that can capture scale structure information which spatial features cannot, addressing the problem that traditional multimodal methods lack the frequency-domain dimension and cannot distinguish intrinsic structural differences in style, forming a multidimensional and non-redundant style feature representation system.

The frequency-domain feature branch is the core innovation of this design, and its specific implementation process is as follows: A one-level two-dimensional discrete wavelet transform is applied to the input image using the Haar wavelet basis, decomposing the image into four sub-bands: low-frequency approximation component, horizontal detail component, vertical detail component, and diagonal detail component. Each sub-band has a size of 1/4 of the original image, corresponding to the overall contour and details in different directions. Each sub-band is independently input into a three-layer convolutional network for feature extraction. The convolution layers all use 3 × 3 kernels, stride 1, and padding 1, with output channels of 64, 128, and 256 respectively. Rectified Linear Unit (ReLU) activation function is applied to introduce nonlinearity, and a batch normalization layer is added after each layer to suppress overfitting. The feature maps of the four sub-bands are concatenated along the channel dimension, transformed into a fixed-length feature vector via global average pooling, and then compressed through a fully connected layer with output dimension of 256, generating the final frequency-domain feature vector. The generation process of the frequency-domain feature vector can be expressed as:

$f_f=\operatorname{FC}\left(\operatorname{GAP}\left(\operatorname{Concat}\left(C_{L L}, C_{L H}, C_{H L}, C_{H H}\right)\right)\right)$          (1)

where, C_LL, C_LH, C_HL, and C_HH are the feature maps extracted by the convolutional network from the four sub-bands, Concat represents concatenation along the channel dimension, GAP is global average pooling, FC is fully connected layer, and $f_f \in R^{256}$is the final output frequency-domain feature vector.

The other three feature branches serve as auxiliary supplements, forming complementary collaboration with the frequency-domain feature to further improve the completeness of multimodal representation. The global semantic branch uses a fine-tuned Residual Network 50-layer (ResNet50) as the backbone network. After removing the top classification layer, a global semantic feature vector of dimension 2048 is output via global average pooling to characterize the overall composition and macroscopic style tendency of the image. The local texture branch combines learnable local binary pattern (LBP) with a Transformer encoder, outputting a local texture feature vector of dimension 512, focusing on capturing micro-level texture details such as artistic brushstrokes. The color feature branch converts the image to Hue, Saturation, Value (HSV) space, combined with color moments and color attention mechanism, outputting a color feature vector of dimension 128 to characterize the color tone preference of artistic styles. The dimensions of the four feature vectors are 2048, 512, 128, and 256 respectively, covering different dimensions of style including macro, micro, color tone, and structure, and achieve comprehensive and accurate characterization of artistic styles through complementary collaboration, providing high-quality feature foundation for subsequent cross-modal fusion.

Figure 2. Structure of the frequency-domain feature extraction module based on discrete wavelet transform

2.3 Cross-modal attention fusion mechanism

To fully explore the intrinsic correlations between multimodal features and to address the limitations of existing fusion methods with fixed weights and insufficient modality interaction [20], and to achieve efficient complementarity and adaptive fusion of multimodal features, this paper proposes an adaptive fusion module based on multi-head cross-attention. The schematic diagram of the module is shown in Figure 3. The core innovation of this module is that, with multi-head cross-attention as the core, it dynamically learns the importance weights of each modality feature, strengthens the contribution of key modality information, suppresses redundant information interference, and deepens the interaction degree between modalities through multiple iterations, generating discriminative fused features that provide high-quality feature support for subsequent style classification.

Figure 3. Adaptive fusion mechanism based on multi-head cross-attention

Feature projection is the basic prerequisite for implementing cross-modal fusion. Its purpose is to map multimodal features of different dimensions into the same feature space, eliminating fusion bias caused by dimensional differences. Let the original feature vectors of the four modalities, global semantic, local texture, color, and frequency-domain, be $f_s \in R^{2048}, f_t \in R^{512}, f_c \in R^{128}$ and $f_f \in R^{256}$, respectively. Through independent linear transformations, they are mapped to the target dimension d=256, obtaining the projected features $z_t, z_c, z_f \in R^{256}$. The linear transformation is calculated as:

$z_m=f_m W_m+b_m$         (2)

where, $m \in\{s, t, c, f\}$ corresponds to the four modalities, $W_m$ is the linear projection matrix with dimensions 2048 × 256, 512 × 256, 128 × 256, 256 × 256, respectively, $b_m \in R^{256}$ is the bias term, adaptively learned through model training. The projected features preserve the core style information of each modality and provide a unified feature basis for subsequent cross-attention computation.

Multi-head cross-attention calculation is the core innovation of this module. By constructing a "query-key-value" interaction pattern, it achieves deep correlation mining between modalities and adaptive weight allocation. Each modality’s projected feature is used as a query, and the projected features of the other three modalities are used as keys and values simultaneously, constructing four cross-attention structures. Each attention structure uses an 8-head multi-head attention mechanism, with each head having a feature dimension of 256/8 = 32. Taking the global semantic feature as the query as an example, its attention computation process is as follows: first, the query, key, and value are mapped to the same dimension through learnable matrices, i.e., $Q_s=z_s W_Q$, $K_m=z_m W_K$, $V_m=z_m W_V$, where W_Q,W_K,$W_V \in R^{256 \times 256}$ are learnable parameters; then, the similarity between the query and each key is computed and normalized through softmax to obtain attention weights, and finally the attention output is calculated by combining the value:

$\operatorname{Att}_{s \rightarrow m}=\operatorname{softmax}\left(\frac{Q_s K_m^T}{\sqrt{d}}\right) V_m$         (3)

where, $d$ is the scaling factor, used to alleviate bias in similarity computation caused by increased dimensionality. The attention outputs corresponding to the three modalities are concatenated, passed through a linear transformation and ReLU activation function, and the fused feature dominated by global semantic features $\tilde{Z}_S$ is obtained. Using the same process, local texture, color, and frequency-domain features are used as queries to obtain the corresponding fused features $\tilde{Z}_t, \tilde{Z}_c, \tilde{Z}_f$.

To further deepen the interaction between modalities and enhance the discriminability of fused features, the above cross-attention computation process is performed twice iteratively, updating the fused features of each modality in each iteration to strengthen complementary information between modalities. After iteration, the four modality-dominated fused features $\tilde{Z}_s, \tilde{Z}_t, \tilde{Z}_c, \tilde{Z}_f$ are concatenated along the channel dimension, forming a feature vector of dimension 4 × 256 = 1024, which is then input into a feedforward network for feature optimization and dimensionality reduction. The feedforward network consists of two fully connected layers, with a hidden layer dimension of 512, using ReLU activation and a dropout layer (dropout rate 0.5) to suppress overfitting. The output layer reduces the feature dimension to 256, obtaining the final multimodal fused feature $F \in R^{256}$, calculated as:

$F=F C_2\left(\operatorname{ReLU}\left(F C_1\left(\operatorname{Concat}\left(\tilde{z_s}, \tilde{z_t}, \tilde{z_c}, \tilde{z_f}\right)\right)\right)\right)$        (4)

Compared with traditional simple concatenation or weighted summation fusion methods, the proposed cross-modal attention fusion mechanism has significant advantages: it can dynamically learn the importance weights of each modality, adaptively adjust the contribution of different modalities according to the style characteristics of the input image, and capture the intrinsic correlations between modalities through cross-attention, effectively suppressing redundant information and highlighting feature cues that play a key role in style recognition, thereby significantly enhancing the discriminability of fused features and providing a foundation for improving subsequent style classification performance.

2.4 Local texture enhancement strategy

The micro-level textures such as brushstrokes and surface texture in artistic images are the core cues for distinguishing different art styles. Traditional local texture extraction methods are difficult to adaptively capture the complex and variable brushstroke patterns in artistic images [21], and cannot effectively mine long-range dependencies between local patches. To address this problem, this paper proposes a local texture enhancement strategy. The core innovation lies in combining learnable LBP with a Transformer encoder to achieve precise capture and deep representation of micro-level textures in artistic images, improving the network’s sensitivity to style details and providing high-quality local texture support for multimodal feature fusion. Figure 4 shows the local texture enhancement strategy and feature extraction process.

The learnable LBP is the core improvement of this strategy, aiming to overcome the limitations of traditional LBP being non-differentiable, unable to train end-to-end, and having limited feature representation capability, enabling adaptive learning of artistic brushstroke textures. In this paper, the traditional LBP is extended to a differentiable continuous form, using convolutional layers to learn the weight distribution of neighboring pixels instead of using fixed binary coding based on grayscale comparison. Specifically, a 3 × 3 convolution kernel corresponding to an 8-neighborhood structure is used, with the convolution kernel parameters as learnable weights, convolving the local region of the input image to generate a continuously valued texture response map. Its computation process can be expressed as:

$T(x, y)=\sum_{i=0}^7 w_i \cdot I\left(x+i_x, y+i_y\right)$         (5)

where, $T(x, y)$ is the response value at coordinate $(x, y)$ in the texture response map, $w_i$ is the learnable neighborhood weight, $I\left(x+i_x, y+i_y\right)$ is the grayscale value of the corresponding pixel in the neighborhood, and $i_x, i_y$ are the coordinate offsets of the neighboring pixel relative to the center pixel. This design adaptively adjusts neighborhood weights through model training, accurately capturing unique brushstroke patterns of different art styles. Compared with traditional LBP with fixed coding, it is more flexible and can effectively adapt to complex textures such as Van Gogh’s swirling brushstrokes and Monet’s pointillism, and its differentiable property allows integration into end-to-end deep learning frameworks for collaborative optimization with subsequent modules.

Figure 4. Local texture enhancement strategy and feature extraction process

To further explore long-range dependencies between local texture patches and strengthen the global correlation of texture features, after generating the texture response map by learnable LBP, a lightweight Transformer encoder is introduced for feature enhancement. The texture response map is first divided into 16 × 16 local image patches, each patch is flattened into a one-dimensional feature vector to form the input sequence. The encoder has six layers, each consisting of a multi-head self-attention mechanism and a feedforward network. The multi-head self-attention is designed with 8 heads, each head having a feature dimension of 64, hidden layer dimension of 256, and the feedforward network uses ReLU activation. A dropout layer with dropout rate 0.5 is added in each layer to suppress overfitting. The Transformer encoder computes the correlation weights between different local patches through self-attention, capturing cross-region texture dependencies. The output sequence of the encoder is transformed into a fixed-length local texture feature vector $f_t \in R^{512}$ through mean pooling, expressed as:

$f_t=\operatorname{MeanPool}(\operatorname{Transformer}(\operatorname{Patch}(T(x, y))))$        (6)

where, Patch represents the image patch division and flattening operation, Transformer is the 6-layer lightweight encoder, and MeanPool is the mean pooling operation.

This local texture enhancement strategy has good adaptability to artistic images and can accurately capture micro-level texture features of different art styles: for Van Gogh’s swirling brushstrokes, the learnable LBP strengthens the response of swirling textures by adjusting neighborhood weights, and the Transformer encoder captures cross-region correlations of the swirling textures; for Impressionist pointillism brushstrokes, this strategy effectively distinguishes the density and distribution characteristics of pointillist strokes, compensating for the insufficient capture of micro-level details by global semantic and color modalities. The local texture feature output of this branch forms a complementary collaboration with global semantic, color, and frequency-domain features, capable of depicting micro-level brushstroke details and integrating texture information globally through the Transformer encoder, further improving the completeness of multimodal features and providing critical texture feature support for subsequent cross-modal fusion and style classification.

2.5 Contrastive learning regularization optimization

The discriminability of fused features directly determines the accuracy of artistic image style recognition, especially in similar style recognition tasks. Existing methods generally have problems of feature boundary ambiguity and insufficient discriminative capability [22, 23]. To address this problem, this paper proposes a contrastive learning regularization optimization strategy. The core innovation lies in introducing Information Noise-Contrastive Estimation (InfoNCE) contrastive loss jointly with cross-entropy loss for optimization. The main loss supervises category classification, and the regularization term constrains the feature space distribution, which together enhance the discriminability of fused features, effectively solving the core difficulty of distinguishing similar style features and providing assurance for high-precision style recognition.

The reasonable design of the loss function is the key to optimization effectiveness. The total loss function constructed in this paper is a weighted sum of cross-entropy loss and InfoNCE contrastive loss, expressed as:

$L=L_{C E}+\lambda L_{C L}$         (7)

where, $L_{C E}$ is the cross-entropy loss, serving as the main loss to supervise the model’s category classification task; $L_{C L}$ is the InfoNCE contrastive loss, serving as the regularization term to constrain the spatial distribution of fused features; $\lambda=0.1$ is the balancing coefficient, set based on multiple experimental validations. This coefficient can effectively balance the contributions of the main loss and the regularization term, avoiding underfitting caused by excessive regularization or insufficient constraint effect, ensuring the model simultaneously considers classification accuracy and feature discriminability. The specific calculation formula of the cross-entropy loss is:

$L_{\mathrm{C} E}=-\frac{1}{N} \sum_{i=1}^N \sum_{c=1}^C y_{i, c} \log \left(\hat{y}_{i, c}\right)$        (8)

where, $N$ is the training batch size, $C$ is the number of art style categories, $y_{i, c}$ is the one-hot encoding of the ground truth label of the $i$-th sample for the $c$-th style category, and $\hat{y}_{i, c}$ is the predicted score of the $i$-th sample belonging to the $c$-th style category. This loss supervises the model to learn the mapping between fused features and style categories by measuring the difference between the predicted distribution and the true distribution, ensuring the model can accurately distinguish different style categories, and serves as the core supervisory signal for style classification.

InfoNCE contrastive loss is the core innovative part of this optimization strategy. Its core function is to constrain the spatial distribution of fused features, strengthening the aggregation of same-style features and the separation of different-style features, especially improving the feature distinction ability for similar styles. The specific calculation formula is:

$L_{C L}=-\frac{1}{N} \sum_{i=1}^N \log \frac{\exp \left(\operatorname{sim}\left(F_i, F_{i+}\right) / \tau\right)}{\sum_{j=1}^N 1_{|j \neq i|} \exp \left(\operatorname{sim}\left(F_i, F_j\right) / \tau\right)}$        (9)

where, $F_i$ is the multimodal fused feature of the $i$-th sample, F_i+ is the positive sample feature belonging to the same style as $F_i$, $\operatorname{sim}($), denotes cosine similarity used to measure the similarity between two feature vectors, and $\tau=0.1$ is the temperature coefficient, set to alleviate gradient vanishing in similarity calculation and make the feature distribution more distinguishable. Positive and negative sample pairs are constructed within the training batch. Positive pairs are selected from different samples of the same style, and negative pairs are selected from samples of different styles. Under the constraint of this loss function, fused features of the same style are forced to aggregate in the embedding space, and fused features of different styles are pushed apart, effectively reducing intra-style feature differences and enlarging inter-style feature distances, especially solving the problem of ambiguous feature boundaries for similar styles.

The classification head serves as the bridge between fused features and style categories, and its structural design directly affects classification performance and model generalization ability. The classification head designed in this paper consists of two fully connected layers. The input is the multimodal fused feature $F \in R^{256}$. The first fully connected layer maps the input feature to 512 dimensions. A ReLU activation function is applied to introduce nonlinearity and enhance the feature fitting ability of the model, and a dropout layer with a dropout rate of 0.5 is added to suppress overfitting, preventing the model from over-relying on noise in training data. The second fully connected layer maps the 512-dimensional feature to the number of style categories $C$, outputting predicted scores $\hat{y}$ for each style category, serving as the basis for cross-entropy loss calculation. This classification head structure is simple and efficient. Through the synergistic effect of the dropout layer and ReLU activation, it ensures effective mapping of features while preventing overfitting. In combination with the joint loss function optimization, it further improves the accuracy and robustness of style recognition.

The proposed multimodal visual feature-based art image style recognition method has its core advantages in the synergistic effect of four innovations, effectively overcoming the key bottlenecks of existing methods and showing significant advancement in both academic innovation and performance. The integrity design of multimodal features, by introducing frequency domain features, compensates for the limitation of traditional methods that only focus on spatial features, achieving comprehensive characterization of the multidimensional attributes of artistic styles. The cross-modal attention fusion mechanism abandons the inefficient simple concatenation fusion method, and through dynamic weight allocation and deep modal interaction, maximizes the complementary value of each modality. The local texture enhancement strategy addresses the insufficient adaptability problem of traditional texture extraction methods, accurately capturing micro details such as artistic brushstrokes. The contrastive learning regularization optimization effectively strengthens feature discriminability, particularly enhancing the differentiation of similar styles. Experimental results show that the four innovations do not exist in isolation but form an organically complementary system, collectively solving the core deficiencies of existing methods such as single-modality features, low fusion efficiency, inaccurate texture capture, and blurred feature boundaries. Compared with existing similar multimodal studies, the core difference of this method is the first systematic introduction of frequency domain features into art style recognition, constructing a four-dimensional complementary modal system, and at the same time, achieving dual enhancement of feature representation and discriminability through the deep combination of cross-attention fusion and contrastive learning. Its performance advantage is particularly significant in similar style recognition tasks, enriching the research ideas and technical pathways of multimodal fusion in the field of image processing.

Although the proposed method achieves good recognition performance, there are still certain limitations, reflecting the objectivity and rigor of the study. In frequency domain feature extraction, only single-level discrete wavelet transform is used for decomposition, which can only capture single-scale frequency information, making it difficult to fully mine the structural differences of artistic styles at different scales, and leaving room for improvement in depicting details of complex styles. In terms of model efficiency, the 6-layer Transformer encoder used in the local texture enhancement strategy can effectively capture dependencies between local blocks, but its computational complexity is relatively high, affecting the inference speed and making it difficult to meet the requirements of real-time recognition scenarios. Regarding sample adaptability, model training depends on sufficient sample data, and for minority art styles with fewer samples, feature learning is insufficient, resulting in relatively lower recognition accuracy and leaving room for improvement in generalization ability. In addition, the robustness of the model to image noise and resolution differences has not been systematically tested, and its adaptability in complex real-world scenarios requires further verification.

In response to the limitations of the proposed method and in combination with research hotspots in the field of image processing, the following specific future research directions are proposed to provide clear extension ideas for subsequent studies. To address the scale limitation of frequency domain feature extraction, multi-scale discrete wavelet transform or wavelet packet transform can be introduced to mine frequency information at different scales and bands, further improving the integrity of multimodal features and enhancing the ability to depict complex artistic styles. To address the problem of high computational complexity, lightweight Transformer structures can be explored, through pruning, knowledge distillation, or using hybrid Convolutional Neural Network (CNN)-Transformer structures, reducing model parameters and computation while ensuring recognition performance, improving inference speed and adapting to real-time application scenarios. To address the insufficient adaptability to minority style samples, Few-shot learning, meta-learning, or similar techniques can be introduced, allowing efficient learning from few samples to enhance the model’s recognition ability for minority styles and expand the applicability of the method. In addition, future research can further study the robustness of the model to noise and resolution differences, optimize the feature extraction process of each feature branch using attention mechanisms, and explore the integration of multimodal features with prior knowledge in the art domain, further enhancing the academic value and practical applicability of the method.