A Multimodal Image Feature Fusion Approach for Crop Health Assessment and Agricultural Yield Prediction

2.1 Overview of the overall framework

This paper proposes an end-to-end algorithm framework combining multimodal adaptive fusion and ST-GCN for crop health status recognition and yield prediction. The framework takes image processing technology as the core support and runs through the whole process of feature extraction, fusion, segmentation, and spatiotemporal modeling. It mainly includes three closely related core stages. In the multimodal feature extraction stage, targeted multi-scale features are extracted according to the characteristics of different modal images, providing a high-quality basis for subsequent fusion; in the cross-modal adaptive fusion and health status recognition stage, the extracted heterogeneous features are deeply fused and pixel-level health status analysis is realized, providing spatiotemporal distribution information for yield prediction; in the spatiotemporal yield prediction stage, the health status features are transformed into a spatiotemporal graph structure and combined with environmental factors to complete yield regression. The three stages form a closed-loop integrated modeling, breaking through the problems of information loss and low efficiency caused by traditional step-by-step processing, realizing end-to-end optimization from multimodal image input to yield prediction output, and taking into account processing accuracy, computational efficiency, and model interpretability.

2.2 Multimodal feature extraction

RGB images carry key phenotypic features of crops such as color, texture, and morphology, and are the core visual basis for crop health status recognition. The targeting and accuracy of feature extraction directly determine the subsequent fusion effect and recognition performance [16, 17]. Traditional backbone networks generally have problems of insufficient capture of texture details and insufficient scale adaptability of phenotypic features in agricultural image processing. Therefore, this paper customizes and improves the Residual Network with Split-Attention Blocks (ResNeSt) network and uses it as the feature extraction encoder for RGB images. The core innovation focuses on the structural optimization of the Split-Attention module. By reconstructing the internal channel interaction and weight allocation mechanism of the module, the selective capture ability of crop phenotypic features is strengthened, providing hierarchical and discriminative RGB basic features for cross-modal fusion.

The optimization of the Split-Attention module is mainly reflected in the dual improvement of the channel grouping strategy and the attention weight calculation method. For the multi-scale characteristics of crop phenotypic features, the traditional fixed channel grouping mode is abandoned, and an adaptive grouping strategy matching the phenotypic scale of crops is adopted. According to the scale differences of crop leaf texture and canopy morphology in RGB images, the number of groups is dynamically adjusted to ensure that phenotypic features at different scales can be accurately extracted. In the attention weight calculation, a phenotypic feature correlation factor is introduced to optimize the weight allocation logic. The core calculation formula is:

${{\text{ }\!\!\alpha\!\!\text{ }}_{\text{i}}}\text{= }\!\!\sigma\!\!\text{ (W}\cdot \text{GAP(}{{\text{F}}_{\text{i}}}\text{)+b)}$         (1)

where, α_i represents the attention weight of the i-th channel group, GAP( ) denotes the global average pooling operation, Fᵢ is the feature map of the i-th channel group, W is the learnable weight matrix, σ is the Sigmoid activation function, and b is the bias term. This improvement makes the attention weights more inclined to channels with significant crop phenotypic features, effectively suppressing background noise interference, solving the problems of texture detail loss and insufficient feature discrimination of the traditional ResNeSt network in agricultural images, and finally outputting multi-scale RGB feature maps covering shallow texture, middle-level morphology, and deep semantic features, providing high-quality support for subsequent cross-modal feature fusion.

Multispectral images contain continuous band information, which can reflect physiological parameters such as chlorophyll content and water status of crop leaves, and are an important basis for crop health diagnosis. The core of feature extraction is to realize the joint capture of spectral and spatial information [18]. Traditional methods use two-dimensional convolutional networks to process multispectral images, which cannot effectively mine the correlation between adjacent bands, resulting in serious spectral information loss and insufficient feature representation capability [19, 20]. Therefore, this paper designs a customized three-dimensional convolutional neural network (3D-CNN). According to the continuity characteristics of multispectral bands, through the optimization of convolution kernel design, stride, and padding strategy, the collaborative extraction of spectral and spatial features is realized, improving the discriminability and integrity of multispectral features.

The core innovation of the customized 3D-CNN lies in the optimization of convolution kernel design and feature enhancement mechanism. A 3 × 3 × 3 three-dimensional convolution kernel is adopted, where the first two dimensions correspond to the image spatial size and the third dimension corresponds to the multispectral band dimension, ensuring that spatial neighborhood information and inter-band correlation features can be captured simultaneously. The core calculation formula of the convolution process is:

${{\text{F}}_{\text{x,y,b}}}\text{= }\!\!\sigma\!\!\text{ }\left( \underset{\text{i=0}}{\overset{\text{2}}{\mathop \sum }}\,\underset{\text{j=0}}{\overset{\text{2}}{\mathop \sum }}\,\underset{\text{k=0}}{\overset{\text{2}}{\mathop \sum }}\,{{\text{W}}_{\text{i,j,k}}}\cdot {{\text{I}}_{\text{x+i,y+j,b+k}}}\text{+b} \right)$        (2)

where, F_x,y,b represents the feature value of the output feature map at position (x,y) and band b, W_i,j,k is the weight of the 3 × 3 × 3 convolution kernel, I_x+i,y+j,b+k is the pixel value of the input multispectral image, b is the bias term, and σ is the ReLU activation function. To reduce spectral information loss, the convolution stride strategy is optimized, and the strides of both spatial and spectral dimensions are set to 1. The “Same” padding method is adopted to ensure that the spatial size and band number of the output feature map are consistent with those of the input image. At the same time, batch normalization layers are embedded to normalize the output features of each layer, reducing feature distribution shift and enhancing feature stability. This effectively solves the problem that traditional two-dimensional convolutional networks cannot capture inter-band correlations and suffer from spectral information loss, and finally outputs multi-scale multispectral feature maps integrating spectral and spatial features.

Thermal infrared images can reflect the temperature distribution of crop canopy and are a key basis for identifying crop stress status, but they generally have problems such as low resolution and blurred temperature anomaly regions. Traditional feature extraction networks are difficult to balance computational efficiency and global temperature capture capability [21, 22]. This paper adopts a lightweight Mobile Vision Transformer (MobileViT) network as the feature extraction backbone for thermal infrared images. The core innovation lies in the simplification of the network structure and the optimization of the attention mechanism. By adjusting network parameters and attention head configuration, the collaborative improvement of global temperature distribution capture and computational efficiency is achieved.

Figure 1. Schematic diagram of modality-specific feature extraction network structure

According to the characteristics of thermal infrared images, two aspects of customized optimization are carried out on the MobileViT network: first, simplifying network parameters by replacing traditional convolution layers with depth-wise separable convolutions to reduce parameter redundancy and computational complexity; second, adjusting the number of attention heads in the Transformer encoder, abandoning the traditional fixed attention head configuration, reducing the number of attention heads from 6 to 4, and adjusting the dimension of attention weight calculation to adapt to the low-resolution characteristics of thermal infrared images. Under the premise of ensuring the ability to capture global temperature distribution, computational efficiency is further improved. The attention weight calculation adopts a simplified scaled dot-product attention mechanism, and the core formula is:

$\text{Attention(Q,K,V)=softmax}\left( \frac{\text{Q}{{\text{K}}^{\text{T}}}}{\sqrt{{{\text{d}}_{\text{k}}}}} \right)\text{V}$          (3)

where, Q, K, and V represent the query, key, and value matrices respectively, and dₖ is the dimension of the query matrix. This optimization enables the network to quickly capture abnormal temperature regions of crop canopy while effectively suppressing background noise interference, solving the problems of low resolution and difficulty in locating temperature anomaly regions in thermal infrared images, and outputting multi-scale thermal infrared feature maps sensitive to temperature changes. Figure 1 shows the schematic diagram of the modality-specific feature extraction network structure.

The RGB, multispectral, and thermal infrared feature extraction branches respectively output multi-scale feature maps covering shallow texture, middle-level semantics, and deep abstraction. The features of each branch complement each other and have different emphases. They not only retain the core information of crop phenotype, spectrum, and temperature, but also have good hierarchical characteristics, providing high-quality basic feature support for subsequent multi-scale cross-modal adaptive fusion, and ensuring that the fusion process can fully mine the complementary value of different modalities.

2.3 Cross-modal adaptive fusion and health status recognition

2.3.1 Cross-modal adaptive fusion module

The core objective of the cross-modal adaptive fusion module is to solve the problems of nonlinear alignment, dynamic weighting, and noise suppression of heterogeneous modal features, to realize deep interaction of multi-scale semantic features, and to provide discriminative and semantically consistent fused features for subsequent health status recognition. The core innovation of this module lies in realizing intra-modal feature enhancement and inter-modal feature alignment in stages. Through customized attention mechanism design, it breaks through the limitation that traditional fusion methods are difficult to balance semantic consistency and heterogeneous information complementarity, ensuring that the core information of different modalities can be fully mined and efficiently fused. Figure 2 shows the internal structure of the multi-scale cross-modal adaptive fusion and aggregation module.

Figure 2. Internal structure of the multi-scale cross-modal adaptive fusion and aggregation module

Intra-modal self-attention enhancement is the basis for achieving effective cross-modal fusion. Its core innovation lies in designing a customized spatial self-attention mechanism for the hierarchical characteristics of multimodal features, strengthening the semantic consistency within a single modality, capturing long-range dependencies within the feature map, and improving the discriminative ability of features. Different from the traditional self-attention mechanism that treats all features equally, the intra-modal self-attention in this paper calculates attention weights on the spatial dimension for multi-scale features output by each branch, highlighting crop health-related feature regions and suppressing background noise interference. The core calculation process is as follows:

$\text{A=softmax}\left( \frac{\text{F}\text{.}{{\text{F}}^{\text{T}}}}{\sqrt{\text{d}}} \right)$        (4)

${{\text{F}}_{\text{enhanced}}}\text{=A}\cdot \text{F}$        (5)

where, F represents the feature map of a certain scale of a single modality, d is the channel dimension of the feature map, A is the generated spatial attention map, and F_enhanced is the feature map after self-attention enhancement. This design generates an attention weight map by calculating the correlation between each pixel and global pixels, so that regions with significant crop health features in the feature map obtain higher weights, effectively solving the problems of isolated local information and insufficient semantic coherence in single-modal features, and providing semantically consistent basic features for subsequent inter-modal alignment.

Inter-modal cross-attention alignment is the core for achieving efficient interaction of heterogeneous modal information. Its innovation lies in reasonably assigning the Query, Key, and Value matrices, and optimizing the dimensional configuration of multi-head attention to achieve accurate alignment and information complementarity of heterogeneous features. In this paper, the RGB branch features are used as Query, and the multispectral and thermal infrared branch features are used as Key and Value respectively to construct a multi-head cross-attention mechanism. With the visual semantic guidance of RGB features, the physiological information of multispectral and the temperature information of thermal infrared are accurately retrieved, realizing the crossing of the semantic gap of heterogeneous modalities. To adapt to the dimensional differences of multimodal features, the dimensional allocation of attention heads is optimized, and the dimension of each attention head is set to match the channel dimension of each modal feature, ensuring the effectiveness of attention calculation.

The core calculation of cross-attention adopts the optimized scaled dot-product attention mechanism, and the specific formulas are as follows:

$\begin{gathered}\text { MultiHead }(Q, K, V) =\text { Concat }\left(\text { head }_1, \text { head }_2, \ldots, \text { head }_{\mathrm{h}}\right) \cdot W_o\end{gathered}$   (6)

$\text{hea}{{\text{d}}_{\text{i}}}\text{=Attention}\left( \text{Q}{{\text{W}}_{{{\text{Q}}_{\text{i}}}}}\text{,K}{{\text{W}}_{{{\text{K}}_{\text{i}}}}}\text{,V}{{\text{W}}_{{{\text{V}}_{\text{i}}}}} \right)$        (7)

where, Q, K, and V represent the feature matrices of RGB, multispectral, and thermal infrared branches respectively, h is the number of attention heads, ${{\text{W}}_{{{\text{Q}}_{\text{i}}}}}$, ${{\text{W}}_{{{\text{K}}_{\text{i}}}}}$, and ${{\text{W}}_{{{\text{V}}_{\text{i}}}}}$ are the learnable projection matrices of the i-th attention head, and Wₒ is the output projection matrix of multi-head attention. This design enables RGB features to accurately retrieve spectral response information related to crop health in multispectral, and temperature signals related to stress status in thermal infrared, realizing accurate alignment and deep interaction of heterogeneous modal features, and providing high-quality aligned features for subsequent adaptive gated fusion.

The adaptive gated fusion unit is the key to achieving efficient fusion of heterogeneous modal features and suppressing invalid noise. Its core innovation lies in designing a scene-adaptive weight adjustment mechanism, which can dynamically allocate the contribution of each modality according to the scene characteristics of input multimodal images, breaking through the limitation that traditional fixed-weight fusion cannot adapt to complex agricultural scenes, and at the same time achieving effective suppression of invalid modal noise and improving the discriminative ability of fused features. This unit takes the features after cross-attention alignment as input, and realizes accurate fusion of heterogeneous features through three steps: feature description, weight mapping, and dynamic fusion.

The specific implementation process of the adaptive gated fusion unit is as follows: first, global average pooling is performed on the features of each modality after cross-attention enhancement, compressing the two-dimensional feature maps into one-dimensional feature description vectors to capture the global feature information of each modality; then the feature description vectors are input into two fully connected layers for dimensional mapping and feature transformation, and the weight coefficients of each modality are obtained through Softmax normalization, ensuring that the sum of weights is 1 and realizing reasonable allocation of modal contributions. The core formulas of weight coefficients and fused features are as follows:

${{\text{v}}_{\text{i}}}\text{=GAP(}{{\text{F}}_{\text{cross,i}}}\text{)}$        (8)

${{\text{w}}_{\text{i}}}\text{=Softmax}\left( {{\text{W}}_{\text{2}}}\cdot \text{ }\!\!\sigma\!\!\text{ }\left( {{\text{W}}_{\text{1}}}\cdot {{\text{v}}_{\text{i}}}\text{+}{{\text{b}}_{\text{1}}} \right)\text{+}{{\text{b}}_{\text{2}}} \right)$          (9)

${{\text{F}}_{\text{fusion,single}}}\text{=}\underset{\text{i=1}}{\overset{\text{3}}{\mathop \sum }}\,{{\text{w}}_{\text{i}}}\cdot {{\text{F}}_{\text{cross,i}}}$         (10)

where, v_i is the feature description vector of the i-th modality, GAP( ) is the global average pooling operation, F_cross,i is the feature map of the i-th modality after cross-attention enhancement, W₁ and W₂ are the learnable weight matrices of the fully connected layers, b₁ and b₂ are bias terms, σ is the ReLU activation function, w_i is the weight coefficient of the i-th modality, and F_{fusion,single} is the fused feature at a single scale. This paper innovatively designs an adaptive weight adjustment mechanism. Through the scene information contained in the feature description vector, the weights of each modality are dynamically updated. For example, when illumination is insufficient, the quality of RGB modal features decreases, the weight is automatically reduced, while the weight of the thermal infrared modality is increased, ensuring that the fused features can always retain core effective information and suppress invalid noise interference.

Multi-scale fusion aggregation aims to solve the problem that single-scale fusion cannot take into account both detail information and semantic information. Its core innovation lies in repeatedly executing the complete process of intra-modal self-attention enhancement, inter-modal cross-attention alignment, and adaptive gated fusion at different feature scales, and realizing efficient aggregation of multi-scale fused features through skip connections, forming a multi-scale enhanced fused feature representation, and providing sufficient feature support for subsequent high-resolution health status recognition. Multimodal features have obvious hierarchical characteristics: low-level features focus on details such as texture, middle-level features focus on semantic information, and high-level features tend to abstract features. Single-scale fusion is difficult to fully exploit the value of features at each level.

Specifically, the multi-scale fusion aggregation process performs the above fusion process for three feature scales: low-level texture, middle-level semantics, and high-level abstraction, respectively obtaining fused features at three scales. Then, a skip connection mechanism is introduced to upsample the low-level fused features and concatenate them with middle-level fused features, and concatenate the middle-level fused features with high-level fused features, realizing complementarity and aggregation of features at different scales, and finally outputting multi-scale enhanced fused feature maps. The innovation of this design lies in that it does not simply superimpose features at different scales, but retains the core information of features at each scale through skip connections, so that the fused features not only contain low-level details such as crop leaf texture, but also cover middle-level semantics such as disease regions, while taking into account the discriminative ability of high-level abstract features, effectively solving the problems of detail loss or semantic ambiguity in traditional fusion methods, and providing a hierarchical and discriminative feature basis for subsequent pixel-level health status recognition.

2.3.2 Health status recognition module

The core objective of the health status recognition module is to achieve high-resolution pixel-level recognition of crop health status based on the multi-scale features after cross-modal adaptive fusion, and to output a continuous health index at the same time, providing accurate spatiotemporal distribution information for subsequent yield prediction. Figure 3 shows the structure diagram of the lightweight health status recognition decoder with dual-output collaboration. The core innovation of this module lies in the customized improvement of the lightweight decoder and the dual-output collaborative design, which not only solves the problems of low segmentation accuracy, blurred boundaries, and computational redundancy of traditional decoders, but also realizes the integration of discrete health 1 × 1 Convolution status classification and continuous health index regression, improving the comprehensiveness and practicality of recognition results, and fully adapting to the actual needs of agricultural monitoring.

Figure 3. Structure diagram of the lightweight health status recognition decoder with dual-output collaboration

The design of the lightweight decoder is based on an improved Feature Pyramid Network (FPN). The core innovation lies in embedding a channel attention module and optimizing the upsampling and skip connection strategies, significantly improving segmentation accuracy and boundary clarity while ensuring computational efficiency. Different from the traditional FPN decoder that only performs simple feature concatenation and upsampling, this paper adds a customized channel attention module at each feature fusion node of the decoder. By dynamically adjusting feature channel weights, the response of key features such as disease regions and nutrient deficiency regions is enhanced, and the interference of background and invalid features is suppressed. The core calculation formulas of the channel attention module are as follows:

${{\text{M}}_{\text{c}}}\text{= }\!\!\sigma\!\!\text{ }\left( {{\text{W}}_{\text{2}}}\cdot \text{GAP(}{{\text{F}}_{\text{fusion}}}\text{)+}{{\text{b}}_{\text{2}}} \right)$         (11)

${{\text{F}}_{\text{att}}}\text{=}{{\text{M}}_{\text{c}}}\cdot {{\text{F}}_{\text{fusion}}}$           (12)

where, F_fusion is the multi-scale fused feature input to the decoder, GAP( ) is the global average pooling operation, W₂ is the learnable weight matrix, b₂ is the bias term, σ is the Sigmoid activation function, M_c is the channel attention weight map, and F_att is the feature after attention enhancement. At the same time, the upsampling strategy is optimized by combining transposed convolution and bilinear interpolation to gradually restore the spatial resolution of the feature map, avoiding blurred boundaries caused by a single upsampling method. The skip connection mechanism is also improved, where the attention-enhanced features of each layer of the decoder are accurately aligned and concatenated with the features of the corresponding scale in the encoder, retaining low-level texture details and high-level semantic information. Finally, a pixel-level classification map with the same size as the input image is generated, effectively solving the problems of low segmentation accuracy and blurred boundaries in traditional decoders.

The dual-output design is another core innovation of this module, aiming to simultaneously realize discrete classification of crop health status and continuous health index regression. Through multi-task collaborative optimization, the comprehensiveness and reliability of recognition results are improved, providing richer spatiotemporal feature support for yield prediction. The discrete classification task targets four typical states: healthy, mild disease, severe disease, and nutrient deficiency, and performs pixel-level category division. The continuous health index regression task outputs continuous values based on Normalized Difference Vegetation Index (NDVI) transformation to quantify the crop health degree. The two tasks complement each other and are jointly optimized, avoiding the limitations of a single output mode.

To realize the collaborative optimization of dual-output tasks, a multi-task joint loss function is designed, which combines classification loss and regression loss with weights, dynamically balancing the training priorities of the two tasks. The core formula is as follows:

${{\text{L}}_{\text{total}}}\text{= }\!\!\alpha\!\!\text{ }\cdot {{\text{L}}_{\text{seg}}}\text{+ }\!\!\beta\!\!\text{ }\cdot {{\text{L}}_{\text{reg}}}$          (13)

where, L_total is the total loss function, α and β are the weight coefficients of classification loss and regression loss respectively, satisfying α + β = 1, and can be adaptively adjusted during the training process; L_seg is the classification loss, which adopts the weighted sum of cross-entropy loss and Dice loss to solve the class imbalance problem and improve segmentation accuracy; L_reg is the regression loss, which adopts mean squared error loss to ensure the accuracy of health index regression. This dual-output design enables classification results and regression results to constrain and promote each other through multi-task collaborative training, ensuring accurate classification of health status categories and quantitative evaluation of health degree. Finally, it outputs pixel-level health status segmentation maps and continuous health index maps, providing basic features containing spatiotemporal distribution and quantitative degree of crop health for subsequent spatiotemporal yield prediction.

2.4 Spatiotemporal yield prediction

The construction of spatiotemporal graph structure is the basis for modeling the relationship between crop health status spatiotemporal dynamics and yield. Its core innovation lies in transforming the pixel-level health status features obtained by image processing into graph-structured data suitable for ST-GCN, realizing the deep integration of image processing results and spatiotemporal modeling, breaking through the limitation that traditional spatiotemporal modeling is disconnected from crop health features, and accurately capturing the spatial correlation and temporal evolution law of crop health status, providing interpretable spatiotemporal feature support for subsequent yield prediction. This construction process closely relies on the output of the health status recognition module, transforming discrete pixel-level information into structured graph nodes and edges, ensuring the pertinence and accuracy of spatiotemporal association modeling. Figure 4 shows the schematic diagram of multi-feature fusion and yield prediction.

Figure 4. Schematic diagram of multi-feature fusion and yield prediction

The construction process of the spatiotemporal graph structure is carried out around three core steps: node division, node feature extraction, and edge construction. Each step is customized according to agricultural scenarios and image processing characteristics. Node division adopts a method consistent with actual agricultural production, dividing multi-temporal health status segmentation maps into several nodes according to plots or fixed sampling areas, ensuring that each node corresponds to an independent agricultural monitoring unit and realizing accurate quantification of the spatial distribution of health status. Node feature extraction focuses on the core information of health status. For each node, two types of key statistics are calculated: the area proportion of each health status category and the average health index. The calculation formula of the area proportion of the c-th health status of the k-th node is ${{\text{r}}_{\text{k,c}}}\text{=}{{\text{S}}_{\text{k,c}}}\text{/}\underset{\text{c=1}}{\overset{\text{C}}{\mathop \sum }}\,{{\text{S}}_{\text{k,c}}}$, and the calculation formula of the average health index is ${{\text{h}}_{\text{k}}}\text{=}\underset{\text{i=1}}{\overset{{{\text{N}}_{\text{k}}}}{\mathop \sum }}\,{{\text{h}}_{\text{k,i}}}\text{/}{{\text{N}}_{\text{k}}}$, where S_k,c is the pixel area of the c-th health status of the k-th node, C is the total number of health status categories, h_k,i is the health index of the i-th pixel in the k-th node, and N_k is the total number of pixels of the k-th node. Edge construction combines spatial correlation characteristics, considering both Euclidean distance and spatial adjacency relationship between nodes. The Euclidean distance between two nodes u and v is calculated as ${{\text{d}}_{\text{u,v}}}\text{=}\sqrt{{{\text{(}{{\text{x}}_{\text{u}}}\text{-}{{\text{x}}_{\text{v}}}\text{)}}^{\text{2}}}\text{+(}{{\text{y}}_{\text{u}}}\text{-}{{\text{y}}_{\text{v}}}{{\text{)}}^{\text{2}}}}$, where (x_u,y_u) and (x_v,y_v) are the spatial coordinates of the two nodes respectively. When the distance is less than a set threshold or there is a spatial adjacency relationship, an edge is constructed between the nodes and assigned a corresponding weight, realizing accurate modeling of spatial correlation of crop health status, and finally forming spatiotemporal graph structure data containing node features, spatial correlation, and time series.

ST-GCN modeling is the core for realizing the mining of the relationship between crop health status spatiotemporal dynamics and yield. Its core innovation lies in customizing and optimizing the traditional ST-GCN. By embedding an attention mechanism in GCN and adding a dropout layer in Gated Recurrent Unit (GRU), it solves the problems of uneven spatial information propagation and easy overfitting in traditional spatiotemporal modeling, realizes accurate capture of crop health evolution laws, and constructs a spatiotemporal feature representation with both accuracy and interpretability. This modeling process takes the constructed spatiotemporal graph structure as input, and is divided into two stages: spatial propagation and temporal capture, realizing integrated modeling of spatiotemporal dynamics of health status, and providing core spatiotemporal feature support for subsequent yield prediction.

The core innovation of the spatial information propagation stage is to embed a spatial attention mechanism in GCN, dynamically adjusting the correlation weights of neighboring nodes, solving the problem that traditional GCN treats all neighboring nodes equally and spatial information propagation is uneven, and strengthening the spatial correlation modeling of crop health status. The optimized node feature update formula of GCN is as follows:

${{\text{F}}_{\text{k,t+1}}}\text{= }\!\!\sigma\!\!\text{ }\left( \underset{\text{v}\in \text{N(k)}}{\mathop \sum }\,{{\text{ }\!\!\alpha\!\!\text{ }}_{\text{k,v}}}\cdot \text{W}\cdot {{\text{F}}_{\text{v,t}}}\text{+b} \right)$           (14)

where, F_k,t+1 is the feature vector of the k-th node after update at step t+1, N(k) is the set of neighboring nodes of the k-th node, α_k,v is the attention weight between node k and neighboring node v, calculated from node feature correlation, W is the learnable weight matrix, b is the bias term, and σ is the ReLU activation function. The attention weight α_k,v is obtained through Softmax normalization to ensure reasonable weight allocation, so that neighboring nodes with significant health status changes obtain higher weights, accurately capturing spatial correlation patterns such as disease spread. In the temporal dimension capture stage, the updated node feature sequence at each time step is input into the optimized GRU, and a dropout layer is added to the GRU hidden layer with a dropout probability of 0.3 to suppress model overfitting. The core update formulas are:

${{\text{r}}_{\text{t}}}\text{= }\!\!\sigma\!\!\text{ }\left( {{\text{W}}_{\text{r}}}\cdot \text{ }\!\![\!\!\text{ }{{\text{h}}_{\text{t-1}}}\text{,}{{\text{F}}_{\text{t}}}\text{ }\!\!]\!\!\text{ +}{{\text{b}}_{\text{r}}} \right)$       (15)

${{\text{z}}_{\text{t}}}\text{= }\!\!\sigma\!\!\text{ }\left( {{\text{W}}_{\text{z}}}\cdot \text{ }\!\![\!\!\text{ }{{\text{h}}_{\text{t-1}}}\text{,}{{\text{F}}_{\text{t}}}\text{ }\!\!]\!\!\text{ +}{{\text{b}}_{\text{z}}} \right)$     (16)

$\tilde{\mathrm{h}}_t=\tanh \left(W_{\mathrm{h}} \cdot\left[r_t \odot \mathrm{~h}_{t-1}, F_t\right]+b_{\mathrm{h}}\right)$       (17)

$\mathrm{h}_t=\left(1-z_t\right) \odot \mathrm{h}_{t-1}+z_t \odot \tilde{\mathrm{~h}}_t$         (18)

where, r_t and z_t are the reset gate and update gate respectively, h_t−1 and h_t are the hidden states at the previous time and current time respectively, $\tilde{\mathrm{h}}_t$ is the candidate hidden state, ⊙ denotes element-wise multiplication, W_r, W_z, and W_h are learnable weight matrices, and b_r, b_z, and b_h are bias terms. This optimization enables GRU to accurately capture the temporal evolution law of crop health status while avoiding overfitting, and outputs feature vectors containing spatiotemporal correlation information.

The core innovation of multi-feature fusion prediction lies in designing a feature normalization and attention-weighted fusion mechanism to achieve efficient fusion of spatiotemporal features output by ST-GCN and environmental factors, highlighting the dominant role of spatiotemporal features while taking into account the influence of environmental factors, improving the accuracy and interpretability of yield prediction, and solving the problems of feature weight imbalance and poor interpretability in traditional fusion methods. In this process, the two types of features are first standardized to eliminate dimensional differences and ensure the effectiveness of fusion. The feature normalization formula is:

${{\text{F}}_{\text{norm}}}\text{=}\frac{\text{F- }\!\!\mu\!\!\text{ }}{\text{ }\!\!\sigma\!\!\text{ }}$        (19)

where, F_norm is the normalized feature, F is the original feature, μ is the feature mean, and σ is the feature standard deviation. Then, a feature attention-weighted fusion module is constructed to dynamically allocate the weights of spatiotemporal features and environmental factors, highlighting the dominant role of spatiotemporal features. The attention weight calculation formulas are:

$\text{ }\!\!\omega\!\!\text{ =Softmax}\left( {{\text{W}}_{\text{a}}}\cdot \text{ }\!\![\!\!\text{ }{{\text{F}}_{\text{st}}}\text{,}{{\text{F}}_{\text{env}}}\text{ }\!\!]\!\!\text{ +}{{\text{b}}_{\text{a}}} \right)$      (20)

${{\text{F}}_{\text{final}}}\text{=}{{\text{ }\!\!\omega\!\!\text{ }}_{\text{1}}}\cdot {{\text{F}}_{\text{st}}}\text{+}{{\text{ }\!\!\omega\!\!\text{ }}_{\text{2}}}\cdot {{\text{F}}_{\text{env}}}$      (21)

where, F_st is the spatiotemporal feature vector output by ST-GCN, F_env is the global feature vector of environmental factors, W_a is the learnable weight matrix, and b_a is the bias term, and ω₁ and ω₂ are the attention weights of spatiotemporal features and environmental factors respectively.

The fused feature vector is input into a multi-layer fully connected network for yield regression. The network uses ReLU as the activation function, and the output layer uses a linear activation function to ensure the continuity and rationality of the yield prediction value. The output formula of the fully connected network is:

$\hat{y}=W_o \cdot \sigma\left(W_3 \cdot F_{\text {final }}+b_3\right)+b_o$           (22)

where, $\hat{y}$ is the final yield prediction value, W₃ and W_o are the learnable weight matrices of the fully connected layers, and b₃ and b_o are bias terms. This multi-feature fusion design realizes reasonable allocation of features through attention weighting, enabling the model to comprehensively consider the spatiotemporal dynamics of crop health status and the collaborative influence of external environmental factors, significantly improving the accuracy and interpretability of yield prediction, and finally outputting accurate crop yield prediction values.

2.5 Training strategy

The core innovation of the training strategy lies in deeply combining multi-task joint optimization with self-supervised contrastive pretraining. Aiming at the key problems of imbalance in multi-task collaborative training and insufficient feature learning in small-sample scenarios, a dedicated training process and loss functions are designed to achieve the coordinated improvement of model training efficiency, recognition accuracy, and generalization ability, providing guarantees for the stable performance of the end-to-end framework. This strategy closely matches the characteristics of multimodal image processing and agricultural scenarios. Through customized pretraining and fine-tuning mechanisms, it addresses the limitations of traditional training methods such as weak generalization ability and imbalance in multi-task training, ensuring that the model can still maintain excellent performance in complex agricultural scenarios and small-sample conditions.

The core innovation of multi-task joint optimization lies in designing a hybrid loss function with adaptive weight adjustment to balance the training priorities of health state recognition and yield prediction tasks, while addressing the class imbalance problem in health state segmentation. The loss function is composed of a weighted combination of pixel-level segmentation loss and yield regression loss. The segmentation loss adopts a weighted sum of cross-entropy loss and Dice loss, effectively alleviating the training bias caused by class imbalance, while the regression loss adopts mean squared error loss to ensure the accuracy of yield prediction. The core formulas are as follows:

${{\text{L}}_{\text{total}}}\text{= }\!\!\alpha\!\!\text{ (t)}\cdot {{\text{L}}_{\text{seg}}}\text{+ }\!\!\beta\!\!\text{ (t)}\cdot {{\text{L}}_{\text{reg}}}$        (23)

${{\text{L}}_{\text{seg}}}\text{= }\!\!\gamma\!\!\text{ }\cdot {{\text{L}}_{\text{ce}}}\text{+(1- }\!\!\gamma\!\!\text{ )}\cdot {{\text{L}}_{\text{dice}}}$          (24)

${{\text{L}}_{\text{dice}}}\text{=1-}\frac{\text{2}\mathop{\sum }_{\text{i=1}}^{\text{N}}{{\text{p}}_{\text{i}}}{{\text{g}}_{\text{i}}}\text{+}\epsilon }{\mathop{\sum }_{\text{i=1}}^{\text{N}}\text{p}_{\text{i}}^{\text{2}}\text{+}\mathop{\sum }_{\text{i=1}}^{\text{N}}\text{g}_{\text{i}}^{\text{2}}\text{+}\epsilon }$       (25)

$L_{\text {reg }}=\frac{1}{M} \sum_{j=1}^M\left(y_j-\bar{y}_j\right)^2$          (26)

where, L_total is the total loss function, α(t) and β(t) are adaptive weight coefficients satisfying α(t)+β(t)=1, and their values are dynamically updated according to the convergence speed of the two tasks during training. When a task converges more slowly, its corresponding weight is automatically increased, achieving dynamic balancing of task priorities. L_seg is the segmentation loss, γ is the weight coefficient between cross-entropy loss L_ce and Dice loss L_dice, and ϵ is a smoothing term to avoid division by zero. p_i and g_i are the predicted value and ground truth label, respectively. L_reg is the regression loss, M is the number of samples, and y_j and ${{\overset{}{\mathop{\text{y}}}\,}_{\text{j}}}$ are the ground truth yield and predicted yield, respectively. This design solves the imbalance problem in multi-task training through adaptive weight adjustment, and alleviates class imbalance through hybrid segmentation loss, ensuring that the two tasks are jointly optimized and improved simultaneously.

The core innovation of self-supervised contrastive pretraining lies in designing customized data augmentation strategies and contrastive learning schemes for multimodal image characteristics. Robust modality-invariant features are learned on unlabeled data, addressing the problem of insufficient multimodal feature learning in small-sample scenarios, and providing high-quality initialization parameters for the feature extraction network. The pretraining process is based on the Simple Framework for Contrastive Learning of Visual Representations (SimCLR) framework. The core idea is to construct positive and negative sample pairs through data augmentation, pulling closer the feature distance of different augmented views of the same image, and pushing away the feature distance of different images, thereby achieving effective learning of modality-invariant features.

Specifically, for RGB, multispectral, and thermal infrared modalities, differentiated data augmentation strategies are designed: RGB images adopt random cropping, horizontal flipping, and brightness adjustment; multispectral images adopt band perturbation and random cropping; thermal infrared images adopt brightness adjustment and random cropping, ensuring that the core features of each modality are retained after augmentation. The core formula of the contrastive loss is as follows:

${{\text{L}}_{\text{contrast}}}\text{=-}\frac{\text{1}}{\text{2N}}\underset{\text{k=1}}{\overset{\text{N}}{\mathop \sum }}\,\text{log}\frac{\text{exp(sim(}{{\text{z}}_{\text{k}}}\text{,z}_{\text{k}}^{\text{+}}\text{)/ }\!\!\tau\!\!\text{ )}}{\mathop{\sum }_{\text{l=1}}^{\text{2N}}{{\text{I}}_{\text{l}\ne \text{k}}}\text{exp(sim(}{{\text{z}}_{\text{k}}}\text{,z}_{\text{l}}^{\text{+}}\text{)/ }\!\!\tau\!\!\text{ )}}$          (27)

where, N is the number of samples in a training batch, z_k is the feature embedding of the k-th sample, $\text{z}_{\text{k}}^{\text{+}}$ is the feature embedding of the positive sample of the k-th sample, τ is the temperature parameter, sim( ) is the cosine similarity function, and I is the indicator function. The pretraining process is only applied to the feature extraction networks of the three modalities. By minimizing the contrastive loss, robust features that are not affected by modality differences and augmentation perturbations are learned, providing good initialization for the subsequent fine-tuning of the end-to-end framework, and effectively improving the generalization ability of the model in small-sample scenarios.

The training process adopts a two-stage mode of “pretraining-fine-tuning” to further improve training efficiency and model performance. First, self-supervised contrastive pretraining is performed on unlabeled multimodal images for the feature extraction networks to obtain initialization parameters. Then, the pretrained parameters are transferred to the entire end-to-end framework, and fine-tuning is performed using the multi-task joint optimization loss function to simultaneously optimize the two tasks of health state recognition and yield prediction. This process not only solves the problem of insufficient feature learning in small-sample scenarios through pretraining, but also achieves collaborative adaptation of each module through fine-tuning, ensuring that the model accurately learns features while achieving efficient collaboration between the two tasks, ultimately improving recognition accuracy, prediction reliability, and generalization ability of the model.

This paper addressed key problems in multimodal remote sensing images for crop health recognition and yield prediction, including difficulty in nonlinear alignment of heterogeneous features, insufficient dynamic adaptation of modality contribution, inadequate spatiotemporal association modeling, and limited generalization ability in small-sample scenarios. Focusing on image processing technology innovation, an end-to-end algorithm framework based on multimodal adaptive fusion and ST-GCN was proposed, systematically completing the integrated modeling of multimodal feature extraction, cross-modal fusion, health state recognition, and yield prediction. The core innovations lie in designing modality-specific feature extraction branches to achieve targeted capture of different modality features; constructing a multi-scale cross-modal adaptive fusion mechanism, which effectively solved the semantic gap and noise suppression problems of heterogeneous modalities through the combination of intra-modal self-attention, inter-modal cross-attention, and adaptive gating units; proposing an ST-GCN-driven spatiotemporal modeling method to achieve accurate association between the spatiotemporal dynamics of crop health states and yield; and designing a multi-task joint optimization and self-supervised contrastive pretraining strategy to balance task priorities and improve generalization ability in small-sample scenarios. Experimental results fully verify that the proposed method significantly outperformed existing advanced methods in health state recognition accuracy, yield prediction reliability, and computational efficiency. Each innovative module effectively improved model performance, demonstrating good stability and generalization ability.

The proposed method forms an innovative paradigm in the deep integration of image processing technology and agricultural applications. It not only enriches the research ideas of multimodal feature fusion and spatiotemporal modeling, providing a new technical pathway for heterogeneous multimodal image processing with important academic value, but also achieves high-resolution recognition of crop health states and reliable yield prediction, providing efficient technical support for precision agricultural monitoring and grain yield estimation, with strong practical application prospects. In future work, focusing on the limitations of the method and combining frontier technologies in the field of image processing, further optimization will be conducted on multimodal data alignment accuracy, lightweight spatiotemporal modeling architectures will be designed to improve computational efficiency, contrastive learning and few-shot learning will be integrated to enhance the recognition ability of extreme diseases in small-sample scenarios, and multimodal data types such as hyperspectral data will be expanded, continuously improving algorithm performance and promoting the in-depth application and development of image processing technology in the field of precision agriculture.