Multimodal Deep Learning for Tapioca Yield Prediction Using Field-Collected Sensor and Imagery Data

Accurate crop production forecasts are crucial to precision agriculture because they enable effective decision-making in resource allocation, supply chain planning, and farm management [1, 2]. Tapioca yield prediction is particularly challenging due to the complex interactions between environmental, soil, and climatic characteristics, as well as variations in management strategies and data accessibility [3-5]. These challenges are made worse by the heterogeneous nature of agricultural data, which includes temporal sensor readings and geographical imaging data [6, 7].

Most conventional yield estimating methods rely on human observations and statistical models, which are frequently labor-intensive, time-consuming, and not real-time adaptive [8-10]. While deep learning (DL) and machine learning (ML) techniques have greatly increased prediction accuracy, they are still unable to capture complicated spatial-temporal correlations and efficiently integrate multimodal data sources [11-13]. While DL models like Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks require large datasets and involve high computational complexity [14, 15], ML models like Random Forest and Support Vector Machines (SVMs) have shown consistent performance but are limited in handling high-dimensional heterogeneous data [16, 17].

A significant technical challenge in tapioca production prediction is efficiently merging multimodal data that captures nonlinear environmental interactions and ensuring real-time prediction with good accuracy and generalization [18]. Because redundant and superfluous features in high-dimensional datasets can impair model performance, effective feature selection is essential [19]. To solve these problems, this study proposes a multimodal architecture that integrates Internet of Things (IoT) sensor data, aerial photography, and sophisticated feature selection and learning techniques. The suggested method combines an enhanced Support Vector Regression (SVR) model for prediction, attention-enabled DenseNet-201 for feature extraction, and Hybrid Frilled Lizard Osprey (HFLO)-based feature selection. While preserving computing efficiency, this methodology is intended to increase feature significance, capture spatial-temporal dependencies, and boost prediction accuracy [20]. To address these issues, the combination of IoT sensor networks and Unmanned Aerial Vehicle (UAV)/satellite imagery, with DL models, gives a promising solution for accurate real-time yield prediction.

1.1 Motivation

Farming is most significant for enhancing food production and growth of a country’s income world-wide. Several studies performed crop, rice, wheat, and paddy yield prediction, but were limited to tapioca yield prediction, which is much needed for agricultural growth. The existing studies obtained better results in agricultural systems; however, they faced some challenges like a lack of real-time accuracy, time-consuming, labour-intensive, and generalization. This motivated to develop DL model for real-time yield prediction through the collection of sensor data and satellite imagery. Thus, the proposed research developed novel fully connected layers and improved support vector regression (ISVR) model for tapioca yield prediction, thereby reducing the time- and labour-intensive challenges.

1.2 Contributions

By combining multimodal data with cutting-edge learning methods, this study seeks to provide a reliable and accurate framework for predicting tapioca yield. The following are this work's main contributions,

• To make use of multimodal data integration, which combines satellite imaging and IoT sensor data to provide more accurate and timely yield prediction than single-source methods.
• To increase data quality and model performance by designing an efficient data pre-processing pipeline that includes image enhancement techniques and sensor data normalization.
• To suggest a novel HFLO optimization technique for feature selection. In contrast to traditional techniques, HFLO combines the exploitation power of Osprey optimization with the exploration capacity of Frilled Lizard optimization (FLO), leading to improved feature subset selection and decreased redundancy.
• To identify the most interesting areas in imaging data and capture long-range spatial correlations using a Position Attention-based DenseNet-201 model for feature extraction improves the network's capacity.
• To create a yield prediction model using ISVR. The HFLO algorithm's refined hyper parameter tuning, improved kernel adaptation for managing nonlinear agricultural data, and better feature selection and parameter optimization all contribute to the improvement.

In this research organization section, as follows: 2 denotes the recent studies based on related work, 3 describes the proposed framework details, 4 illustrates the results & discussion, and 5 represents the conclusion & future scope.

Tapioca is a vital root crop whose yield is influenced by multiple environmental, soil, and physiological factors. Traditional yield estimation methods depend upon statistical models and manual sampling, which often lack real-time accuracy and are time-consuming and labour-intensive. The combination of IoT sensor networks and UAV/satellite imagery, with DL models, gives a promising solution for accurate real-time yield prediction. In this study, sensor data (soil moisture, weather parameters, pH, NPK levels, temperature, and leaf chlorophyll content) are collected using devices like Davis Vantage Pro2, Decagon 5TE and SPAD-502, while image data are captured using DJI Phantom 4 Multispectral UAVs and Sentinel-2 satellite imagery. Pre-processing of sensor data involves missing value imputation, normalization, and feature selection using the HFLO algorithm, while image data undergo resizing, augmentation, Enhanced Median Filtering approach. Image based features are extracted by a Position Attention-aided DenseNet-201 model. Furthermore, the features from image data and sensor data using a concatenation mechanism. Finally, fully connected layers and ISVR are used to identify the tapioca crop yield prediction. The framework of the proposed methodology is denoted as Figure 1.

tu_pian_10.png

Figure 1. Framework of proposed methodology

3.1 Pre-processing

In this context, the preprocessing method is described to process the IoT sensor data and imagery data. The sensor data is efficiently preprocessed using missing value imputation and normalization methods. Here, image preprocessing for improving the quality and remove unwanted noise. Here, resizing, augmentation, and enhanced median filtering efficiently unify the strengths of preprocessing satellite images, resulting in enhanced image quality and better feature protection. This pre-processing step is essential for subsequent analyses in satellite data and IoT applications.

3.1.1 Sensor data processing

Missing value imputation: Due to hardware failures, transmission faults, or external interference, sensor data frequently contains missing or partial values. The integrity of downstream data analysis depends on how these missing values are handled. To confirm and close these gaps, imputation techniques are used [30]. The type of data and the anticipated effect on analysis accuracy determine the method to use.

Normalization: Normalization is a crucial stage in the preparation of sensor data, especially when the data comes from multiple sources of different sizes. This process transforms the data into a common scale without distorting variances in the range of values. Common normalization methods include min-max normalization, which scales data to a fixed range (typically 0 to 1), and z-score normalization, which ensures that all sensor features contribute equally to analysis, such as ML models [31], preventing any one variable from disproportionately influencing the results. Eq. (1) is used in min-max normalization to scale data to a specified range (0 to 1).

$y^{\prime}=\frac{y-y_{\min }}{y_{\max }-y_{\min }}$           (1)

Increases the stability of model training by maintaining the data distribution while guaranteeing that all features are on the same scale.

3.1.2 Imagery data processing

Resizing: Resizing is commonly used in sensor data processing, especially when dealing with time-series or image-based sensor outputs. It involves adjusting the dimensions or length of the data to a uniform size, which is essential for consistent input into analytical models or neural networks [32]. For instance, in time-series analysis, sensor readings may be resampled to a fixed time interval, ensuring that each sample has the same number of data points. In image data from sensors (e.g., thermal or optical sensors), resizing ensures compatibility with model input dimensions while maintaining the aspect ratio or key features of the original data.

Augmentation: Augmentation refers to the process of artificially increasing the diversity and volume of sensor data by applying various transformations. This is particularly useful in training ML models, where more varied data can improve generalization and robustness. In the context of sensor data, augmentation techniques may include adding noise, shifting time windows, rotating or flipping images, scaling values [33], or simulating environmental variations. These transformations help models learn to be resilient to real-world variability and reduce overfitting on limited datasets.

3.1.3 Enhanced median filtering

In contrast, utilizing enhanced median filtering is utilized to improve image quality and to reduce unwanted noise [34]. Although regular median filtering is also used for this purpose, it tends to produce more errors in this case. To solve that, this research uses enhanced median filtering. Next, bringing in the average error point as a metric helps reduce the error in the images and gives better results. The error point metrics as follows:

$A=\frac{1}{M \cdot F} \sum_{x=1}^M \sum_{y=1}^F\left|o_{x, y}-\hat{o}_{x, y}\right|$          (2)

where, $o_{x, y}$ is the image at trace $x$, point $o$ in the original image, and $\hat{o}_{x, y}$ denotes the filter image. Here, the number of samples per trace. Furthermore, the outputs indicate that median filtering approaches perform the best, with the modified adaptive median filter having the minimum average error.

3.2 Feature selection using Hybrid Frilled Lizard Osprey optimization

In order to reduce redundancy and improve prediction accuracy, the suggested HFLO algorithm is used for optimal feature selection. The presented HFLO approach incorporates the exploration ability of the FLO and the exploitation robustness of the osprey optimization; the hybrid approach obtains a better balance between global search and local search. This helps in avoiding premature convergence and ensures a more accurate and diverse selection of optimal features. As a result, the approach improves predictive performance, reduces computational complexity, and enhances convergence speed compared to employing individual algorithms alone. Finally, this presented hybrid approach offers a strong and efficient solution for extracting the most relevant features from the collected database.

3.2.1 Initialization

Initialize the population of candidate feature subsets by simulating the behaviours of frilled lizard and ospreys optimization, aiming for a balance between exploration and exploitation. Define the factors: $D$ is the total number of features in the database, population is $P, A \in\{0,1\}^{P \times D}$ is a binary population matrix where each row represents a feature subset.

3.2.2 Exploration using Frilled Lizard Optimization [35]

The HFLO algorithm blends the exploitation power of the Osprey Optimization Algorithm (OOA) with the exploration capacity of the FLO. While local refinement is carried out in the exploitation phase to enhance convergence toward optimal feature subsets, candidate solutions are updated using global search strategies during the exploration phase to prevent local optima. Each agent's position is updated iteratively as:

$C I_i=\left\{P_F: O_F<O_i\right.$ and $\left.F \neq i\right\}$            (3)

where, $C I$ is the candidate feature set for the $i^{\text {th}} \mathrm{FL}, P_F$ denotes the population feature with a better $O_k$ is the objective function value than the $i^{\text {th }} \mathrm{FL}$, and $i=1,2, \ldots, N$ along with $k \in\{1,2, \ldots, N\}, F$ represents the features. The FLO algorithm assumes that each searcher randomly selects one feature from the set of selected feature positions and proceeds to predict. The movement of the searcher toward the candidate feature is mathematically modelled, and this approach is employed to update each individual’s position in the population, as defined in Eq. (4). If the new position has a better objective value, it becomes the individual’s new position, as shown in Eq. (5).

$\begin{gathered}p_{i, d}^{x 1}=p_{i, d}+r \cdot\left(S I_{i, d}-t \cdot p_{i, d}\right), \\ d=1,2, \ldots, m\end{gathered}$        (4)

$P_i=\left\{\begin{array}{cc}P_i^{x 1}, & O_i^{x 1}<O_i \\ P_1, & \text { else }\end{array}\right.$          (5)

Let $P_i^{x 1}$ is the new suggested position of $i^{\text {th }}$ searcher based on the first phase of FLO, $p_{i, d}^{x 1}$ denotes its $d^{t h}$ dimension, $O_i^{x 1}$ is the objective function amount, $r$ is a random number with a normal distribution from the interval [0, 1], O_i represents the $i^{\text {th}}$ objective function, $S I_{i, d}$ is the $d^{\text {th}}$ dimension of the selected feature for $i^{\text {th}}$ searcher, $t$ denotes the randomly taken from the set $\{1,2\}$, and $m$ represents the number of decision variables.

3.2.3 Osprey using exploitation [36]

This phase deeply explores the features and discovers the features to catch. In a simulation of this real conduct, the second degree of updating the records in the OOA is modeled. The hunt agent’s function in the search space is created by means of small adjustments resulting from modeling the identity of the function to the proper role. The OOA’s exploitation improved power inside the neighborhood search and caused convergence closer to higher solutions towards the found solution. In the OOA design, a new random role was initially decided for each member of the data. This simulated the herbal conduct of search agents. The exploitation segment is given as:

$a_{i, j}^{P 2}=x_{i, j}+\frac{l b_j+r \cdot\left(u b_j-l b_j\right)}{t}, j=1,2, \ldots \ldots \ldots, m$,           t=1,2,........, T(6)

$A_{i, j}^{P 2}=\left\{\begin{array}{r}a_{i, j}^{P 2}, l b_j \leq a_{i, j}^{P 2} \leq u b_j ; \\ l b_j, \quad a_{i, j}^{P 2}<l b_j ; \\ u b_j, \quad a_{i, j}^{P 2}>u b_j,\end{array}\right.$            (7)

$A_i=\left\{\begin{array}{r}A_i^{P 2}, F<F_i \\ A_i, \text { else},\end{array}\right.$            (8)

where, $A_i^{P 2}$ is the $i^{t h}$ search agent's original place based on the next phase of the OOA, $r$ represents the random number, $l b$ represents the lower bound, $u b$ represents the upper bound, $a_{i, j}^{P 2}$ is the $j^{\text {th}}$ aspect, $F$ is its fitness role, $r_{i, j}$ are random numbers in the range $[0, 1], t$ is the iteration number of the method, and $T$ is the total number of iterations. The fitness function is calculated as:

$F=\lambda * E+(1-\lambda) * \frac{|S|}{d}$           (9)

where, $E$ signifies the prediction error, $|S|$ signifies the number of selected features, and $\lambda \in[0,1]$ provides a balance between feature reduction and accuracy. The population size $N=30$, maximum iterations $T=100$, and control parameters $\alpha=0.5, \beta=0.3$ constitute the algorithm's set parameters. In order to balance convergence speed and solution quality, these parameters were chosen through empirical tweaking. For moderate feature dimensions, the HFLO algorithm's computing complexity of $O(N \times T \times d)$ is acceptable. All things considered, the HFLO algorithm offers a good trade-off between exploration and exploitation, resulting in the selection of an ideal subset of pertinent characteristics that enhances model generalization and lowers computing cost. Algorithm 1 provides the HFLO optimization pseudo code.

3.3 Position attention – DenseNet-201 model for feature extraction

Employing the suggested feature extraction model (Position Attention DenseNet-201) helps the model understand tapioca images better. DenseNet-201 model was better at extracting deep features from images while enhancing captured information. The position attention module (PAM) identifies long-range dependencies, enables efficient learning, and finds significant regions like diseased leaves, specific plant shapes, or textures. This effectiveness enables the model to focus on the tapioca areas and ignore the background (i.e., soil moisture, weather parameters, pH, NPK levels, temperature, and leaf chlorophyll content). Subsequently, the suggested model accurately predicts health conditions or growth stages. The integration enhances accuracy, efficient learning, and reduces computational power.

3.3.1 DenseNet-201

This model is an effective CNN widely employed for feature extraction in DL challenges, particularly in computer vision (Figure 2). Its framework is characterized by dense connections; here each layer receives inputs from all preceding layers, promoting feature reuse and efficient gradient flow. This framework enables DenseNet-201 to extra high [37], hierarchical features with fewer parameters compared to other deep networks like ResNet152 [38], making it both precise and relatively lightweight. When utilized as a feature extractor, the final classification layer is removed, and the output from the last convolutional block or global average pooling layer is utilized as a high-quality feature vector. These features are particularly effective in applications such as medical imagery, plant leaves, fine-grained classification, and object detection, often outperforming the effectiveness of deep models. Although integrating a position attention model in the DenseNet-201 model, it demands reduced complexity due to its dense connectivity, DenseNet-201 offers excellent generalization and has shown competitive performance on benchmark databases, making it a reliable choice for transfer learning and feature extraction pipelines.

3.3.2 Position attention

Image understanding depends significantly on discriminative feature representations, which can generally be improved by the inclusion of long-range contextual cues. To suggest adding a PAM. As shown in Figure 2, this module is tailored to learn spatial dependencies over the whole image, allowing the model to represent rich contextual relations beyond local domains. In doing so, the PAM actually boosts the representational capacity of the local features. In the next sections, explain how the PAM adaptively sums up spatial contextual information and achieves better image understanding task performance.

Figure 3 shows the local feature $P \in \Re^{c \times h \times w}$ is fed into convolution layers to build two new feature maps $G$ and $M$, separately, $\{G, M\} \in \Re^{c \times h \times w}$. Following the restructure them to $\Re^{c \times N}$, where $N=h \times w$ represents the number of pixels. Afterward, to calculate the attention map $A \in \Re^{N \times N}$, multiply the transpose of $M$ and $G$ use a softmax layer.

$a_{y x}=\frac{\exp \left(G_x \cdot M_y\right)}{\sum_{x=1}^N \exp \left(G_x \cdot M_y\right)}$           (10)

tu_pian_11.png

Figure 2. DenseNet-201 model

tu_pian_12.png

Figure 3. Position attention module (PAM)

Here $a_{j x}$ quantifies how the $x$th position affects the $y$th position. The two feature representations that are more alike at a position increase their correlation with one another. At the same time, transmit feature $P$ into a convolution layer to build a new feature map $F \in \Re^{c \times h \times w}$ and reshape. Send it to $\Re^{c \times N}$. Next, multiply $L$ by the transpose of $A$ and reshape the result into $\Re^{c \times h \times w}$ matrix. The final output $R \in \Re^{c \times h \times w}$ is obtained by multiplying it by a scale parameter $\alpha$ and performing an element-wise sum operation on the features $P$.

$R_y=\alpha \sum_{x=1}^N\left(a_{y x} L_x\right)+P_y$           (11)

Here, $\alpha$ is initially set to 0 and subsequently learns to attach increasing weight. Eq. (11) indicates that the feature R at each place is a weighted sum. All the features in all positions and original features.

3.4 Concatenation mechanism for feature fusion

Concatenation is a fundamental feature fusion method in multisource DL employed to integrate heterogeneous data sources into a unified field representation. In contrast, sensor data and UAV multispectral imagery capture complementary information about tapioca and environmental conditions. The developed concatenation mechanism enables the model to exploit both spatial (image-based) and non-spatial (sensor-based) features jointly for enhanced prediction accuracy.

Mechanism: Let $I_s \in \Re^{n_s}$ is the IoT sensor data vector, and $I_x \in \Re^{h \times w \times c}$ is the UAV/satellite imagery patch with B spectral bands. Each modality is first passed through its own encoder network, sensor feature embedding as follows:

$s_z=f_z\left(I_z\right) \in \Re^{d_z}$          (12)

Here, $f_z$ denotes the fully connected neural network (MLP) for tabular sensor data, $s_z$ is the dense feature embedding of sensor data. Image embeddings are expressed in Eq. (13):

$s_x=f_x\left(I_x\right) \in \Re^{d_x}$             (13)

where, CNN or pretrained image encoder for extracting features from multispectral images is denoted as $f_x$, and $s_x$ is the image feature embedding. Concatenation (fusion layer): The embeddings from both modalities are concatenated to form a joint feature vector:

$s_f=\left[s_z \| s_x\right] \in \Re^{d_s+d_x}$          (14)

Here, vector concatenation is represented as ||, $s_f$ is the concatenated multimodal. The method vector captures both conditions, allowing the model to learn interactions. As a result, this method offers simplicity and effectiveness, while being well-suited for easy-to-implement and compatible neural network architectures. Features and network architecture dimensions are given in Table 2.

Table 2. Features and network architecture dimensions

3.5 Tapioca yield prediction using improved support vector regression

In this study, an improved SVR model is utilized for tapioca yield prediction. To improve the learning ability, initially, integrates a CNN model [39] with SVR, because its performed efficient and reduces the time consumption. This approach is a supervised learning algorithm derived from SVMs [40, 41], developed for predicting continuous values rather than class labels. The main thing in SVR is to find a function that approximates the data within a certain $\varepsilon$ margin of tolerance, while maintaining model simplicity. As a mathematical and statistical theory-based regression model, the objective of SVR is to create a functional relationship between input variables $a=\left(a_1, a_2, a_3, \ldots, a_k\right)^T$ and the respective response $b$. Through the introduction of the non-linear feature mapping function $\psi(a)$, SVR is able to map the input data into high dimensional linear space. Thus, the relationship can be represented as a linear regression function in the highdimensional space.

$\hat{b}(a)=\omega^T \psi(a)+s$           (15)

Here, $\widehat{b}(a)$ is the approximation response, the weighted vector denoted as $\omega$, and $s$ is the bias term. To develop a wellgenerated strategy, the $\varepsilon$-insensitive constraint is employed. Given a training dataset or subset ($a_x, b_x$) , where $a=$ $1,2,3, \ldots, n$, the linear regression task can be reformulated as a constrained convex problem.

$\min =\frac{1}{2}\|\omega\|^2 ;\left\{\begin{array}{c}\omega^T \psi\left(a^x\right)+s-b^x \leq \varepsilon \\ b^x-\omega^T \psi\left(a^x\right)-s \leq \varepsilon \\ x=1, \ldots, n\end{array}\right.$          (16)

Eq. (16) is reformulated by incorporating the slack variables $\xi_x^{+}$ and $\xi_x^{-}$ as follows:

$\min =\frac{1}{2}\|\omega\|^2+C \sum_{x=1}^n \xi^{+(x)}+\xi^{-(x)} ;\left\{\begin{array}{c}\omega^T \psi\left(a^x\right)+c-b^x \leq \varepsilon+\xi^{-(x)} \\ b^x-\omega^T \psi\left(a^x\right)-s \leq \varepsilon+\xi^{-(x)} \\ \xi^{+(x)}, \xi^{-(x)} \geq 0 \\ x=1, \ldots, n\end{array}\right.$            (17)

Here, the regularized parameter is represented as C > 0. Utilizing Lagrangian theory, the dual formulation of Eq. (17) can be obtained.

$\begin{gathered}\max =\left\{\begin{array}{c}-\frac{1}{2} \sum_{x, y=1}^n\left(\alpha^{+(x)}-\alpha^{-(x)}\right)\left(\alpha^{+(y)}-\alpha^{-(y)}\right) \\ k\left\langle a^x, a^y\right\rangle+\sum_{x=1}^n\left(\alpha^{+(x)}-\alpha^{-(x)}\right) b^x \\ -\sum_{x=1}^m\left(\alpha^{+(x)}+\alpha^{-(x)}\right) \varepsilon \\ \left\{\begin{array}{c}\sum_{x=1}^n\left(\alpha^{+(x)}-\alpha^{-(x)}\right)=0 \\ 0 \leq \alpha^{+(x)}, \alpha^{-(x)} \leq C \\ x=1, \ldots, n\end{array}\right.\end{array}\right.\end{gathered}$          (18)

Let the kernel function is $k\left\langle a^x, a^y\right\rangle$. Compute $\alpha_x^{+}$ and $\alpha_x^{-}$ are initially obtained from Eq. (19).

$\omega=\sum_{x=1}^n\left(\alpha_x-\alpha_x^*\right) *\left(a_x\right)$          (19)

$f(a)=\sum_{x=1}^n\left(\alpha_x-\alpha_x^*\right) k\left(a_x, a\right)+b$           (20)

Here, $f(a)$ denotes the final regression output, $k\left(a, a^{\prime}\right):=$$<\phi(a)\left(a^{\prime}\right)>$ and $\phi$ represent the mapping function, $k$ is the kernel function, and $b$ is the respective response. Finally, fully connected layers to identify the tapioca crop yield prediction, while efficient results enhance performance.

The term improved SVR in this work refers to improving the standard SVR by fine-tuning parameters and optimizing feature representation. The fused multimodal feature vector created by merging sensor and image information is fed into the SVR model. In particular, satellite imagery is processed using an attention-based DenseNet-201 to generate a 256-dimensional feature vector after IoT sensor data is encoded into a 128-dimensional vector. These are combined to create a single feature vector that is fed into the SVR model. Table 3 uses an RBF kernel with hyper parameters that are optimized by the HFLO technique. Therefore, rather than changing the fundamental SVR formulation, the improvement is found in the use of deep multimodal feature fusion and better parameter selection, leading to improved prediction accuracy and generalization. Figure 4 depicts the ISVR model for yield prediction.