A Multimodal-Multitask Learning Framework for Intelligent Mango Grading and Decision Support

In farming, accurate estimation of the quality and maturity of fruits is essential in the post-harvest phase of production as well as the price evaluation process. For mangoes, these features are key determinants of their preference, usability, and price [1]. The conventional practice of grading commercial mangoes in Malaysia mainly utilizes visual inspection, depending upon parameters like maturity level, outer appearance, size, and any visible flaws in their surface. This procedure is time-consuming, biased, and may lead to inconsistent results because of differences among the operators as well as inherent differences in fruits themselves.

 Thus, there exists a need for a smart automated grading system for commercial purposes. In recent times, as a response to the emerging requirements in precision agriculture, the need for automation and a data-oriented approach in assessing these characteristics of fruits is on the rise. With recent advancements in machine learning and computer vision, automated fruit classification systems have been developed, especially by using images [2]. However, the current systems mainly classify fruits using the unimodal technique, where the prediction of these qualities is seen as distinct from each other. In this system, the physical and physiological state of the fruit is not fully considered; the inclusion of additional information, such as the weight of the fruit, improves such classifications [3].

To overcome the mentioned challenges, the suggested research employs a multimodal and multitask learning system. In this approach, the visual features of mango images are combined with weight features to predict maturity and quality, leading to increased prediction accuracy. A machine learning system is suggested for combining multimodal features and making predictions based on multimodal data. Also, the rule-based agentic Artificial Intelligence (AI) system is employed to improve the intelligence level of the system through input validation, feedback, and decision-making processes. The suggested system is capable of evaluating the image quality, guaranteeing proper data acquisition, and providing recommendations to the user at any time. Through the combination of multimodal learning, multitask prediction, and agentic intelligence, the suggested system goes beyond classification approaches.

This section outlines the characteristics of the dataset, the structure of the model proposed in this research, and the procedures adopted in the model training. It provides information regarding the data gathering process and analysis, and the training and validation approach adopted.

3.1 Dataset description

The mango dataset comprises 276 data points, where each data point is associated with two images, front and back, of the respective mango, along with its weight value. These are labeled based on their ripeness level (P- Unripe, K- Ripe) and Malaysian Standard (MS) grade (P, 1, 2). The size of the classes of the given dataset is depicted in Table 2. The images have various colors, textures, and lighting, which are useful for classification, whereas the weights provide information regarding the physical properties of the fruit. Figure 1 shows some of the representative samples from the mango datasets being used in the current study. The dataset used in the current study can be accessed at https://www.kaggle.com/code/mypapit/harumanis-mango-mass-weight-estimation-from-photo.

Table 2. Class-wise sample size distribution

image003.png

Figure 1. Representative examples of the mango dataset

Each of the mango samples is made up of front and back Red, Green, and Blue (RGB) images, making a total of 552 images. Data augmentation is performed using random rotation, flip, and changing brightness to increase the number of images to 1,656 images. During the partition of the dataset, the GroupShuffleSplit approach is used, where the mango sample is taken as the group. As a result, each image related to the same mango sample, including its augmented form, is allocated to either the training or test set exclusively. The dataset was, therefore, split into 1,242 images for training and 414 images for testing.

3.1.1 Image pre-processing

The images of the mango fruits are resized into the dimensions of 224 × 224 pixels for uniformity in all samples. The visual feature extraction consisted of three types of complementary features. Color features were extracted based on color histograms that provided the distributions of hue, saturation, and intensity components [10]. Texture features were extracted through Gray-Level Co-occurrence Matrix (GLCM)-based techniques like contrast, correlation, energy, and homogeneity. Moreover, statistical measures like mean, standard deviation, skewness, and kurtosis were extracted to analyze the intensity distribution [11]. The feature set obtained from the front and back images are then concatenated together to create one feature vector that is further fused with normalized weight features. The feature vector obtained from each image is as follows:

$F_i=\left[f_1, f_2, f_3, \ldots \ldots, f_n\right]$   (1)

where, n is the number of image features. As the samples have images of mangoes from front and back view, their feature vectors are concatenated together as:

$F_{i m g}=\frac{1}{N} \sum_{i=1}^N F_i$   (2)

where, N stands for the total number of different viewpoints at which the fruit can be seen. In this study, N = 2 as the research is done on two viewpoints: front view and back view of the same mango.

3.1.2 Weight pre-processing

The mango weight data are considered as numeric attributes and are scaled and normalized to ensure uniformity in their range [12]. This ensures that the weight values play an effective role in model training without overshadowing other attributes [13]. The normalization for weight is performed through min-max normalization and it is given by:

$W_{norm}=\frac{W-W_{min}}{W_{max}-W_{min}}$   (3)

where, W is the original weight of the mango sample, W_min is the minimum weight, while W_max is the maximum weight found in the dataset. The process of normalization guarantees that the weight values will be within the range of [0, 1]. This allows the use of normalized weight values in conjunction with the image features to generate the input to the model.

3.2 Model architecture

The suggested architecture, as illustrated in Figure 2, adopts an organized pipeline for intelligent decision support in mango fruit analysis. The first stage entails acquiring data and preprocessing image and weight data [14]. Feature extraction transforms the input features into discriminative feature vectors, which are combined using an early fusion approach and fed to a multimodal multitask learning classifier for predicting maturity and quality simultaneously [15]. The rule-based module acts as a validator and helps in making decisions. The recommendation and decision support component allows the system to evolve with time, with suitable buying and selling decisions [16].

image007.png

Figure 2. Multimodal mango analysis framework

3.2.1 Multimodal-multitask Gradient Boosting pipeline

image008.png

Figure 3. Early fusion Gradient Boosting architecture

The model architecture begins by fusing diverse features at an early stage, as depicted in Figure 3. Independent feature vectors derived from different angles of an object are initially merged (for example, through averaging to mitigate viewpoint effects and variance) and subsequently combined with weighted structured feature sets to construct one high-dimensional feature vector [17]. The fused feature vector is defined as:

$F_{fusion}=F_ {img} \oplus W_{norm}$   (4)

where, ⊕ represents the concatenation symbol. The procedure enables the model to learn the relationships between modalities, hence improving the joint learning across several objectives. The fused features are then processed using the Gradient Boosting algorithm that includes the construction of Decision Trees (DTs) in stages. The first stage of the procedure involves the construction of a prediction model:

$F_0(x)=\arg \min _{\{\gamma\}} \sum_{i=1}^N L\left(y_i, \gamma\right)$   (5)

where, γ represents the constant prediction value, N is the total number of training examples, L is the loss function and y_i represents the true class label. Initially, the model makes a base prediction using the selected loss function on all the training examples. This baseline is used as a starting point for boosting. Then, the residual error at the m^th iteration becomes:

$r_{i m}=-\left[\frac{\delta L\left(y_{i,} F\left(x_i\right)\right)}{\delta F\left(x_i\right)}\right]$   (6)

In each iteration, the tree learns to fit the residuals from the preceding model, minimizing loss through the assignment of weights to weak learners and optimal splitting of the most significant features [18]. The model is then updated to:

$F_m(x)=F_{m-1}(x)+\eta h_m(x)$   (7)

where, $\eta$ is the learning rate, $F_{m-1}(x)$ is the previous model and $h_m(x)$ is the newly trained DT. The prediction function of the model is updated based on the addition of the weighted contribution of the new weak learner.

3.2.2 Agentic logic and decision making

The suggested framework, as presented in Figure 4, incorporates the design of an agentic input validation unit for ensuring the accuracy and reliability of mango evaluation. Multi-view RGB images and weight values are initially fed into an agentic framework that automatically executes a quality assurance process. This includes operations such as blur detection, alignment consistency, illumination equalization, duplication/sampling validation, and weight value validity verification, facilitated by the use of a closed-loop feedback system [19]. The rule-based validation module executes a set of quality tests that are predefined prior to extraction of features. The blur is detected by measuring the laplacian variance, with rejection of images that fall below the predefined value. The illumination quality is tested using the histogram brightness analysis, which helps to detect the images that are either underexposed or overexposed. The orientation is checked to confirm availability of complementary views for both front and back sides, and Structural Similarity Index Measure (SSIM) index is used to ensure that the images belong to the same sample of the fruit. Finally, the weight measurement of the fruit is checked against the predefined acceptable values, which are calculated on the basis of the dataset.

image017.png

Figure 4. Rule-based decision support framework

Once the inputs are validated, task-specific feature extraction is executed, and the resultant features are combined to form a high-dimensional feature vector. The fused feature vector is then employed as the input in a multitasking Gradient Boosting approach where a sequence of optimized DTs are trained for detecting the interaction between different nonlinear features to simultaneously classify ripeness and grade [20]. The decision support system has been included to show the application of the proposed framework. The buy/sell suggestion and price forecasting system has been developed as a prototype to highlight how the system could be used in reality [21].

3.3 Experimental setup

This section deals with the evaluation of various machine learning models along with different types of fusion approaches for the classification of mango. This experiment has been conducted in three stages: baseline model comparison, fusion-based experiment, and final system selection. In the first stage, seven models have been trained by the extracted feature dataset, where Random Forest (RF), Support Vector Machine (SVM), Naive Bayes (NB), DT, k-Nearest Neighbor (KNN), Extreme Gradient (XG) Boosting, and Gradient Boosting are evaluated. The fusion-based experiments, like Multi-View and Stacking, along with ensemble-level methods, have shown some variations, making the results less reliable [22]. As per these findings, Gradient Boosting, along with the early fusion method, is considered as the classifier since they showed consistency in all scenarios [23].

Under the early fusion approach, the disparate features derived from both the RGB images and the normalized weight attribute are concatenated at the feature level to create one single feature vector. Before concatenation, all the numeric features are scaled down to the same standard to avoid biasness in any particular feature because of disparity in their magnitude. The fused feature vector holds all the relevant information pertaining to color, texture, statistical, and weight attributes and forms the common input feature vector for the Gradient Boosting classifier.

The suggested multi-task model involves a common multimodal feature representation which is obtained from early feature fusion. Two separate Gradient Boosting models are then trained for ripeness classification and Malaysian grade classification respectively, thereby sharing the feature space but keeping task specific model parameters independent.

3.3.1 Model training

The GroupShuffleSplit strategy is employed to avoid data leakage in order to place the front and back pictures of the same fruit in the same fold. The dataset is split into 75% training (1242 samples after augmentation) and 25% testing (414 samples). In order to validate the model’s ability to generalize, the performance of the model is evaluated through 5-fold cross-validation [24]. The feature extraction process involves the extraction of color, texture, and statistical features, which are then weighted and fused through early fusion. RobustScaler is used to scale the features in order to minimize the influence of the outliers. Initially, the training is done using default parameters of the Gradient Boosting algorithm (n_estimators = 100, learning_rate = 0.1, max_depth = 3, subsample = 1.0, loss = deviance), for both ripeness and grade predictions. The model learns step-by-step through sequential DTs, making corrections to its previous mistakes [25].

Prior studies have proved that fruit classification can be reliably accomplished through small datasets [26]. However, in the current research, the model is trained on 276 samples of mango fruits for two classifications- fruit maturity and quality assessment, by using multimodal data (images and weight). However, increasing the amount of data is important in future to generalize to different conditions under which the mangoes have been cultivated and imaged.

3.3.2 Hyperparameter tuning and optimization

The hyperparameters are adjusted in order to optimize the performance and yet maintain generalization. The main hyperparameters are nestimators, learning_rate, and max_depth, which refer to the number of trees, tree contribution, and the complexity of the model, respectively [27]. Parameter combinations are evaluated using cross-validation, with n_estimators ranging from 50 to 150, learning_rate between 0.05 and 0.2, and max_depth between 3 and 7. The results suggest that adding more estimators greater than 100 would yield marginal benefits but at the expense of additional calculations, while increasing tree depth will result in overfitting within the context of fusion. The optimal model setting is therefore (n_estimators = 100, learning_rate = 0.1, max_depth = 5).

3.4 Model evaluation

The performance of the model is analyzed using the following metrics [28]:

•Accuracy: The percentage of the number of mangoes where the model correctly predicts maturity and quality from the total number of tested mangoes.

•Precision: The percentage of mangoes classified under a certain category (e.g. "ripe good") that really belong to that category.

•Recall: Number of mangoes from the given class (for example, “overripe”) that are correctly recognized by the model.

•F1-Score: Balanced measure based on precision and recall metrics to be applied for classification tasks, in case class imbalance (e.g., ripe versus unripe) exists.

•Confusion Matrix: A table showing how many mangoes are correctly and incorrectly classified into each maturity/quality class (e.g., true ripe, false ripe, missed ripe, etc.).

This study provides an efficient and scalable solution for automated mango quality evaluation and recommendation through a multimodal multitasking agent-based model. The visual data extracted from multi-view images is combined with structured weight information to provide reliable predictions. The addition of an agentic validation and reasoning layer increases the robustness of the model because it guarantees the credibility of the input information and allows for making intelligent decisions regarding feedback, which solves the problem of the transition from the purely computational model to its practical application. In addition, the experiments conducted on the practical example prove the applicability of the suggested system in real life. Future research will include validation of the proposed framework through large-scale datasets and comparison of its results with those of other state-of-the-art deep learning fruit grading models. In general, this research makes contributions not only in the field of automated fruit grading techniques but also in intelligent agriculture technologies.