Comparative Evaluation of Machine Learning Models for Diabetes Prediction: A Focus on Ensemble Methods

Diabetes represents the most widespread metabolic condition and poses a serious challenge to the global healthcare system due to its associated complications. It is now recognized as a long-term, non-infectious illness and ranks among the most severe health issues following cardiovascular and cerebrovascular diseases [1]. For example, as reported in the study [2], Saudi Arabia ranks second in diabetes prevalence within the Middle East and holds the seventh position worldwide, as reported by the World Health Organization. This condition, characterized by elevated blood glucose levels [3], requires significant healthcare resources and effective management strategies due to its widespread impact. Early and accurate detection of people susceptible to developing diabetes is critical to the implementation of preventive interventions and the proper management of the disease [4]. In the past, predicting diabetes without specialized tools has been a significant challenge for physicians and specialists [5]. However, recent advances in machine learning (ML) provide a significant chance to enhance the prediction and diagnosis of diabetes [6].

The present study addresses the need for a detailed comparative evaluation of different ML models to address the challenges associated with diabetes prediction. Using a meticulous approach, the study conducts a detailed assessment of machine learning techniques. Classical techniques such as Decision Tree (DT), Naive Bayes (NB), K-Nearest Neighbors (KNN) and Logistic Regression (LR) are contrasted with ensemble methods like Random Forest (RF), XGBoost and AdaBoost, while more sophisticated paradigms, including Artificial Neural Networks (ANN) and Support Vector Machines (SVM), are also incorporated to provide a comprehensive evaluation. The main objective of the study is to use a standardized set of evaluation metrics to examine how these models perform in terms of accuracy, precision, recall, F1 score, and other relevant measures to determine which of the above ML models performs better in diabetes prediction. By taking such a comprehensive approach, we aim to provide a more nuanced and comprehensive perspective on the relative effectiveness of these models for the specific task of diabetes prediction.

The ability to predict diabetes with greater accuracy and reliability has significant implications for both clinical practice and public health. A thorough comparative evaluation allows the identification of the most effective model, facilitates its implementation in clinical settings, and improves diabetes prevention and treatment strategies. This study adds to the existing body of research by offering a direct, yet systematic comparison of several ML models based on a broad set of evaluation metrics. For this purpose, publicly available diabetes datasets are used, which are preprocessed and divided into training and test sets. The selected ML models are trained and evaluated using the specified metrics, and validation through cross-validation methods is applied to guarantee the reliability of the results.

3.1 Dataset and processing

3.1.1 Data description

For this study, data was obtained from a publicly available dataset on diabetes among Pima Indian individuals, accessed through the Kaggle platform [14]. The dataset was designed to evaluate the chances of developing diabetes by analyzing several clinical variables. All records correspond exclusively to women of Pima Indian ancestry, each at least 21 years old.

The dataset contained 768 records, each characterized by eight diagnostic attributes and a variable that specifies whether diabetes is present or not. Table 1 presents the attributes included in the dataset.

Table 1. Dataset attributes

The target variable, called “outcome,” is binary: a value of 1 signifies that diabetes is present, while a value of 0 means it is not. This variable is critical to the predictive purpose of the analysis. As shown in Figure 1, of the 768 available records, 500 records, or 65.10% of the total, correspond to patients without diabetes, while the remaining 268 records, or 34.90%, correspond to patients with diabetes. This distribution provides an adequate basis for training and evaluating predictive models, allowing a clear differentiation by distinguishing individuals with and without diabetes.

1.png

Figure 1. Distribution of positive and negative cases of diabetes

3.1.2 Training and testing data

To thoroughly evaluate different approaches using machine learning to predict diabetes, the dataset was carefully split, with 80% used for training the models and the remaining 20% reserved for evaluation. The training dataset included a representative selection of patients, covering key clinical and demographic factors such as blood glucose levels, arterial pressure, body mass index (BMI), and genetic predisposition to diabetes. This subset of data was utilized for optimizing the parameters of the ML models, allowing them to learn to recognize relationships and trends present in the clinical data. In contrast, the test dataset, made up of the leftover 20%, was set aside solely for assessing how well the models performed. This subset was not used during the training process, ensuring an unbiased and objective evaluation of the models' performance on previously unseen data.

3.1.3 Handling of unbalanced data

Given the significant imbalance in the dataset, with 65.10% of records from patients without diabetes and 34.90% of records from patients with diabetes, the SMOTE technique was applied to mitigate this problem. SMOTE was applied to produce synthetic instances of the minority class (patients with diabetes) by generating new examples through interpolation of existing data. This technique increased the presence of underrepresented cases in the training dataset, balanced the proportion of records, avoided overfitting, and enhanced the model’s capacity to generalize to unseen data.

Imbalances between classes frequently occur in medical datasets, with the less frequent class usually corresponding to the condition being studied. SMOTE, a technique that generates synthetic instances by interpolating between minority samples and their closest neighbors, was used to address this issue [15]. Unlike methods such as ADASYN, which can introduce noise in complex regions, SMOTE maintains better class separation. Additionally, SMOTE contributes to a more representative and balanced sample, which improves model performance.

3.1.4 Cross validation

To guarantee the stability and dependability of the developed machine learning models, a 5-fold cross-validation approach was implemented. As shown in Figure 2, this approach consisted of splitting the dataset into five equal portions. During each iteration, four of these sections were allocated for model training, whereas the last one was designated for evaluating how well it performed. This process was carried out five times, with a different segment chosen for validation in each cycle, allowing for a clearer and more reliable evaluation of how well the model performs.

2.png

Figure 2. Cross-validation representation

3.2 Machine learning models

Nine algorithms based on supervised learning were selected to assess their effectiveness in predicting diabetes. To perform this evaluation, the Python programming language was used, which is known for its wide range of libraries specialized in ML. In addition, Jupyter Notebook was utilized as an interactive platform, providing a flexible and efficient environment for coding, data analysis, and result visualization. Its features allowed for seamless execution of code, facilitating both the implementation of ML models and the interpretation of their outcomes. Each of the chosen models is described below.

3.2.1 Decision tree

This ML approach is employed to address problems involving either classification or regression. As indicated in the study [16], this method allows for the identification of relationships between data in a dataset by building tree-like structures. To build a decision tree, the data is split into multiple subsets, producing a sequence of decision nodes that further branches into additional nodes. The initial node, called the root node, is the starting point of this division, while the final nodes, which are not further divided, are called leaf nodes. Decision trees are developed using a top-down structured model, which facilitates the identification of underlying trends and associations in the dataset.

3.2.2 Naïve bayes

NB utilizes Bayes' theorem as its foundation for classification, assuming that the features in the dataset are independent of one another [17]. It is a simple yet powerful approach, especially well-suited for processing large datasets efficiently.

3.2.3 K-Nearest Neighbors

The KNN classifier is a supervised ML model that makes no assumptions about the underlying data distribution, making it non-parametric. It works by considering the distance between data points, classifying a new instance according to the most common class among its closest neighbors [18]. In other words, KNN assigns a class to an object based on the predominant classes among its nearest neighboring data samples.

3.2.4 Logistic regression

RL is a statistical analysis and ML technique applied to estimate the likelihood of a dichotomous result. RL models rely on the independent relationship between predictor variables to estimate the likelihood of an event taking place [19].

3.2.5 Random Forest

RF is a machine learning approach that can be employed for solving tasks involving either classification or regression. This ensemble method builds multiple decision trees from the training data and evaluates the test data against them [20]. Through the integration of outputs from every tree, RF enhances accuracy and minimizes overfitting, producing a model that is more stable and capable of generalizing to new data.

3.2.6 Artificial Neural Networks

ANN represent a machine learning approach modeled after the organization and operation of the human brain. These networks consist of three main layers: the input layer, responsible for receiving the data; the hidden layers, where processing and feature extraction take place; and the output layer, which produces the final prediction or classification. Each layer is interconnected, enabling data flow and processing to generate precise outcomes [19]. At the initial stage, the layer receives the raw data, the hidden layers handle computations and extract features, and the output layer delivers the final prediction.

3.2.7 Support victor machine

SVM is a supervised learning method employed to separate data into two clearly defined classes. This method is trained on a previously classified data set, allowing a model to be built that reflects the characteristics of that data. The purpose of an SVM classifier is to determine which of the two categories a new data point belongs to base on the trained model [21]. Thus, SVM acts as a linear classifier that can effectively handle non-binary classification problems.

3.2.8 XGBoost

XGBoost is a sophisticated machine learning technique mainly applied to tackle both classification and regression tasks. Its key advantage is the ability to build multiple decision trees in sequence, where each new tree aims to correct the mistakes of the previous ones. This method of continuously adding trees and adjusting the features allows the model to grow and improve its accuracy with each iteration [22].

3.2.9 AdaBoost

The AdaBoost functions as an algorithm in machine learning created to improve the effectiveness of models that classify data into two categories. This iterative algorithm adjusts the weights of misclassified instances in each cycle, focusing on the most difficult examples to progressively improve the model. As an ensemble learning method, AdaBoost increases the efficiency of classifiers by identifying and correcting their errors, transforming weak classifiers into robust and accurate classifiers [23].

3.3 Evaluation metrics

3.3.1 Accuracy

It is a metric that evaluates how accurate a model's predictions are compared to the total set of available data [24]. In other words, it represents the ratio of accurate predictions compared to the total amount of data assessed.

3.3.2 Precision

It is a measure used to assess how accurately a model predicts both positive and negative outcomes [25]. It focuses on the proportion of correct predictions within a given category relative to all predictions made in that category.

3.3.3 Recall

Also known as sensitivity, it is an important metric within statistics and machine learning that evaluates the effectiveness of a classification model in identifying positive instances. More specifically, it quantifies the fraction of correctly identified positive instances out of the total number of true positives, offering insight into the model’s ability to minimize false negatives [26].

3.3.4 F1 score

Serving as a performance indicator, the F1 score merges information from both precision and recall into one unified measure, offering a comprehensive assessment of a classifier’s effectiveness. It ensures a balanced assessment by considering both Type I and Type II errors, making it particularly useful in scenarios where an even trade-off between these two factors is crucial. With values ranging from zero to one, an F1 score close to one indicates better classifier performance [27], i.e., an effective combination of precision and recall in classifying data.

3.3.5 The region beneath the receiver operating characteristic curve (ROC-AUC)

ROC-AUC serves as a key metric for assessing a model’s capacity to differentiate among different classes [28]. It provides insight into the overall effectiveness of a classifier by measuring how effectively it separates positive and negative instances. A higher AUC value indicates an improved capacity of the model to differentiate among the various classes. This measure is essential for understanding the effectiveness of a model in accurately classifying data.

3.3.6 Area under the precision-recall curve

AUPRC is a metric designed to assess the accuracy of a classification model, particularly in scenarios involving imbalanced datasets [29]. This metric evaluates the relationship between the accuracy and recall of the model, helping to understand how well the model handles both positive and negative cases. Additionally, AUPRC is critical in such settings because it focuses on the minority class, providing more informative insights than AUC-ROC when positive instances are rare.

This study evaluated how well several supervised machine learning models can predict diabetes. Although previous studies have examined machine learning approaches for diabetes prediction, the majority have focused on assessing the effectiveness of a single model or a limited group of models, without providing a comprehensive comparison across multiple models using the same set of evaluation metrics. The absence of direct comparisons has made it challenging to determine which model performs best for this particular task.

In this study, a comparative analysis of the results obtained through evaluation of test data and cross-validation reveals notable differences in the performance of machine learning models for diabetes prediction. In cross-validation, the RF model stood out with an accuracy of 0.816, an AUC-ROC of 0.911, and an AUPRC of 0.904, indicating superior performance in terms of discrimination and overall accuracy. It was followed by the XGBoost, ANN, SVM, AdaBoost, and RL models, which also showed significant performance. However, when evaluated on test data, AdaBoost stood out with the highest accuracy of 0.766, an AUC-ROC of 0.839, and an AUPRC of 0.735, indicating greater robustness to unseen data. It was followed by the RL, Random Forest, and XGBoost models, which also achieved remarkable performance.

By comparing the results with previous studies that evaluated machine learning models using similar data, it was observed that various factors such as blood pressure, maternal history, blood sugar concentration, age, insulin levels, dermal thickness, and the diabetes pedigree feature are useful for estimating the likelihood of diabetes onset. In the study by Bandhu et al. [9], decision tree models were found to have an accuracy of 78.89%, while in the study by Abdalrada et al. [13], logistic regression models showed high accuracy with a rate of 77.6% and good sensitivity and specificity. Furthermore, the study by Muntiari et al. [30] highlighted that the Random Forest algorithm achieved an accuracy of 93.5%, establishing itself as an effective method for diabetes diagnosis. In the study [7], the authors applied advanced techniques to handle class imbalance, including SMOTE and ADASYN, along with K=10 cross-validation. Their optimized extra trees model achieved strong results: 92.5% accuracy, 93.7% F1-score, and an AUC-ROC of 92.47%. In contrast, the present study used SMOTE and K=5 cross-validation, with AdaBoost emerging as the best performer on the test set, while Random Forest showed robust results during cross-validation, achieving an AUC-ROC of 0.911 (91.1%). While both studies report high predictive performance, the slightly lower AUC-ROC in the present study may be influenced by differences in model architecture, validation strategy, and data resampling techniques. Specifically, while Aouragh et al. [7] employed both SMOTE and ADASYN to enhance minority class representation, our approach relied solely on SMOTE, which may affect the balance and diversity of synthetic samples. Additionally, the use of fewer cross-validation folds (K=5 vs. K=10) could lead to increased variance in performance estimates. Despite these differences, both studies reinforce the importance of ensemble methods and class balancing in improving diabetes prediction. However, the nuanced methodological choices underscore the need for careful tuning and evaluation to ensure model generalizability.

The observations indicate that the Random Forest and AdaBoost ensemble models demonstrate the highest consistency when assessed using cross-validation and test datasets. Furthermore, XGBoost, ANN, SVM, and RL also show significant performance with both scoring methods in diabetes prediction when analyzed with different metrics, including AUC-ROC and AUPRC, and considering factors such as blood pressure, pregnancy, glucose level, age, and insulin, among others. This comparison highlights the necessity of applying various evaluation techniques to obtain a comprehensive understanding of how machine learning models perform in predicting diabetes. The detailed and comprehensive comparison allows the identification of the most effective and robust models, providing a solid foundation for future research and clinical applications in this field.

From a clinical perspective, deploying these models has the potential to significantly enhance decision-making processes. Integrating the predictive system with electronic health records (EHRs) could allow for real-time assessment of patient risk. This would allow clinicians to identify high-risk patients early and allocate resources more efficiently. Such integration can also facilitate seamless workflows, reduce diagnostic delays, and support personalized treatment strategies. However, for successful adoption, considerations such as model interpretability, data privacy, and regulatory compliance must be addressed to ensure safe and effective clinical implementation.

However, a significant limitation of this study is its reliance on the Pima Indian dataset, which only includes female patients from a single ethnic group. This homogeneity limits the applicability of the models to populations with diverse gender and ethnic backgrounds. To ensure robustness and applicability in real-world clinical settings, future work should consider training and validating the models on more heterogeneous datasets that incorporate demographic variables such as gender, ethnicity, and age.