Advancing Diabetic Retinopathy Severity Classification Through Stacked Generalization in Ensemble Deep Learning Models

Diabetes, characterized by elevated and fluctuating blood sugar levels, stands as an incurable disease impacting over 400 million individuals globally. It is projected that by 2040, approximately 642 million middle-aged individuals will be affected by diabetes [1]. This chronic condition not only poses a direct threat to life but also precipitates secondary complications in various bodily systems, including the heart, blood vessels, kidneys, and eyes. Among these complications, DR, a consequence of retinal vessel damage induced by high blood sugar, is a prominent concern.

DR is typically categorized into two principal stages: non-proliferative and proliferative [2]. The non-proliferative stage, being the initial phase, is marked by blurred vision due to leakage from minute blood vessels. The proliferative stage represents an advanced progression of the disease. In the absence of early detection, the retina commences the formation of new blood vessels, which are prone to bleeding. The severity of bleeding correlates with the risk of intermittent or permanent vision loss [3].

The necessity of early detection of DR to prevent vision impairment is well-acknowledged. Annual diabetic eye screenings are advocated for individuals with diabetes. These screenings entail the examination of retinal vessels through photographic imaging, facilitating the early identification of DR prior to any significant impact on vision [4]. Timely diagnosis enables patients to mitigate the side effects of DR through lifestyle modifications.

In the diagnostic process, photographs from diabetic eye screenings are scrutinized by medical professionals, with a focus on blood vessel and retinal feature analysis. To enhance this diagnostic approach, the application of deep learning methods is being explored. Deep learning, a subset of computer science, endeavors to emulate human brain functions to address complex issues [5]. The implementation of deep learning techniques offers the potential to utilize existing eye screening data for the predictive analysis of new screening images.

The literature has seen the publication of several datasets, notably APTOS 2019 [6], EyePACS [7], IDRiD [8], and Messidor [9], all aimed at advancing automated deep learning-based tools for DR detection. These datasets are composed of retinal eye fundus color images, each presenting varying severity levels of DR, and have been clinically graded by medical experts. The primary challenges associated with these datasets are their size and the distribution of classes within them. For instance, IDRiD exhibits a limited number of image samples across each class, presenting a significant hurdle for deep learning-based research. Conversely, the APTOS 2019 dataset is characterized by an imbalance in the quantity of images per class, which introduces bias into the classification results.

Focusing on the APTOS 2019 dataset, studies utilizing the transfer learning approach have yielded commendable results in classification performance. Sheikh and Qidwai's investigation involving four pre-trained networks-DenseNet, VGG, ResNet, and Inception models-revealed that the utilization of PNG format images in model training enhanced classification performance when compared to JPEG format images [10]. They underscored the critical role of normalization in achieving more precise and consistent outcomes. Their research documented classification accuracies on the APTOS 2019 dataset using PNG format images post-normalization: 90.5% with DenseNet121, 79.3% with VGG16, 84.7% with ResNet50, and 85.8% with InceptionV3 models.

In a similar vein, Rahman and Dola explored the efficacy of data augmentation techniques applied to original RGB format images as a preliminary step in dataset balancing [11]. They then applied three deep learning networks within a transfer learning framework to the APTOS 2019 dataset. Their findings indicated classification accuracies of 96.54% for DenseNet169, 90.93% for DenseNet121, and 90.93% for ResNet50 models.

In a distinct approach, Lu et al. [12] implemented an enhancement of images by integrating YPbPr and brightness spaces. Their methodology incorporated various image processing techniques including resizing, contrast-limited adaptive histogram equalization, and background removal. A custom architecture was developed, comprising two distinct blocks: the ShuffleNet V2 basic block and the ShuffleNet V2 block for spatial downsampling. This approach yielded a maximum accuracy of 96.66% on the APTOS 2019 dataset.

Yadav et al. [13] explored multiple architectures, such as InceptionResNetV2, InceptionV3, Xception, MobileNetV2, VGG19, and DenseNet201. Their findings highlighted InceptionResNetV2 architecture as the most accurate model.

Considering the importance of image processing in computer vision problems, Samanta et al. [14] applied crucial preprocessing steps such as the elimination of non-informative black backgrounds and the application of CLAHE (Contrastive Limited Adaptive Histogram Equalization). Utilizing a transfer learning approach with the DenseNet121 network on the APTOS 2019 dataset, they achieved a kappa score of 0.8836 on validation data and 0.9809 on training data.

Sugeno et al. [15] commenced their study by removing blurry or duplicate images from the dataset, followed by extracting non-essential content from the background. They also applied a Laplacian filter and introduced a masking method for red or white lesions. Post-preprocessing, the EfficientNetB3 model was implemented under a transfer learning framework, achieving a classification accuracy of 84% on the APTOS 2019 dataset.

In a similar endeavor, Gangwar and Ravi [16] focused on the Messidor and APTOS 2019 datasets to determine DR severity. Techniques such as blurring, background elimination, and data augmentation were employed during preprocessing. Their architecture included a modified pre-trained InceptionResNet-v2 model with additional convolutional layers. The proposed model attained test accuracies of 72.33% and 82.18% on the Messidor and APTOS 2019 datasets, respectively.

Islam et al. [17] adopted the CLAHE technique for image enhancement on the APTOS 2019 dataset as an initial step. Moreover, they utilized supervised contrastive learning, a method involving two augmented versions of the same input to identify similar feature sets for varying input images' detection processes. This technique led to classification accuracies of 98.36% for DR and non-DR categorization and 84.36% for predicting DR severity.

Ingle and Ambad [18] incorporated CLAHE in part of their research, developing two models using CLAHE and FastAI model generation techniques. They implemented image preprocessing steps, such as rotation, scaling, and lighting, to improve model accuracies before introducing the augmented data to a pretrained ResNet50 model. Their findings indicated accuracies of 93% for the CLAHE-based method and 95% for the FastAI-based approach on the processed APTOS 2019 dataset.

Hayati et al. [19] also adopted a CLAHE-based image enhancement technique to develop an accurate classifier. Post image preprocessing, they utilized pretrained models like VGG16, ResNet34, InceptionV3, and EfficientNetB4, reporting an accuracy of 97.83% on the APTOS 2019 dataset with the EfficientNetB4 model.

Aswini et al. [20] introduced a novel histogram equalization method to augment contrast. Subsequently, they employed the Grey Level Cooccurrence Matrix (GLCM) for feature extraction and fed these features into an Improved AlexNet Model-based CNN (IAM-CNN) classifier to ascertain DR severity levels.

In various studies, feature vectors have been pivotal in extracting retinal information, with the classification of these features indicating potential for enhanced results. For example, Bodapati et al. analyzed the feature maps of the pretrained VGG16 model, achieving a classification accuracy of 84.31% on the APTOS 2019 dataset [21].

Shaik and Cherukuri [22] applied CLAHE separately to the RGB channels of retinal images. They utilized the VGG16 network for feature extraction, reporting classification accuracies of 85.54% on the APTOS 2019 dataset and 66.41% on the IDRiD dataset.

In a study conducted by Shaik and Cherukuri [23], the Xception architecture was employed to extract lesion-specific features from retinal images. Utilizing these features, a neural SVM was trained, achieving accuracies of 63.24% on the IDRiD dataset and 84.31% on the APTOS 2019 dataset.

Canayaz [24] adopted a neural network-based feature extraction methodology for classifying images from both the APTOS 2019 and Messidor-2 datasets. The study employed DenseNet121 and EfficientNetB0 models for deep feature extraction, followed by the application of feature selection algorithms. The selected features were then utilized in conjunction with classical machine learning models, such as support vector machine, random forest, and k-nearest neighbor, culminating in an accuracy of 95.85%.

Zhang et al. [25] implemented a multi-point attention-based semi-supervised learning approach on the APTOS 2019 dataset, reporting an accuracy of 90.3%. Cahoon et al. [26] noted the high imbalance of classes within the APTOS 2019 dataset and advocated for data augmentation to achieve class balance, resulting in a reported accuracy of 80% post-balancing.

The concept of model or dataset ensembling has been explored in several studies to improve the prediction performance for DR severity levels. Karki and Kulkarni [27] merged two DR datasets, namely APTOS 2019 and EyePACS, to create an ensemble of deep learning models. They trained models including EfficientNetB1, EfficientNetB2, EfficientNetB3, and EfficientNetB5, and achieved a kappa score of 92.43%.

Sikder et al. [28] extracted histogram features from retinal images and constructed an ensemble of decision trees using the bagging method. They documented a classification accuracy of 91.07% on the APTOS 2019 dataset. In a subsequent study, Sikder et al. [29] eliminated noisy and duplicate images and adjusted the contrast levels of images as a preprocessing measure. The proposed ensemble of classifiers reported an accuracy score of 94.20% on the APTOS 2019 dataset.

Ensemble techniques have been a focal point in numerous studies involving deep neural networks. Bodapati et al. [30] implemented a multi-model fusion strategy using pretrained VGG16 and Inception models. Their findings indicated that this multi-model fusion significantly enhanced classification performance, achieving a DR detection accuracy of 97.82% and a DR severity prediction accuracy of 82.54% on the APTOS 2019 dataset. Kaushik et al. [31] employed a technique of concatenating the outputs of three custom-built CNNs, which were then input into an additional dense layer for classification. This approach resulted in an accuracy of 97.92% for DR detection and 87.45% for DR grading.

Yue et al. [32] introduced an attention-driven cascaded network concept, focused on detecting the severity level of DR. This model extracts context features from the current layer to aid in the generation of lesion-aware information for subsequent layers, reporting an accuracy of 83.40% on the APTOS 2019 dataset.

Playout et al. [33] developed a custom vision transformer and compared its performance with ResNet152, WideResnet101, and Optic-Net71 CNN models on the APTOS 2019 dataset. Among these, the Optic-Net71 emerged as the most accurate, with an accuracy of 92.59%.

Nahiduzzaman et al. [34] proposed an extreme learning machine architecture. Following feature scaling, the extracted features were fed into this network, culminating in a classification accuracy of 97.27% on the APTOS 2019 dataset. Bhuvaneswari and Vaidyanathan [35] generated feature maps using convolutional layers and then constructed an ensemble comprising support vector machine, random forest, and Adaboost classifiers, reporting an accuracy of 96.2%.

Some studies have concentrated specifically on the detection of the presence or absence of DR, rather than assessing its severity. In these instances, retinal images within datasets are reclassified into two groups: DR and no_DR. The primary aim of such analyses is to efficiently identify DR cases, without delving into the severity level of the condition.

Cinarer and Kilic [36] applied a Gaussian filter and data augmentation to the reorganized APTOS 2019 dataset, categorized into DR and no_DR groups. Their approach yielded classification accuracies of 95.26% for the AlexNet model and 98.17% for the VGG16 model. Dwivedi and Attry [37] explored various models within transfer learning approaches for binary classification of DR on the APTOS 2019 dataset. The highest recorded accuracy was 96.73% using the MobileNetV2 model.

Sanjana et al. [38] utilized a range of models including Xception, InceptionResNetV2, MobileNetV2, DenseNet121, and NASNetMobile on the APTOS 2019 dataset for DR detection. The respective accuracies achieved were 86.25%, 96.25%, 93.75%, 81.25%, and 80%.

Padmanayana and Anoop [39] focused on creating an accurate binary classifier for the APTOS 2019 dataset. Their methodology encompassed image processing techniques such as cropping, resizing, Gaussian blur, and CLAHE filtering, followed by inputting the images into a custom-built CNN model. They reported a test accuracy of 94.6% in DR detection.

Zhang et al. [40] adopted a source-free transfer learning approach, initially training models from scratch on the extensive EyePACS dataset, specifically for retinopathy images. Post-learning phase, these models were employed to classify the APTOS 2019 dataset, which was reclassified into two classes: DR and no-DR. This method achieved an accuracy of 91.2% on the binary dataset for DR classification.

In a study focusing on DR detection, Adriman et al. [41] extracted Local Binary Patterns (LBP) from retinal images in the APTOS 2019 dataset. They then trained models such as ResNet, DenseNet, and DetNet using these LBP feature sets. The highest accuracy of 96.35% was observed with the ResNet34 model, while Densenet121 and DetNet59 models achieved accuracies of 84.05% and 93.99%, respectively.

Li et al. [42] innovated in the field of binary DR detection by implementing a graph network. Utilizing auto-encoder feature learning for retinal feature extraction, they applied a graph convolutional network (GCN) with these features, achieving an accuracy of 94.4% on the APTOS 2019 dataset.

While previous studies have made significant strides in the machine learning-based diagnosis of DR, there remains a lack of research employing a stacked generalization approach over a soft voting-based fusion concept. The current study introduces an ensemble learning model that incorporates decision-level fusion and a stacked generalization approach using a 1D CNN architecture, aiming to enhance the classification of DR across four severity levels.

The key contributions of this research are outlined as follows: i) The proposed model was tested on the APTOS 2019 dataset, renowned in DR research, to assess its generalization capabilities and mitigate bias in unbalanced databases. ii) The individual performances of five advanced deep learning architectures, namely VGG16, VGG19, EfficientNetB5, EfficientNetB7, and EfficientNetv2S, were evaluated. iii) A combination of the decisions from individual classifiers was achieved using two majority voting schemas (hard and soft voting) for decision-level fusion. iv) The ensemble learning approach, not previously applied in DR classification literature, was anticipated to yield more accurate results than base models, particularly through the novel implementation of a stacked generalization ensemble concept. v) Experimental results underscored the effectiveness and robustness of the proposed ensemble approach in accurately determining the severity level of DR, as evidenced by high general accuracy, F1-scores, precision, and recall values.

The structure of the remainder of the study is organized as follows: Section 2 introduces the materials and methods used; Section 3 presents the experimental results; Section 4 discusses these findings; and Section 5 concludes the study.

6.png

Figure 6. Confusion matrices of the models

Table 1. The performances of individual and ensemble usage of models (%)

For validating the performance of both base and ensemble models, a K-Fold Cross-validation schema was employed, with the K-value set at five. This meant that the data was divided into training and test sets at an 80/20 ratio, with this arrangement repeated five times, each with a uniquely constituted 20% test set. The models' performances were validated based on scores obtained from the confusion matrices for each fold.

The final confusion matrices for each individual model, as well as the ensemble versions utilizing hard voting, soft voting, and the stacked generalization approach, are presented in Figure 6. The derived Accuracy, Precision, Recall, and F1 Scores from these matrices (Figure 6) are summarized in Table 1.

According to Table 1, the VGG16 and VGG19 models demonstrated comparable classification accuracies, largely due to their shared VGG architectural framework. These networks feature a sequential layer arrangement, which accounts for their higher parameter count and consequent computational and memory demands. When comparing methods with closely matched accuracies, such as VGG16 and VGG19, the F1 score, a combination of precision and recall, offers deeper insights. Notably, VGG16, being more compact and faster than VGG19, emerged as more efficient in DR classification, achieving an F1 Score approximately 1% higher. This advantage can be ascribed to the limited sample size of the APTOS 2019 dataset, which is not extensive enough to fully exploit the parameter-rich VGG19 network. Consequently, the scaled-down VGG model displayed slightly superior performance in certain classes, as indicated by its higher F-Score.

The EfficientNet architecture has been adapted into various sub-versions, each varying in width, depth, resolution, and other compound aspects. This study focused on two advanced iterations of the original EfficientNet baseline-B5 and B7-along with EfficientNet V2S, a custom variant designed to expedite training. From the 5-fold validation results, the EfficientNet B5 model displayed the lowest performance across all metrics, a consequence of its compact structure intended for versatile implementation across platforms. Specifically, its accuracy, precision, and recall were 88.12%, 78.64%, and 85.87%, respectively. In contrast, EfficientNet B7, characterized by greater depth and width than its counterparts and more sophisticated connections than VGG's sequential networks, outperformed all other individual models. It achieved accuracy, precision, and recall scores of 94.07%, 89.53%, and 90.32%, respectively, with the highest F1 score at 89.92%. EfficientNetV2S, incorporating custom adjustments to the EfficientNet framework, such as the removal of superfluous layers and parameter optimization, mirrored the performance of EfficientNetB7 closely. These results underscore the impact of network architecture adjustments on model efficiency and effectiveness.

Table 2. Class-wise performances of ensemble approaches

In this study, three ensemble approaches were implemented alongside individual models. The hard voting approach combined the class assignment results from each model, with the final decision based on the majority vote. A crucial aspect of this method is the use of an odd number of models to avoid a tie in votes, representing a significant trade-off in hard voting. On the other hand, the soft voting approach aggregated class assignment probabilities, with the final classification based on the class having the highest cumulative probability. The key challenge in this method is efficient memory utilization. Both hard and soft voting are categorized as decision-level fusion techniques. Additionally, a stacked generalization approach was employed, which involved a 1D CNN architecture for classification, processing the output from the soft voting method. The design of the 1D CNN is critical, as it requires careful consideration of various parameter trade-offs. The overarching goal of these ensemble methods is to improve classification performance by capitalizing on the strengths of each base model, thereby achieving a more accurate and reliable outcome.

The results outlined in Table 1 demonstrate that all three ensemble methods surpassed individual network performances across all evaluated metrics. Both hard and soft voting methods achieved a classification accuracy of 94.84%. Notably, the soft voting approach was more effective in class label determination, evidenced by its higher precision score of 91.10%. The stacked generalization approach outperformed the others in terms of classification accuracy, achieving the highest rate at 95.55%. Furthermore, this approach also attained the highest precision score, which contributed to the highest F1 score of 92.25%, indicating a superior balance of precision and recall compared to the other methods.

As the ensemble model approaches yielded superior results, detailed class-wise metrics for the three utilized ensemble methods are provided in Table 2. This allows for a more granular analysis of each method's performance across different classes.

The three ensemble approaches demonstrated comparable performance for the 'No DR' and 'Moderate' DR classes, with each approach achieving around 98% and 93% accuracy, respectively. Notably, the hard voting approach, which focuses on model outputs rather than class probabilities, offers computational advantages due to its lower complexity. In the classification of the 'Mild' and 'Proliferative' DR classes, the stacked generalization method slightly surpassed both hard and soft voting methods. However, its most notable superiority was evident in the classification of the 'Severe' DR class, where it outperformed the other two approaches. In terms of precision, which measures the accuracy of positive identifications, stacked generalization was approximately 5% more precise than both hard and soft voting. However, hard and soft voting recorded recall scores—indicating the proportion of actual positives correctly identified—nearly 2% higher than that of stacked generalization. Consequently, this led to the stacked generalization approach achieving F1 scores, a balance of precision and recall, that were 2% and 1.5% higher than those of the hard and soft voting methods, respectively.

This research focused on developing ensembles of deep learning models, employing hard voting, soft voting, and stacked generalization techniques, to classify the severity levels of DR. The APTOS 2019 dataset, encompassing four severity levels of the condition and a healthy state, was utilized for experimentation.

Due to the varied conditions under which retinal images were collected, a range of image preprocessing methods were applied. These included the removal of non-informative backgrounds, application of blurring, resizing, and data augmentation. Hyperparameter tuning was conducted to optimize the training of the base models, exploring different configurations of neurons, learning rates, dropout rates, activation functions, and optimization algorithms. Techniques such as early stopping and learning rate scheduling were also employed.

Models based on the VGG architecture, known for their simplicity but associated with high parameter counts, were contrasted with the more recent EfficientNet series, designed for enhanced performance through efficient network scaling. Two versions of VGG and three variations of EfficientNet were selected, aiming to investigate the impact of model complexity, parameter count, and computational efficiency on the classification of DR. The base models for the ensemble approaches included VGG16, VGG19, EfficientNetB5, EfficientNetB7, and EfficientNetV2S.

The results revealed that ensembles outperformed individual models in precision, recall, accuracy, and F1 scores. The stacked generalization method achieved the highest accuracy at 95.55%, while both hard voting and soft voting models attained an accuracy of 94.84%. Compared to prior state-of-the-art research, the proposed ensemble methods exhibited superior performance across all metrics. This signifies a significant contribution to the existing literature by providing more accurate results.

A limitation of this study is the increased computational demand associated with constructing ensemble models, necessitating the training of multiple models and the amalgamation of their outputs. Despite this, an application for predicting the severity of DR is planned, utilizing pre-trained networks for image testing to reduce processing times. This application could serve as a valuable diagnostic tool, offering improved accuracy over many existing methods. Future research will explore the utilization of diverse networks such as NasNet, DenseNet, ResNet, and lighter networks like AlexNet and ShuffleNet within the stacked generalization ensemble framework to potentially enhance performance and reduce computational costs. The development of this application will facilitate practical and swift analyses of fundus images obtained from eye screening tests, significantly improving the accuracy and reliability of clinical results.