A Hybrid Diabetic Retinopathy Neural Network Model for Early Diabetic Retinopathy Detection and Classification of Fundus Images

The escalating concern surrounding diabetic retinopathy (DR), a complex visual condition, underscores the urgent need for advanced and precise diagnostic techniques. If left unchecked, DR could potentially progress to severe vision impairment and eventually, blindness. As the global prevalence of diabetes continues to rise, there is a critical demand for automated diagnostic methods that can swiftly and accurately detect the onset and progression of DR. These automated techniques not only ensure early diagnosis but also hasten the initiation of therapeutic measures, playing a vital role in mitigating the adverse impact of DR on visual health.

Over the past two decades, diabetes prevalence has surged dramatically. According to the International Diabetes Federation's Atlas [1], nearly 500 million individuals of all ages globally are affected. This number is projected to rise to 700 million by 2045, underscoring the global reach of this issue. The IDF Diabetes Atlas predicts that by 2040, one-third of all individuals with diabetes will develop DR, the most common microvascular complication. By then, over 200 million people worldwide are expected to be affected by DR.

Diabetic retinopathy is characterized by retinal damage induced by diabetes. Two forms of DR are recognized: non-proliferative diabetic retinopathy (NPDR) and proliferative diabetic retinopathy (PDR). NPDR, typically causing mild vision loss, provokes macular edema due to the swelling and leaking of small retinal blood vessels. PDR, a more hazardous type of DR, instigates the growth of new blood vessels within the retina, which may rupture and induce severe vision loss and ocular pain. Risk factors for DR include types 1 and 2 diabetes, Hispanic or African ancestry, hypertension, hypercholesterolemia, pregnancy, and familial history of DR. Symptoms include blurred vision, spots and floaters, blind spots in the central retina, and diminished night vision [2].

Early DR symptoms encompass visual impairment and moving spots within the retina. If left untreated, advanced stages of the condition can culminate in total blindness. The disease is signaled by various forms of retinal lesions, including hard exudates, soft exudates, and hemorrhages/microaneurysms. Early detection is often missed in patients, which underscores the importance of regular eye exams, ideally twice a year, to facilitate early treatment and prevent further complications. DR is classified according to severity, with mild cases identified and treated in the early stages, while PDR is typically diagnosed after widespread progression [3].

The advent of machine learning (ML) algorithms and artificial intelligence (AI) technology offers healthcare professionals precise strategies for various diseases and clinical predictions. These include cholesterol, short-term glucose prediction, continuous glucose monitoring, chronic kidney disease, acute lung failure, and cancer. Machine learning, a branch of AI, has significantly simplified early DR detection. Another pervasive method is deep learning [4], which has proven particularly effective in predicting health issues. Deep learning techniques are increasingly utilized in diabetic retinopathy prediction due to their exceptional ability to manage complex data and identify crucial patterns. These techniques allow the automatic extraction of key features from intricate retinal images, obviating the need for manual feature engineering. This capability enables precise generalization and continuous adaptation, crucial for early detection and intervention in a degenerative condition like diabetic retinopathy. Furthermore, deep learning's capacity to detect minute changes, expedite the diagnostic process, and reduce the requirement for expert involvement enhances its effectiveness in improving patient outcomes and minimizing vision loss [5]. These methods routinely predict disease progression at an early stage aiming for optimal, precise results without human intervention, potentially saving millions of lives. Figure 1 illustrates the relationship between DL, ML, and AI.

1.png

Figure 1. Similarities among AI, ML, and DL

This research was driven by the desire to improve DR diagnosis from fundus images using DL. Manual diagnosis of these medical images is a labor-intensive process requiring highly trained experts who specialize in diagnosing ocular diseases based on imaging and other evidence. The requirement for such specialized medical professionals makes diagnosis potentially costly. Moreover, given human limitations, only a finite number of patients can be attended to at any given time. Human errors, as with many medical diagnostic techniques, also pose potential problems. Thus, it is often advised that individuals with significant medical concerns seek a second opinion. Considering these limitations, any effort towards automation would result in cost savings, reduced diagnostic errors, and increased throughput, allowing more patients to be attended to around the clock. This does not advocate for replacing experts with automated systems but rather suggests that the results from such a system could assist specialists in treating their patients more efficiently [6].

1.1 Global statistics in diabetic retinopathy

According to the American Diabetes Association (ADA), the utilization of Optical Coherence Tomography (OCT) in diagnosing intraretinal diseases has revolutionized the detection and treatment of diabetic retinopathy. The data derived from OCT allows for a more precise measurement of retinal fundus thickness and the identification of microvascular abnormalities [7]. This highlights the significant public health impact of diabetic retinopathy. Since 1980, the incidence of diabetes mellitus (DM) in men has surged by 110%, and in women, by 58%. These rates are anticipated to rise by 9% and 7.9% worldwide, respectively, in 2014. According to current estimates, an additional 422 million individuals will be diagnosed with diabetes in the coming years, a number that could increase to 629 million by 2045 [8]. The introduction of computer-assisted diagnostic systems could alleviate the workload of eye specialists in routine screenings. In addition, various algorithms, feature extraction methods, and classification techniques are being employed for the early detection of DR [9].

1.2 Objective and motivation of the work

The primary objective of this study is to design a method for automated detection of DR in retinal fundus images. If not diagnosed and treated promptly, DR, a severe complication of diabetes, can result in irreversible vision loss. Traditional DR diagnostic procedures involve a labor-intensive, subjective manual examination of retinal photographs by an ophthalmologist.

The main aim of this research is to devise a computer-aided design (CAD) system that can enable ophthalmologists to diagnose diabetic retinopathy (DR) swiftly and accurately. Deep learning algorithms can be trained to automatically extract relevant information from retinal images, thereby expediting and enhancing the diagnostic process.

Furthermore, in underserved regions where ophthalmologists may be scarce, the application of deep learning for DR detection could facilitate easier access to healthcare for individuals with diabetes. Widespread implementation of CAD systems for screening and early detection of DR is critical to prevent blindness due to diabetes. By improving early detection and treatment, research on DR diagnosis using deep learning techniques could significantly impact public health.

1.3 Organization of the study

The organization of this study is as follows: Section 2 provides a review of research on the application of deep learning in aiding patients with diabetic retinopathy. In Section 3, various deep learning methods and datasets related to DR classification are explored. Section 4 presents the results of performance comparisons using diverse metrics. Section 5 discusses the findings and potential future research directions.

The dataset that was used and the deep learning methods applied are outlined in this section.

3.1 Dataset description

The dataset used in this study is the diabetic retinopathy dataset, which was obtained from Kaggle. The dataset contains 2750 retinal images that have been categorized into five groups based on the severity of diabetic retinopathy: Healthy (Not DR) with 1000 images, Mild DR with 370 images, Moderate DR with 900 images, Proliferative DR with 290 images, and Severe DR with 190 images. These groups reflect the progression of the disease from healthy retinas to severe cases that involve abnormal growth of blood vessels and possible visual impairment. This dataset can be utilized to teach machine learning algorithms to recognize the severity of diabetic retinopathy by analyzing retinal images, thereby aiding in the early detection and management of the disease. Figure 2 displays a sample of the fundus images database.

2.png

Figure 2. Fundus image database

Database categories and the quantity of associated fundus photos are listed in Table 2.

Table 2. Categorization of the images in the database

Classes	Number of Images
Healthy (Not DR)	1000
Milder DR	370
Moderate DR	900
Proliferate DR	290
Severe DR	190

3.png

Figure 3. Data distribution graph

The above Figure 3 shows a distribution graph of the used diabetic retinopathy dataset. Images can be split into four or five distinct groups according to the degree of DR (shown by the x-axis) and the sheer volume of images (represented by the y-axis). The first category is labeled “Healthy,” which includes 1000 images of eyes that do not show any signs of diabetic retinopathy. The second category is labeled “Mild DR” which includes 370 images, and the third category is labeled “Moderate DR” includes 900 images. And the fourth category labeled “Proliferate DR” includes 290 images, and the fifth and final category labeled “severe DR” includes 190 images of eyes that show the most severe signs of diabetic retinopathy.

3.2 Data preprocessing

A series of data preprocessing and visualization processes for a classification problem involving an image dataset as described in the accompanying code snippets. The ‘cls_df’ list is first populated with class labels that have been taken from the image file paths. The count of photos for each class is then determined by utilizing the pandas data frame ‘data_df’, which is created from this list. A bar chart is used to display the resulting class distribution, with class labels on the x-axis and image counts on the y-axis. The code also shows a selection of the dataset’s first 15 photographs. Matplotlib is used to display the photos in a 3×5 grid after being resized to the typical size of 224×224 pixels and read using OpenCV. Overall, the code carries out tasks in preprocessing such as extracting class labels, generating class-wise image counts, and giving visual cues into the distribution of classes in the dataset and sample photos.

3.2.1 Data splitting

The train_test_split function is utilised to split the data, typically done in machine learning and data analysis. The purpose of this operation is to divide a given dataset into two distinct subsets: a training set (x_train and y_train) and a testing set (x_test and y_test). The training set consists of both image features (train_image_features) and their related labels (train_image_lbls). The test_size option is set to 0.2, which means that 20% of the data will be used for the testing set and 80% for the training set. This data partitioning has various important uses. Offering different datasets for training and testing, it facilitates the evaluation and validation of machine learning models in the first place. The model is trained using the training set, which enables it to recognize patterns and connections in the data. The performance of the model and its generalizability to fresh. unexplored data are evaluated using the testing set, which the model has not seen during training. This method, which divides the data into training and testing sets, helps prevent overfitting by offering an unbiased evaluation of model performance. Overfitting occurs when a model performs well on training data but badly on fresh data.

3.3 Hybrid DRNN (Diabetic Retinopathy Neural Network) model

This work proposed a novel model named is hybrid DRNN model. The hybrid model consisting of ResNet-152 and DenseNet-121 is a deep learning model that incorporates two distinct state-of-the-art architectures, ResNet & DenseNet, to enhance performance on image classification tasks.

To train very deep neural networks, the notion of residual learning was first developed by ResNet (short for Residual Network). ResNet's residual blocks route the input through a shortcut connection that facilitates the gradient's passage through the network, thereby preventing the problem of gradients that dissipate. In this way, ResNet may be used to create very deep neural networks, which are otherwise challenging to train.

The dense connectivity produced by DenseNet feeds forward from one layer to the next. This dense connectivity permits each layer to receive the output of all antecedent layers, allowing the network to capture a greater variety of features at various scales.

Improved performance on image classification tasks was achieved using a hybrid model that incorporates ResNet-152 and DenseNet-121. In particular, the model is built from many layers of ResNet-152 followed by several levels of DenseNet-121, with Each layer having a lot of layers of convolution and pooling layers. The ResNet-152 blocks take high-level data from the picture that comes in, and then more granular features are captured by the DenseNet-121 blocks. Finally, at the network's end, a classification layer is added to obtain the predicted class label as shown in Figure 4, which depicts the architecture of the DRNN model.

4.png

Figure 4. Architecture of the DRNN model

Table 3. Model summary

Layer (type)	Output Shape	Param #	Connected to
input_layer (InputLayer)	[(None, 196, 196, 3)]	0	[]
-resnet152 (Functional)	(None, None, None, 2048)	58370944	['input_layer[0][0]']
densenet121 (Functional)	(None, None, None, 1024)	7037504	['input_layer[0][0]']
flatten (Flatten)	(None, 100352	0	['resnet152[0][0]']
flatten_1 (Flatten)	(None, 36864)	0	['densenet121[0][0]']
concatenate (Concatenate)	(None, 137216)	0	['flatten[0][0]', 'flatten_1[0][0]']
flatten_2 (Flatten)	(None, 137216)	0	['concatenate[0][0]']
dense (Dense)	(None, 1024)	140510208	['flatten_2[0][0]']
dense_1 (Dense)	(None, 1024)	1049600	['dense[0][0]']
dense_2 (Dense)	(None, 2)	2050	['dense_1[0][0]']
Total params		206,970,306
Trainable params		206,735,234
Non-trainable params		235,072

The state-of-the-art on several image classification benchmarks, including the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset, have been demonstrated with the hybrid ResNet-152/DenseNet-121 model. Moreover, the use of hybrid ResNet/DenseNet architectures has been explored in other medical imaging applications, such as pulmonary nodule detection and breast cancer histopathology image classification, demonstrating the versatility of this approach.

The following Table 3 shows a summary of the proposed DRNN model. In this model of summary, there are 206,970,306 total parameters, 206,735,234 trainable parameters, and 235,072 non-trainable parameters given.

3.4 Classification methods

Diabetic retinopathy fundus image classification using DL has been the subject of a large body of research. In this paper, we have compared some deep learning approaches including CNN [25], RNN [26], deep belief network, and GoogleNet with our novel approach. But, here, we have discussed only two methods i.e., CNN, and RNN.

3.4.1 Convolutional Neural Network or (CNN)

Deep learning is a popular research topic. It's effective across a wide range of data formats. The use of a DL model allows for the creation of a CNN-based image categorization network. Figure 5 depicts the architecture of CNN. In a computer, an image is created by connecting pixels. A collection of pixels may depict the image's edge, shadow, or another pattern. By using convolution, it is feasible to locate these patterns. Each pixel in a picture is represented by a digital matrix that stores and manipulates the image. The matrix of image pixels must be multiplied by a "filter" matrix in order to create spot patterns. The filter size is variable, and the manner in which multiplication is performed by entirely dependent on the filter size. Convolution may be done by beginning with the first pixel in the image's pixel matrix and extracting a portion of it according to the size of the filter. Convolution is then applied to the following image pixel, and this procedure is done as many as necessary until all of the matrix's image pixels have been processed [27].

Convolving is the procedure of combining multiple variables into one. In order to pass on useful information to the next layer, convolution is utilized to generate a feature map. The next sort of layer within the CNN framework is the pooling layer. This layer decreased the output or feature map in order to prevent overfitting. As the most advanced layer, the FCL is the highest classification. This layer "flattens" the result of the layers which came before it so that it can be utilized as an input in the next phase. Batch normalizations can be used to standardize feature maps collected at each stage. Dropouts are sometimes used to hasten computational processes [28].

5.png

Figure 5. Basic architecture of the CNN

The edges, a DR categorization characteristic, could be taught to the first layer, whereas the layer with the greatest depth learned to interpret hard exudate. Complex functional mapping that occurs between input and output variables is generated by the network's convolutional blocks, which are activated at the top layer.

Each layer of convolution is followed by a batch of normalization of the output variables. Each block receives an additional batch normalization in sequence as the number of unique feature maps increases.

Layers in CNN:

Based on the layer designs of the network shown in Figure 6, an overview of the primary layers used in a CNN.

• Input Layer: After being processed, the CNN algorithm's input picture should contain image data at the input layer. We've established that a three-by-three matrix adequately reflects visual information. We have to rearrange it so that it's all in one neat column. Let's say we have a picture that is 28 by 28 pixels, which is 784. We need to turn it into 784 by 1 before we can feed it into the input. Input has dimension (784, m), where m represents the number of training cases.
• Convolution Layer: The inputs to the algorithm are convolved in two dimensions. Dot products of inputs and weights integrated across channels. Multiple receptive fields have used the same filter weights. The used filter has as many levels as the number of channels in the volume of the input, and the volume of the output is as deep as the number of selections.
• Pooling Layer: To generate activation maps, convolutional layers are used. The activation maps are down-sampled nonlinearly by the pooling layer. Since pooling is destructive (it discards data), the current trend is to employ less extensive filter capacities and abandon pooling altogether.
• Fully connected layer: This layer can be thought of as the last step in learning because it links the visual traits that were learned to the outputs that were wanted. Most of the time, they can do things like classifying and encoding. Most of the time, a vector is the output, which is then sent through Softmax to show how confidently the classification was made. Images of brain tissue and nerves are used to make links between layers. It connects neurons in a particular layer to those in another. It is used to train images to be classified into various categories.
• Softmax or Logistic Layer: The ultimate layer of CNN is the Logistic or Softmax layer. It's the final thing in the FC layer. Logical methods are used for two-classification problems, while softmax is used for more than two. In the output of the layer, the label is stored as a single image of the brain.

6.png

Figure 6. Convolutional Neural Network (CNN) layers

3.4.2 Recurrent Neural Network (RNN)

7.png

Figure 7. Recurrent neural network architecture

A type of artificial neural network known as recurrent neural network (RNN) is designed to handle a sequence of data, represented as x(t)=x(1), x,..., x, where t is the step number. RNNs are particularly useful for processing speech and language inputs that have a specific order, as illustrated in Figure 7. If we are interested in predicting the next word in an NLP problem, it is necessary to know what words have come before it. Recurrent neural networks, also known as RNNs, are so-called because they do the same thing for each item in a series, and the result depends on what they did before. RNNs can also be thought of as having a "memory" that keeps track of all that has been computed so far [29].

A notation for an RNN may be seen on the left side of the image above, while the unrolling of an RNN into its full network can be seen on the right. With "unroll," we mean to delete the series's network from our DVRs. The network will be unfurled into a three-layer network of neurons, with a single layer corresponding to each of the words, if the pattern we are interested in is a three-word sentence.

• Input: A network is given the information x(t) at each step in time t. It's possible, for instance, that x1 stands for a single-shot vector that exactly matches a single term in an expression.
• Concealed state: The system's concealed state, represented by h(t), is called the "memory at time t. H(t) is created by fusing the current input with the hidden state from the preceding time step. In other words, h(t) = Ux(t) + W(t-1)h(t). Many people believe that f is an unpredictable transformation, like tanh or ReLU.
• Weights: The Recurrent Neural Network has weight matrices U which decides the amount of strength of the input-to-hidden relationships, W which determines the strength of the concealed-to-hidden recurrent relationships, and V that determines the strength of the hidden-to-output relationships. U, V, and W are all constants that persist throughout time.
• Output: o(t) represents the network's output at each time step. I only drew an arrow after o(t) to indicate that it is typically unpredictable, particularly as the network's layer count is getting smaller.

Working of RNN:

With the assistance of the following example, you can see how an RNN works as shown in Figure 8:

Example:

Complex networks may have more than one input layer, multiple intermediate layers, and a single output layer. Thus, biases and weights for Hidden Layer 1 could be (w1, b1), Hidden Layer 2 could be (w2, b2), and Hidden Layer 3 could be (w3, b3), similar to other types of neural networks. Each of these levels is self-contained, meaning it does not rely on the results of other layers.

8.png

Figure 8. Weight distribution in RNN

Currently, the RNN will act as follows depicted in Figure 9:

• By standardizing the weights and biases across all layers, RNN transforms autonomous activations into dependent ones. Each result can be used as an input for the following hidden layer, simplifying the process of adding fresh variables and committing them to memory.
• Therefore, the three hidden layers might be merged into one continuous layer with common biases and weights.

9.png

Figure 9. Looping mechanism in RNN

A Method for Evaluating the present situation:

$h_t=\mathrm{f}\left\{h_{t-1}+X_t\right)$             (1)

where, h_t present state, h_t-1 preceding state, x_t input state.

Here is the equation for the activated function (tanh):

$h_t=\tanh \left(W h h h_{t-1}+W x h X_t\right)$               (2)

Weight at the recurrent nerve (Whh) and weight at the input (Wxh) to a nerve cell.

The technique for calculating output is

$\mathrm{yt}=$ Whyht                   (3)

where, Y_t Outcome, W_hy Output weighting layer.

RNN is used for training

• The network receives a single step of the info.
• Then, use the present inputs and their previous state to figure out what their present state is.
• For the following time step, ht is changed to h_t-1.
• One can go back in time as many times as needed to solve a problem and then combine the data including all the states that came before.
• When every stage of the calculation has been completed, the outcome is calculated based on the final state.
• The mistake is then found by comparing the outcome with the actual outcome, or the result that was supposed to occur.
• The error is subsequently transmitted backward by means of the network in order to modify the amount of weight. This develops the network (RNN).

3.5 Proposed system flowchart

This section detailed a proposed flowchart for this system's implementation.

3.6 Pseudocode of the proposed model

Figure 10, shows the flow of the proposed DRNN model.

Start

1. Get the Kaggle dataset for diabetic retinopathy.
2. Import the Diabetic Retinopathy Dataset.
3. Preprocess the dataset using image resizing, and normalization.
4. Divide the data in train and test in 80-20%.
5. A hybrid DRNN model with some hyperparameters is applied for training.
6. Test the trained model
7. Measure the DRNN model's efficacy with respect to a variety of criteria, including accuracy, recall, sensitivity, and f1-score.,
8. Detect and classify diabetic retinopathy.

Stop

10.png

Figure 10. Working flowchart of the proposed methodology

Patients with DR who receive therapy and diagnosis later are more inclined to lose their vision. Early warning signs must be identified before the disease's severity can be established, at which point the appropriate course of treatment can be chosen. The project focus on using an algorithm based on deep learning to classify DR fundus photos according to severity. Deep learning is thought to be a useful method for predicting and categorizing diseases on a technical level. A more typical eye problem is diabetic retinopathy (DR), which is brought on by elevated blood sugar. Over half of people under 70 worldwide have diabetes. Despite early diagnosis and treatment, DR patients typically lose their vision. The severity of the issue must be determined after identifying the warning signs in order to select the finest course of action. In this research, the severity of DR fundus images is classified using deep learning. This study presents a new method that uses DRNNs. which is built using two different deep learning models i.e., ResNet152, and DenseNet-121 for categorizing fundus DR images. In addition, this paper also reviewed some deep-learning-based approaches such as CNNs, RNNs, DBNs, and GoogleNet for use with fundus pictures in the diagnosis of diabetic retinopathy. This paper uses the diabetic retinopathy dataset to test the performance metrics for the model, such as sensitivity, recall, f1-score, accuracy, specificity, and precision. The proposed DRNN model obtained the highest 96.91% of accuracy, whereas the other models have 73.81% accuracy for CNN, 95.48% accuracy for RNN, 92.12% accuracy for DBN, and 82.18% accuracy for GoogleNet, respectively. It is clearly shown that the proposed DRNN model outperformed the other models.

This project effectively finds diabetes by using deep learning on pictures of the fundus. It could be used in the future as one way to find diabetes. In the future, work will be done to improve the classification of DR by using the most recent deep learning methods rather than human grading to find cases of diabetic retinopathy.