Deep Learning-Based Segmentation and Classification of COVID-19 Infection Severity Levels from CT Scans

A new coronavirus known as COVID-19 has been recognized as of December 2019 in Wuhan, China. It was quickly identified as an infectious disorder with several confirmed cases around the nation. It belongs to the Severe Acute Respiratory Syndrome Corona Virus 2 (SARS-CoV-2) family, produced by particular proteins on the surface of these pathogens. The COVID-19 infection is caused by the coronavirus SARS-CoV-2, which is the 7th coronavirus harmful to humans. The COVID-19 epidemic is placed the country in a state of inhibition, intending to limit contamination and minimize deaths [1]. Until now, 80% of diseases have been minor, 15% have been chronic and require respiratory support, and 5% have been serious and require ventilatory help. Temperature, lethargy, persistent cough, and digestive problems are the most common SARS-CoV-2 symptoms. Severe infections, such as pneumonia and acute respiratory distress syndrome, affect one out of every six patients. People who are believed to cause COVID-19 disease ask to be quickly identified as infected or not, to be perfectly secluded, to have the right diagnosis, and to notify the people they have interacted with [2]. Reverse-Transcriptase Polymerase Chain Reaction (RT-PCR) assays [3] are commonly required in conjunction imaging to validate a treatment that is time and cost-intensive. Furthermore, the RT-PCR results in a higher prevalence of false negatives; thus, an automated effective tool is required to achieve earlier treatment and aid in the reduction of the infection's incidence.

A wide range of pulmonary abnormalities, analogous to SARS, have been identified on X-ray and CT scans to identify individuals with COVID-19 [4]. This imaging technique aids clinicians in detecting COVID-19's effects on various tissues at various phases of infection. As respiratory indications are one of the key symptoms of COVID-19, this is applied to the heart and lungs. This procedure needs a specified period of time to properly categorize pneumonia, COVID-19, and healthy people. So, different deep learning frameworks, particularly Deep Convolutional Neural Networks (DCNN), have been suggested to automatically analyze the COVID-19 epidemic.

By using the medicinal datasets comprising CT scans, such frameworks have revealed outcomes in the identification of COVID-19 [5, 6]. Also, CT images are more reliable compared to X-ray scans, so the primary treatment of COVID-19 is becoming simpler. But, many COVID-19 datasets are not available for researchers or hard to find. The severe signs of COVID-19 will lead to a misclassification of CT scans. This problem is resolved by analyzing the training images which threaten nearly similar pixels as COVID-19. For this reason, it is vital to separate COVID-19 and non-COVID-19 patients.

The recognition and treatment of COVID-19 with the help of common image analytics and machine learning frameworks is a tedious and time-consuming procedure owing to the practical challenges in image acquisition and performance of analytics tools. To combat this difficulty, 2D frameworks were designed recently, as well as the performance of 3D frameworks on a partition of the CT images was improved. Besides, a stack of 2D partition frameworks known as 2.5D frameworks was fused to enhance the 3D partition. Typical 3D partition frameworks like 3D U-Net request a massive quantity of images for training, and the learning speed was very slow for healthcare systems, especially during the COVID-19 epidemic. From this viewpoint, a quick, precise and machine-agnostic partition and quantification framework [7] was designed. First, a pre-processing step was employed to cast the pulmonary CT scan into a machine-agnostic typical embedding space. Then, a highly accurate partition framework like 2.5D U-Net was employed on the typical embedding space. As well, an enhanced analysis method was used for training that views the dynamic alteration of diseased ROIs for COVID-19. On the other hand, the time and storage use for learning was high. So, an E2.5D U-Net-based deep transfer learning framework [8] was developed which integrates weight transfer block after the convolutional block in the standard U-Net. This weight transfer block was used to adjust weight values to the feature maps and attain better partition precision. In this framework, different pre-trained deep learning structures (ResNet18, DenseNet121, InceptionV3, Xception and MobileNetV2) were built by dynamically fine-tuning their weights related to the input samples and obtain the optimal weights for E2.5D U-Net. Based on this E2.5D U-Net framework, the COVID-19 infected ROIs were partitioned from the CT images to recognize and diagnose the patients. But it needs to classify the severity of infection to diagnose patients at an earlier stage.

Therefore, this paper introduces a feature extraction with a classification unit for training deep features associated with disease severity. First, the E2.5D U-Net-based partition is applied to segment the COVID-19 infected ROIs from the lung CT images. After that, each segment is fed to DenseNet201 for extracting the deep features. These extracted features are used for training the linear Support Vector Machine (SVM), cubic SVM, K-Nearest Neighbor (KNN), Linear Discriminant Analysis (LDA) and AdaBoost decision tree for categorizing the infection severity levels into early, progressive, and severe. This helps us recognize patients with severe infections and diagnose them at a prior stage. However, it needs additional features related to the diseases to increase its accuracy. So, it applies a multi-modeling classification framework to extract the mixture of deep and handcrafted features from CT scans to distinguish COVID-19, pneumonia and healthy patients. In this framework, different handcrafted features are extracted for each CT segment and concatenated with the deep features to obtain the single feature vector. Then, the obtained feature vector is learned by the multi-modeling classifier called modified Inception to classify the severity level of particular COVID-19 infected patients. Thus, it can effectively increase the accuracy of recognizing and diagnosing the patients who have COVID-19 infections. The remaining sections of this article include: Section 2 studies the recent work related to the partition and categorization of COVID-19 patients from CT samples. Section 3 explains the methodology of proposed COVID-19 classification frameworks and Section 4 displays their efficiencies. Section 5 concludes the entire work and suggests the future scope.

In this section, the proposed feature extraction and categorization of COVID-19 infection severity are described briefly. Figure 1 illustrates the schematic representation of proposed feature extraction and multi-modeling classification of COVID-19 severity levels from CT scans. First, the consecutive CT slices are processed to generate 3D volume pixels which characterize the image structure in 3D. This representation makes CT scan in 3 different planes, i.e. xy, yz and xz planes.  Then, an E2.5D U-Net model (as illustrated in Figure 2) is performed to segment the ROIs from the lung CT scans [8]. Then, the DenseNet201 deep model is applied to extract the deep features from each segment. Also, the handcrafted features are extracted and concatenated with the extracted deep features to get a single feature vector. This feature vector is fed to the linear SVM, cubic SVM, KNN, LDA, AdaBoost and modified inception classifiers independently for categorizing the severity levels of COVID-19 patients effectively.

1.png

Figure 1. Schematic representation of feature extraction and multi-modeling classification framework

2.png

Figure 2. Structure of E2.5D U-Net model for CT scan segmentation

3.1 Deep feature extraction with different machine learning-based classification

In this phase, the pre-trained deep model called DenseNet201 is applied for feature extraction from the chest CT images. Table 1 shows the feature extraction methods that are carried out by passing the segmented CT image through the DenseNet201 structure's consecutive levels. The DenseNet-201 is a 201-layer CNN. ImageNet database is used to load a pre-learned variation of the structure which has been learned on over a million photos. This model is initially learned on ImageNet before being learned on lung CT pictures. It consists of the convolution layer (112×112) at the beginning and 4 dense blocks followed by the FC-1000 and SoftMax layers. The outcome of the single convolutional layer is defined by:

$h_{i, j}=f\left(\sum_{k}^{P} \sum_{l}^{Q} w_{k, l} x_{i+k, j+l}+b_{k, l}\right)$            (1)

In Eq. (1), h is the neuron output, x is the input, w is the weight, and b denotes the bias. Also, P, Q denote the range of weight variables, k, l denote variable indices and i, j denote input indices. Every convolutional unit performs ReLU, regularization, max-pooling functions and the ReLU is applied as an activation factor in the DenseNet201. During the training phase, this pre-trained DenseNet201 is considered to mine the related deep characteristics from COVID-19 CT scans.

The segmented entry scan of size 224×224×3 is passed to the initial convolutional unit that gives the scan of dimension 112×112×64 via convolving the entry scan using 64 kernels. The last FC unit contains a size of 1000. In this framework, FC-1000 unit characteristics are mined and passed to the linear SVM [18], cubic SVM, KNN [19], LDA and AdaBoost [20]. These classifiers are trained independently to categorize the severity levels of COVID-19 cases into three major stages: early, progressive and severe. But it needs additional features and multi-modeling classification for enhancing the precision of classifying the infection severity levels.

Table 1. DenseNet201 Structure

3.2 Multi-modeling classification for COVID-19 infection severity prediction

In this framework, handcrafted features are also extracted from the CT images such as Local Binary Patterns (LBPs), Gray-Level Co-occurrence Matrix (GLCM), Local Ternary Patterns (LTPs), and Local Derivative Patterns (LDPs). These are the texture features in the lung CT images. The features extracted by the GLCM include contrast, correlation, divergence, power, entropy, homogeneity, average, variation and standard variance. The LTP analyzes the local gray-scale variance and the LDP determines the texture of an image in different directions of a pixel.

Once all the handcrafted features are extracted, these are concatenated with the deep features directly to get the final unified feature vector. The unified feature vector is defined as:

$fused feature vector_{1 \times q}=\sum_{i=1}^{5}\left\{\begin{array}{c}\text { f DenseNet } 201_{1 \times m}, f L B P_{1 \times p}, f G L C M_{1 \times g}, \\ f L T P_{1 \times l}, f L D P_{1 \times d}\end{array}\right\}$            (2)

In Eq. (2), f defines the fused vector and

f DenseNet $201_{1 \times m}=\left\{\begin{array}{c}\text { DenseNet } 201_{1 \times 1}, \text { DenseNet } 201_{1 \times 2}, \\ \ldots, \text { DenseNet } 201_{1 \times n}\end{array}\right\}$            (3)

$f L B P_{1 \times p}=\left\{L B P_{1 \times 1}, L B P_{1 \times 2}, \ldots, L B P_{1 \times n}\right\}$             (4)

$f G L C M_{1 \times g}=\left\{G L C M_{1 \times 1}, G L C M_{1 \times 2}, \ldots, G L C M_{1 \times n}\right\}$               (5)

$f L T P_{1 \times l}=\left\{L T P_{1 \times 1}, L T P_{1 \times 2}, \ldots, L T P_{1 \times n}\right\}$                (6)

$f L D P_{1 \times l}=\left\{L D P_{1 \times 1}, L D P_{1 \times 2}, \ldots, L D P_{1 \times n}\right\}$                   (7)

Moreover, a modified Inception classifier is used for categorizing the infection severity levels using the fused severity feature vector. The modified Inception structure is presented in Table 2. The variation between Inception and modified Inception in the classifier lies in the final fully connected layer. The obtained severity feature vector is given as input to this network and learned with the help of one-hot encoding of output classes. The one-hot encoding is used to label chest CT scans to indicate the positive COVID-19 case or not (i.e., normal). Here, each encoded label comprises 2-elements array with one of the elements being “hot” (i.e., 1) and “not” (i.e., 0). Finally, the SoftMax is executed to categorize the severity levels into early, progressive, severe and healthy stages. Based on the infection severity levels, the physicians can successfully diagnose COVID-19 patients in prior stages of infection.

Table 2. Structure of modified inception classifier

For classifying the severity level of COVID-19, the SoftMax loss function is defined as:

$L(x)=-\frac{1}{N} \sum_{i=1}^{N} Y_{i y_{i}} \log p_{i y_{i}}$              (8)

In Eq. (8), $N$ is the number of classes, $Y_{i y_{i}}$ is the one-hot encoding of severity level class and $p_{i y_{i}}$ is the $y_{i}$-th element of predicted probability vector for $x_{i}$.

Algorithm:

Input: Training lung CT images

Output: Classified infection severity levels: early, progressive or severe

Begin

Get the training lung CT images and pre-process them using resizing and regularization;

Augment the number of training images;

Fine-tune the deep learner structures and execute the E2.5D U-Net;

Obtain the segmented ROIs of COVID-19 infection;

Extract the deep from each segment using DenseNet201;

Extract the handcrafted features from each CT sample;

Fuse deep and handcrafted features to get a unified feature vector;

Train the SVM, KNN, LDA, AdaBoost and modified Inception classifiers independently using the fused feature vector;

Categorize the COVID-19 severity levels and diagnose the patients at an earlier stage;

End

Hence, this framework can simultaneously categorize multiple COVID-19 CT scans based on their severity levels for early diagnosis for early diagnosis.