Explainable Artificial Intelligence (XAI): Classification of Medical Thermal Images of Neonates Using Class Activation Maps

Measuring the neonatal temperature distribution by using both thermography and thermometer, Clark and Stothers showed that the results of both techniques were similar [8]. Clark and Stothers’s study proved that thermal monitoring of neonates could be realized for the first time.

Necrotizing enterocolitis (NEC) disease which causes a crucial neonatal health problem that affects low-birth weight infants in NICU [9] has been studied by Nur [10]. According to her findings, unhealthy neonates' thermal asymmetries are higher than healthy neonates'.

Abbas et al. [11] proposed a non-contact monitoring system to measure respiratory of newborn babies. Abbas et al. [12] analyzed the heat flux during different medical scenarios such as an open radiant warmer and convective incubator to compensate the external effects while taking the thermograms from NICU.

The abdominal and foot regions were measured for extremely premature neonates by Knobel-Dail et al. [13] using thermography and thermistors. They pointed out the regional variations in thermal condition of neonates with this study.

We have been also working on this topic over the past three years and we showed how neonatal thermograms can be classified as healthy and unhealthy by using deep learning methods such as CNNs [7].

These studies about neonatal monitoring with thermography and artificial intelligence shows the evidences of non-contact and harmless thermal monitoring system for neonates. However to give an important suggestions to doctors that are interested in diseases about neonates, we have to explain the decisions of our intelligent systems also known as models that can learn.

When we come to the explanations of the models, we encounter three main explanation methods as numerical, rule-based, and visual [14].

The numerical methods measure the contributions of input features starting from zero to all or all to zero by using quantitative methods such as Information Gain (IG) [15]. On the other hand, the rule-based methods use the IG and extract various rules from inputs to outputs, and the best known rule-based method is Decision Trees [16]. While these methods give information about classification process, they do not efficiently work in big models such as CNNs because of their limitations in view of time and computational costs.

The visual expressions are typically used in convolution-based methods by creating activation maps (also known as salient masks or heat-maps). Using the activation maps, the contributions of each pixels can be represented on the input images. One of the visual expression methods is Class Activation Maps (CAMs) [17]. With the developing of CAMs, researchers have been able to uncover the class-related pixels contributions to the classification process [18, 19].

The three methods explained above can be combined to increase performance but it should not be forgotten that there is always a trade-off between the performance of a model and its explain-ability.

To explain how classes are detected for the neonates, a CAM structure is needed. For building a CAM structure we need CNNs model and transfer learning method. While the CNN learns the important features of the images and classifies them into classes desired, transfer learning reduces the needed time to train a CNN model effectively and improves the model performance. The models used in transfer learning are described as pre-trained models. By using a pre-trained model users do not train a CNN model from scratch, they only realize fine-tuning operations on pre-trained models which were trained with millions of different images by big companies to train model so that decrease the training time.

As seen in Figure 4, first of all the important features such as edge, corner and texture are extracted by a pre-trained model and second of all the last layer of the pre-trained model is changed according to outputs desired to produce class-related activations. Then the model is trained and the class-related activations are obtained.

4.png

Figure 4. The pipeline of the activation maps obtaining

4.1 Convolutional neural networks (CNNs)

To classify an image, it is necessary that important features (edges, corners and textures) are obtained. The CNN is one of the most used deep learning models due to its structure that can effectively learns deep features of images and classifies them into classes [20].

CNNs consists of two main sides as convolutional and neural which are dedicated to feature engineering [21, 22] and classification processes, respectively. A CNN model is displayed in Figure 5.

The convolutional side learns regional textures that are defined at the beginning like 3x3 or 5x5 sizes and the neural side learns global textures of its inputs. The meaningful features are extracted and classified according to classes desired.

The convolutional side (like 1., 3., 5., 7., and 9. parts of Figure 5) can consist of dozens of different convolutional layers. While first convolutional layers learn low-level (small regional) features such as corners and edges, last layers typically learn high-level (big regional) features such as textures [23].

Computational capability is an important key that is needed when a model is trained. In order to reduce dimensions and avoid high computation power, the pooling operation (2., 4., 6., 8., and 10. parts of Figure 5) is used after the convolutional layers.

4.2 Pre-trained networks (transfer learning)

Transfer learning is the process of taking a CNN model that has been trained with large data sets and processors with high capacities.

It is difficult to train a CNN from the scratch because of needing both high computational capabilities and thousands of labeled images. Pre-trained models come up with a solution to overcome those lack of needs [24-27]. A pre-trained model can be used for both feature extraction and fine-tuning issues.

5.png

Figure 5. A CNNs model with pre-trained VGG16 architecture

Since lower-level features such as edges are typically the same in each classification process, instead of re-learning these features for every training, the weights of the first convolution layers of the pre-trained models can be directly used. Thus, the CNNs models learn convolution weights that find textures like tissues.

We used VGG model which can be seen in Figure 5 with 16 layers also known as VGG16 that has 5 pooling layers, 13 convolutional layers, and 3 neural layers [27] (p.s. End-to-end 21,137,986 features, 224x224x3 input size and 7x7x512 the last convolutional layer size).

4.3 Class activation maps (CAMs)

6.png

Figure 6. The activations that are needed to use in visualization

There are visualization techniques in CNNs such as layers' outputs visualizations and filter visualizations. But these techniques are not enough to assess the decisions. CAMs are used to determine which parts of image are learned by CNNs.

The layers' outputs give information about how each layers affect their connected layers, and which types of convolutions are learned by filter visualization. Since we need to explain how CNNs work, advanced techniques such as CAMs are necessary to visualize important regions (i.e. class-related activations) of images.

We are trying to find activations seen in Figure 6 because those activations are related to diseases and if we apply them to images we are going to find the essential regions for classification. It is shown that how a CAM is built in Figure 7 and the detailed representation of the CAM side is given in Figure 8.

If Figure 5 and Figure 7 are compared, it is seen that some changes important are needed. To change the VGG16 model into CAMs model, we need to add a new convolution layer that its size is equal to our number of desired outputs as the last convolution.

As can be seen in 9. part of Figure 5, there are three convolutional layers (#512, #512, and #512). Since our output has two classes as healthy/unhealthy, the last convolution's layer size has been changed from #512 to #2 (10. part of Figure 7 and Figure 8).

After obtaining the last convolution with the same sizes of output desired, Global Average Pooling (GAP) (Eq. (1)) [28] is applied to each last layer (11. part of Figure 7 and Figure 8).

$G A P_{n}=\sum_{x, y} f_{n}(x, y)$      (1)

where, fn(x, y) is width x height sized n th convolutional layer. With the GAP operation 1x1 sized convolutional layers are obtained, and then a neural layer is added as the last layer to the model (12. part of Figure 7 and Figure 8).

7.png

Figure 7. A CAM model created by pre-trained VGG16 model

8.png

Figure 8. The detailed representation of CAMs and global average pooling

When it comes to the activation maps the following equation is calculated:

$I_{c}=\sum_{n} w_{n}^{c} G A P_{n}$      (2)

where, I_c is the activation map calculated for class c, $ w_{n}^{c}$represents the weight of class c for n th layer (i.e. the importance of GAP_n for the class c). Finally Softmax function [29] is calculated with the following equation:

$S F T_{c}=\frac{e^{I_{C}}}{\sum_{c} e^{I_{c}}}$      (3)

The Softmax function gives probabilities that sum of them is equal to 1 for example 0.3 healthy and 0.7 unhealthy.

4.4 Evaluation of the results

Typically three metrics are used to evaluate the results. The ratio of correctly classified data to all data is described as accuracy (Eq. (4)).

$\operatorname{accuracy}=\frac{T P+T N}{T P+T N+F P+F N}$      (4)

where, TP is number of the real patient data labeled as patient, FP is number of healthy data labeled as patient, TN is number of healthy data labeled as healthy, and FN is number of the unhealthy data as healthy. The confusion matrix is obtained by using those four expressions as shown at Table 4.

Table 4. The confusion matrix

By using a confusion matrix, evaluations become more understandable and metrics such as specificity and sensitivity can be directly calculated.

The ratio of the TN to all healthy data is described as specificity (Eq. (5)) and the ratio of the TP to all unhealthy data is described as sensitivity (Eq. (6)).

specificity $=\frac{T N}{T N+F P}$      (5)

sensitivity $=\frac{T P}{T P+F N}$      (6)

Monitoring systems in the medicine are extremely crucial for early diagnosis of diseases and thermal information gives us ability to assess the health status of patients. The thermography method provides temperature values of the related skins, and neonatal monitoring could be realized by using both traditional and advanced ways.

Image classification problems have been solved more efficiently (over 90% accuracy) with the development of deep learning models such as CNNs recently. However, their decision process still keeps its secret and lots of researcher tries to find the answer how CNNs works.

So far CNNs have been used to classify the neonatal thermograms as healthy and unhealthy, but the question of how CNNs have decided could not be known. With the development of CAMs, we’ve been able to see important activations in convolutional layers.

With the realizing study, we both classify the neonates as healthy and unhealthy and show how CNNs makes a decision by using CAMs. To avoid training CNNs from scratch, VGG16 has been used as a pre-trained model; therefore, both time and computational costs decreased.

The developed model classified the thermograms of neonates with 80.701% (sensitivity) and 96.842% (specificity). Due to CAMs requirements such as global average pooling layer, most of the visual information are left at the last layer of the pre-trained model. This shows the model better learns the healthy neonates' thermograms than unhealthy neonates'. Normally we showed in our previous studies that a CNN model classifies both healthy and unhealthy neonatal thermograms with the same performance as about 95% accuracy.

Our main findings about explain-ability show that the CNNs are looking for neck, armpit, and abdomen regions' thermal distribution. Moreover, the class-related activations of the healthy babies are on the armpit and abdomen regions whereas activations of the unhealthy babies are on the neck and abdomen regions.

To conclude, our research results show that:

• Because of the CAMs restricted structure, VGG’s classification ability decreases but in our study we successfully trained the VGG model and achieved sensitivity-specificity as 80.701% and 96.842% respectively.
• The decision process of the health status detection was not known before this study, by highlighting the main areas that effect the outputs we show that how VGG16 decides on neonatal thermograms.

These results have vital importance for both us and medical specialists because the results show that the CNNs learn the specific regions of neonates being healthy and unhealthy.

In future studies we will be focusing on disease-specific activations and giving their importance for every disease neonates have.