Convolutional Neural Networks-Based Approach to Detect Neonatal Respiratory System Anomalies with Limited Thermal Image

Convolutional Neural Networks-Based Approach to Detect Neonatal Respiratory System Anomalies with Limited Thermal Image

Saim Ervural* Murat Ceylan

Department of Electrical-Electronics Engineering, Faculty of Engineering, KTO Karatay University, Konya 42020, Turkey

Department of Electrical and Electronics Engineering, Faculty of Engineering and Natural Sciences, Konya Technical University, Konya 42130, Turkey

Corresponding Author Email: 
saim.ervural@karatay.edu.tr
Page: 
437-442
|
DOI: 
https://doi.org/10.18280/ts.380222
Received: 
13 February 2021
|
Revised: 
17 March 2021
|
Accepted: 
28 March 2021
|
Available online: 
30 April 2021
| Citation

© 2021 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).

OPEN ACCESS

Abstract: 

Respiratory system diseases in neonates are thought-about major causes of neonatal morbidity and mortality, particularly in developing countries Early diagnosis and management of these diseases is very important. Thermal imaging stands out as a harmless non-ionizing method, and monitoring of temperature changes or thermal symmetry is used as a diagnostic tool in medicine. This study aims to detect respiratory abnormalities of neonates by artificial intelligence using limited thermal image. Convolutional neural network (CNN) models, although a powerful classification tool, require a balanced and large amount of data. The conditions that require the attention of infants in neonatal intensive care units make medical imaging difficult. It may not always be possible to have much data in the neonatal thermal image database, as in some real-world problems. To overcome this, an effective deep learning model and various data enhancement techniques were used and their effects on the classification results were observed. Neonates with respiratory abnormalities were evaluated in one class, with cardiovascular diseases and abdominal abnormalities were evaluated in the other class. As a result, when the number of images is increased by 4 times with data augmentation, it was determined that the classification accuracy increased from 84.5% to 90.9%.

Keywords: 

convolutional neural networks, data augmentation. infrared thermography, neonatal disease classification. pre-diagnosis system, respiratory system anomalies

1. Introduction

The first 28 days of life (the neonatal period) is the most vulnerable time for a baby’s survival. Neonates face the highest risk of dying in their first month of life at an average global rate of 18 deaths per 1,000 live births in 2018 [1]. According to the data published by the American Academy of Pediatrics, 10 out of 100 newborns need respiratory support after birth, and 1% requires in-depth resuscitation [2]. Risk factors such as meconium aspiration, prematurity, gestational diabetes, cesarean delivery, maternal chorioamnionitis and structural lung disorders cause respiratory abnormalities in neonates [3]. With the first breath the infant takes at birth, the lungs are opened and the fluid inside is replaced by air. This event usually takes place within the first 2-6 hours after birth. In some cases, the discharge of this fluid is delayed and the oxygen from some lungs does not change so that the baby cannot get all the oxygen it needs and suffers from respiratory distress [4]. In cases such as neonatal transient tachypnea or free air accumulation between visceral and parietal pleura (pneumothorax), it may cause the infant to remain oxygen deprived. If respiratory system anomalies are not detected at an early stage, they can cause cell and brain damage and even have fatal consequences [5]. Since rapid intervention is needed in such cases, the results produced by the computer-aided pre-diagnosis system right after birth can be life-saving.

Therefore, monitoring and evaluating the health status of neonates is very important for early intervention. The value of body temperature gives important information about life functions [6]. Diseases and inflammations cause local temperature changes on the body surface [7]. These changes, which cannot be distinguished at visible wavelengths in the early stages of diseases, can be distinguished at infrared wavelengths. In other words, a thermal imbalance that cannot be detected by the human eye can be detected by infrared detectors. Thermal cameras, convert infrared radiation emitted from the object into electrical signals, depending on the heat intensity [8]. Infrared thermography, a non-contact and non-invasive method, has been used in various medical studies on thermoregulation [9], breast cancer detection [10], neonatal follow-up [11], urology [12], and vascular diseases [13].

Infants should not be left out of the incubator for long periods. However, long-term imaging in the radiology department can cause impaired thermal balance and cause hypothermia by disrupting their physiological conditions [14]. Moreover, methods involving radiation and contact can be dangerous for the infant. To prevent these problems, a system that will help experts in the early diagnosis of diseases should be implemented. First, in 1980, Clark and Stothers analyzed heat distributions in the body using neonatal thermograms. The average skin temperature obtained from thermography and thermocouple is shown by comparison [11]. Saxena and Villital (1999) evaluated the feasibility of clinical application of IRT in the pediatric population and aimed to identify pathological conditions that could be followed up and diagnosed [15]. Christidis et al. (2003) made a thermal analysis of the first hour of neonates and found that the peripheral regions rapidly cooled shortly after birth [16]. In 2012, Abbas et al. examined the temperature change of the neonates and obtained results to prove that the environment in which they are located prevents the correct temperature measurement [17]. Abbas and Leonhardt explained the newborn infrared thermography pattern clustering in 2014 and stated that abnormalities such as tumors, inflammations, and infections cause local temperature increases or asymmetric thermograms. In 2019, health status detection of neonates using infrared thermography and deep convolutional neural networks was carried out by Ornek et al. in another study [18]. Savasci et al. have worked on the classification of unhealthy and healthy neonates using artificial neural networks in neonatal intensive care units using medical thermography in 2019 [19]. The first research results of this study were presented in an abstract format in 2020 using the narrow-scope deep learning model and only with the accuracy metric [20].

With the developments in machine learning and the spread of convolutional neural networks (CNN) in recent years, operations such as object recognition, classification, segmentation on images are performed with high specificity and sensitivity. However, the problem here is that the created network needs a lot of images of the same class in order to learn effectively. In this study, images taken from 34 neonatal were used. Thermal database was recorded within the neonatal intensive care unit (NICU) of Selcuk University Medicine Faculty. For the training of the CNN model, data augmentation was performed by applying brightness increase, sharpening, and contrast changing methods to thermal images. In addition to the original limited dataset which was created with 20 images from each neonate, an augmented dataset was obtained and the effects of data augmentation on the classification results were observed. The original data set contains 680 thermal images from 34 neonates, while the augmented data set contains 2060 thermal images. As a result of the experiment, how the data augmentation affects the classification accuracy was examined and a comparative study was presented.

2. Materials and Methods

In this section, firstly, the thermal imaging process and the disease classes in dataset are described. Then, the data augmentation methods used are explained and how the classification is done with the convolutional neural network model is expressed.

2.1 Used data and measurement setup

Thermal images were taken by the Variocam HD (InfraTec©) infrared camera, and recorded using a user interface named IRBIS, at the Selcuk University Medicine Faculty, Neonatal Intensive Care Unit. The camera resolution is 1.024 × 768 with a thermal sensitivity of about 0.01℃. A total of 34 neonates, 16 with pulmonary diseases, 10 with cardiovascular diseases, 8 with renal diseases with abdominal region anomaly were used. Images were obtained from neonates lying in supine position at a distance of 60 - 100 cm. Thermograms have been converted to raw RGB images by using portable computer and thermal camera application. The measurement setup is shown in Figure 1.

Figure 1. Thermogram recording process a) thermal camera, b) neonate, c) incubator, d) computer

In order to protect the temperature values of the infants, the image acquisition was set as one minute, and 100 thermal images were obtained from each newborn for approximately 60 seconds. Thus, 3400 thermal images were recorded from 34 newborn infants. In practice, 20 images from each neonate were randomly selected and used. Some of the thermal images used in the application are shown in Figure 2.

Figure 2. Image samples captured by thermal camera

2.2 Neonatal diseases and image dataset

In the study, neonates whose disease information was labelled by pediatricians were divided into 3 groups which respiratory system anomalies, cardiovascular and abdominal diseases. Then pulmonary diseases were examined in the first class and other diseases in the second class. Grouping process was done as shown in Table 1.

In practice, 320 images of 16 neonates with pulmonary abnormalities and 360 images of 18 neonates with heart diseases, abdominal and renal diseases were used. Of the 34 neonates, 11 were selected and reserved for testing, and 23 were used for training the model.

Table 1. Determination of dataset

Patient

Label

Group

Class

1

Respiratory Distress

Respiratory Anomalies

1

2

Respiratory Distress

Respiratory Anomalies

3

Respiratory Distress

Respiratory Anomalies

4

Respiratory Distress

Respiratory Anomalies

5

Respiratory Distress

Respiratory Anomalies

6

Respiratory Distress

Respiratory Anomalies

7

Respiratory Distress

Respiratory Anomalies

8

Respiratory Distress

Respiratory Anomalies

9

Respiratory Distress

Respiratory Anomalies

10

Respiratory Distress

Respiratory Anomalies

11

Esophageal Atresia

Respiratory Anomalies

12

Transient Tachypnea

Respiratory Anomalies

13

Transient Tachypnea

Respiratory Anomalies

14

Pneumothorax

Respiratory Anomalies

15

Respiratory Distress

Respiratory Anomalies

16

Esophageal Atresia

Respiratory Anomalies

17

Patent Ductus Arteriosus

Cardiovascular Diseases

2

18

Hypoplastic Left Heart

Cardiovascular Diseases

19

Aort Coarctation

Cardiovascular Diseases

20

Patent Ductus Arteriosus

Cardiovascular Diseases

21

Pulmonary Banding

Cardiovascular Diseases

22

Congenital Heart Anomaly

Cardiovascular Diseases

23

Aort Coarctation

Cardiovascular Diseases

24

Congenital Heart Anomaly

Cardiovascular Diseases

25

Atrial Septal Defect

Cardiovascular Diseases

26

Pulmonary Hypertension

Cardiovascular Diseases

27

Necrotizing Enterocolitis

Abdominal Anomalies

28

Necrotizing Enterocolitis

Abdominal Anomalies

29

Intestinal Obstruction

Abdominal Anomalies

30

Diaphragm Hernia

Abdominal Anomalies

31

Necrotizing Enterocolitis

Abdominal Anomalies

32

Intestinal Atresia

Abdominal Anomalies

33

Necrotizing Enterocolitis

Abdominal Anomalies

34

Kidney Failure

Abdominal Anomalies

2.3 Data augmentation

Figure 3. Data augmentation process

Figure 4. Augmented images (a) brightness enhancement, (b) contrast changing, (c) sharpening

Immunity in newborns begins to mature from the 8th week after birth. Infants under observation in NICU are vulnerable to environmental influences and diseases in their first weeks of life [21]. Newborns who are dependent on vital support and medical monitoring devices cannot be removed from the incubator where the temperature balance is maintained. Although thermal imaging is performed without leaving the incubator, the protective top cover of the incubator is opened during imaging and the thermal insulation is broken for a short time. Given this sensitive situation of neonatal intensive care units, creating a thermal image database of thousands of images CNN needs to achieve successful classification can pose medical drawbacks. Another reason that reveals the need to increase data is that it is difficult to find the sufficient sample cases from each of the many different diseases that can be encountered in newborns. Therefore, it becomes necessary to make an efficient classification study using a limited data set. Accordingly, in this study, data augmentation methods which brightness enhancement, contrast changing, and sharpening were used to obtain the amount of data required by deep learning. The data augmentation process and occurred training set size given in Figure 3.

220 of the 680 images in the original data set were reserved for testing the model and the remaining 460 images were increased to 1380 images with 3 different data augmentation technique. The following data-augmentation techniques were used in this study:

Brightness enhancement;

$B(i, j)=f(i, j)+(a)$               (1)  

Contrast changing;

$C(i, j)=f(i, j) *(b)$                   (2)

Sharpening;

$S(i, j)=f(i, j)+\tau * H(i, j)$               (3)

Here, $f(i, j)$ represents the actual pixel values, a represents the amount of light, and b represents the contrast coefficient. In this study, a and b values were chosen as 0.3 and 0.5, respectively, experimentally. Sharpness is expressed as the contrast between different colors. Conversely, sharpening images means increasing the contrast of the edges where different colors intersect. The term $\tau$ represents a tuning parameter that typically ranges from 0 to 2; high values of $\tau$ may create undesirable effects in the output image. Additionally, $H(i, j)$ used to implement a high-pass filter. Augmented images are shown in Figure 4.

With data augmentation, the total size of the dataset used in the application has reached 2060.

After removing 220 images from 11 infants allocated for testing, the remaining images were used to train the model.

2.4 CNN model

CNN is a deep learning model that produces high-performance results in mostly image-based implementations such as image classification, object recognition, noise cancellation, image segmentation etc. CNNs contain 2 basis layers which convolution and a fully connected network. There are two processes in the convolution layer: convolution and pooling. Feature extraction from the image with convolution process, feature selection is done by pooling process. The fully connected layer consists of two parts, flattening and network layers. The feature matrix obtained in the flattening layer is transformed into a feature vector.

Figure 5. CNN model

Table 2. Description of CNN architecture

Layer

Type

Configuration

Activation

Size

Number

1

Convolution

ReLU

3x3

32

2

Max-Pooling

-

2x2

-

3

Convolution

ReLU

3x3

64

4

Max-Pooling

-

2x2

-

5

Convolution

ReLU

3x3

64

6

Max-Pooling

-

2x2

-

7

Drop-Out

-

0,25

-

8

Dense

ReLU

128

1

9

Dense

ReLU

64

1

10

Dense (Output)

Softmax

1

1

Figure 6. Representation of the proposed system in the form of a flow chart

The fully connected network layer consists of two parts, flattening and network layers. The feature matrix obtained in the flattening layer is transformed into a feature vector. The classification of this feature vector obtained from the network layers is performed by softmax classifier. Softmax operate generates probability-based loss value by exploitation score values created by artificial neural network. As seen in Figure 5, proposed CNN architecture included three convolutional, three pooling, and three dense layers. We resized the dimensions of the input images from 512 x 512 to 64 x 64. The first convolutional layer consisted of 32 different 3 x 3 dimensional layers (activation function = relu), whereas the second and third convolutional layer contained 64 different 3 x 3 dimensional layers (activation function = relu). The dimensions of the three used pooling layers were 2 x 2 (pooling = max pooling). The first two dense layers consisted of 128, and 64 neurons, respectively (activation function = relu), and the third dense layer, called the output layer, included one neuron (activation function = sigmoid). The output values varied from 0 to 1.

The description of used CNN model is given in Table 2. The block representation of the system, which includes the steps of obtaining thermograms, data augmentation, training of the CNN algorithm, and evaluation of the results is in Figure 6.

3. Evaluation

Obtained results evaluated by confusion matrix, accuracy, sensitivity and specificity metrics.

Rows and columns of confusion matrix are as follows in Table 3;

Table 3. Confusion matrix

 

Real

Positive

Negative

Prediction

Positive

TP (true positive)

FP (false positive)

Negative

FN (false negative)

TN (true negative)

Accuracy (4), sensitivity (5) and specificity (6) metrics were calculated to evaluate the results of classification of thermal images.

Accuracy $=\frac{\text { TP+TN }}{\text { TP+TN+FP+FN }}$               (4)

Sensitivity $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$           (5)

Specificity $=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$          (6)

4. Results and Conclusions

In this study, CNN model and data enhancement methods were used to determine respiratory system anomalies in neonates. In the classification, neonates with various pulmonary abnormalities are in the 1st class, and neonates with other diseases (cardiovascular diseases, abdominal diseases) constitute the 2nd class. In the first part of the study, the classification was made with 20 images from each neonate and 680 images of 34 infants with 16 pulmonary diseases and 18 other diseases. In the second part, all the images of the infants and the images obtained with different augmentation techniques were used and the classification was made with 2060 images. The confusion matrices of the classification is given in Table 4 and Table 5. Table 6 shows the results of Accuracy, sensitivity, and specificity.

The evaluation of original limited dataset results obtained as 84% specificity, 85% sensitivity, 85% accuracy. When the number of the thermal images was increased from 680 to 2060, all evaluation metrics values of augmented dataset results were increased. For example, the test accuracy metric rised from 85% to 91% and sensitivity calculated as 100%.

The results showed that it is possible to get better results with data increase and the model used is suitable for the detection of neonatal diseases from thermal images. In order to propose a pre-diagnosis method that can detect respiratory system anomalies in newborns, a dataset containing different disease classes has been defined. For this reason, it was placed as the second / opposite class in the cardiovascular and abdominal diseases dataset, which is predicted to show thermal asymmetry between the neck and groin. This study may be a motivation for the classification tasks of cardiovascular diseases or abdominal anomalies in future studies. Promising results were obtained in reducing neonatal mortality by thermal imaging. Using more thermal data or increasing the number of data with different image enhancement techniques may have better results.

Table 4. Confusion matrix obtained by original dataset

 

Real

Respiratory

Other

Prediction

Respiratory

80

14

Other

20

106

Table 5. Confusion matrix obtained by augmented dataset

 

Real

Respiratory

Other

Prediction

Respiratory

80

0

Other

20

120

Table 6. Accuracy, sensitivity and specificity results

 

Original Dataset

Augmented Dataset

Accuracy

0.85

0.91

Sensitivity

0.80

0.80

Specificity

0.88

1

Acknowledgment

This study was supported by the Scientific and Technological Research Council of Turkey (TUBITAK, project number: 215E019).

  References

[1] Hug, L., Alexander, M., You, D., Alkema, L., for Child, U.I.A.G. (2019). National, regional, and global levels and trends in neonatal mortality between 1990 and 2017, with scenario-based projections to 2030: A systematic analysis. The Lancet Global Health, 7(6): e710-e720. https://doi.org/10.1016/S2214-109X(19)30163-9

[2] Barber, C.A., Wyckoff, M.H. (2006). Use and efficacy of endotracheal versus intravenous epinephrine during neonatal cardiopulmonary resuscitation in the delivery room. Pediatrics, 118(3): 1028-1034. https://doi.org/10.1542/peds.2006-0416 

[3] Baseer, K.A.A., Mohamed, M., Abd-Elmawgood, E.A. (2020). Risk factors of respiratory diseases among neonates in Neonatal Intensive Care Unit of Qena University Hospital, Egypt. Ann Glob Health, 86(1):22. https://doi.org/10.5334/aogh.2739

[4] Soll, R. (1999). Early versus delayed selective surfactant treatment for neonatal respiratory distress syndrome. Cochrane Database of Systematic Reviews, (4). https://doi.org/10.1002/14651858.CD001456

[5] Berger, R., Garnier, Y. (1999). Pathophysiology of perinatal brain damage. Brain Research Reviews, 30(2): 107-134.

[6] Lahiri, B.B., Bagavathiappan, S., Jayakumar, T., Philip, J. (2012). Medical applications of infrared thermography: A review. Infrared Physics & Technology, 55(4): 221-235. https://doi.org/10.1016/j.infrared.2012.03.007

[7] Knobel, R.B., Guenther, B.D., Rice, H.E. (2011). Thermoregulation and thermography in neonatal physiology and disease. Biological Research for Nursing, 13(3): 274-282. https://doi.org/10.1177/1099800411403467

[8] Govindarajan, J. (2007). A case for joint development of IR cameras in India. Journal on Intelligent Electronic Systems, 1(1). https://doi.org/10.18000/ijies.30005

[9] Kammersgaard, T.S., Malmkvist, J., Pedersen, L.J. (2013). Infrared thermography–a non-invasive tool to evaluate thermal status of neonatal pigs based on surface temperature. Animal, 7(12): 2026-2034. https://doi.org/10.1017/S1751731113001778

[10] Arora, N., Martins, D., Ruggerio, D., Tousimis, E., Swistel, A.J., Osborne, M.P., Simmons, R.M. (2008). Effectiveness of a noninvasive digital infrared thermal imaging system in the detection of breast cancer. The American Journal of Surgery, 196(4): 523-526. https://doi.org/10.1016/j.amjsurg.2008.06.015

[11] Clark, R.P., Stothers, J.K. (1980). Neonatal skin temperature distribution using infra‐red colour thermography. The Journal of Physiology, 302(1): 323-333. https://doi.org/10.1113/jphysiol.1980.sp013245

[12] Rasor, J.S., Zlotta, A.R., Edwards, S.D., Schulman, C.C. (1993). Transurethral needle ablation (TUNA): Thermal gradient mapping and comparison of lesion size in a tissue model and in patients with benign prostatic hyperplasia. European Urology, 24: 411-414. https://doi.org/10.1159/000474339

[13] Bagavathiappan, S., Saravanan, T., Philip, J., Jayakumar, T., Raj, B., Karunanithi, R., Jagadeesan, K. (2008). Investigation of peripheral vascular disorders using thermal imaging. The British Journal of Diabetes & Vascular Disease, 8(2): 102-104. https://doi.org/10.1177/14746514080080020901

[14] Brown, D.J., Brugger, H., Boyd, J., Paal, P. (2012). Accidental hypothermia. New England Journal of Medicine, 367(20): 1930-1938.http://dx.doi.org/10.1056/NEJMra1114208

[15] Saxena, A., Willital, G. (2008). Infrared thermography: Experience from a decade of pediatric imaging. European Journal of Pediatrics, 167:757-764. https://doi:10.1007/s00431-007-0583-z 

[16] Christidis, I., Zotter, H., Rosegger, H., Engele, H., Kurz, R., Kerbl, R. (2003). Infrared thermography in newborns: the first hour after birth. Gynäkologisch-geburtshilfliche Rundschau, 43(1): 31-35. https://doi.org/10.1159/000067168

[17] Abbas, A.K., Heimann, K., Blazek, V., Orlikowsky, T., Leonhardt, S. (2012). Neonatal infrared thermography imaging: Analysis of heat flux during different clinical scenarios. Infrared Physics & Technology, 55(6): 538-548. https://doi.org/10.1016/j.infrared.2012.07.001

[18] Ornek, A.H., Ceylan, M., Ervural, S. (2019). Health status detection of neonates using infrared thermography and deep convolutional neural networks. Infrared Physics & Technology, 103: 103044. https://doi.org/10.1016/j.infrared.2019.103044

[19] Savasci, D., Ornek, A. H., Ervural, S., Ceylan, M., Konak, M., Soylu, H. (2019). Classification of unhealthy and healthy neonates in neonatal intensive care units using medical thermography processing and artificial neural network. In Classification Techniques for Medical Image Analysis and Computer Aided Diagnosis, pp. 1-29. Academic Press. https://doi.org/10.1016/B978-0-12-818004-4.00001-7

[20] Ervural, S., Ceylan, M. (2020). Detection of pulmonary abnormalities neonates using thermal imaging and convolutional neural network. Conference on Machine Intelligence in Medical Imaging.

[21] Offit, P.A., Quarles, J., Gerber, M.A., Hackett, C.J., Marcuse, E.K., Kollman, T.R., Landry, S. (2002). Addressing parents’ concerns: do multiple vaccines overwhelm or weaken the infant’s immune system? Pediatrics, 109(1): 124-129. https://doi.org/10.1542/peds.109.1.124