Deep Multi-View Breast Cancer Detection: A Multi-View Concatenated Infrared Thermal Images Based Breast Cancer Detection System Using Deep Transfer Learning

The cancer of Breast is the most commonly diagnosed non-cutaneous malignancy in women’s all over the world. Its timely and early detection in the initial stages proved to be very helpful and thus eventually enhances the chances of patient’s survival [1]. Breast cancer ranks fifth when it comes to the number deaths caused by any cancer worldwide as per GLOBOCAN data 2020. The cancer of breast is the major reason of women’s mortality and morbidity surpassing lung cancer in 2020 [2]. Breast cancer basically originates from the tissues of breast and especially from the interior lining of the lobules or the milk ducts in the breast. Breast cancer cells are the result of the RNA or DNA modification and mutation which eventually transformed a normal cell into a cancer cells. Basically bacteria, chemicals in the air, electromagnetic radiation, viruses, fungi, mechanical cell-level injury, parasites, heat, water, food, free radicals, aging of DNA or RNA and evolution are all can be the reason behind this type of harmful mutation of DNA which leads to cancer of breast [3]. Apart from this, the major risk factors associated with breast cancer are the obesity, family background or history, race, genetics, alcohol consumption, avoid doing physical labor for long time etc. [4]. Detection of breast cancer at an early stage accompanied by the advance medical treatment will play a major role to avoid mortality from this type of cancer. The mostly employed ways to diagnose breast cancer is by performing regular screening tests and these test may include manual screening by physicians as well as utilizing the medical imaging techniques like Thermography, Mammography, and Ultrasound etc. [5]. Even there are some developed countries which makes it mandatory for women’s over 30 years of age and associated with high risk factors to go through these screening test on a yearly basis to decrease the high mortality rate caused by breast cancer in their countries [6].

The Mammography and Thermography are the widely used medical imaging techniques employed for the detection of breast cancer. Since the Mammography imaging technique suffers from the drawbacks of high cost as well as patient exposure to high radiation which could result in other health problems [7]. Due all these major drawbacks, it’s the Thermography imaging technique which is getting popular and prescribed as a complementary test by the doctors for the early screening of breast cancer especially in case of some major symptoms. Since this thermography technique is noninvasive and contactless, which makes its popular among female patients as compare to other imaging techniques. This thermal imaging technique is based on the concept of capturing the infrared radiation emitted from the skin of human body which determines bilateral symmetry patterns in normal cases and any variation in such symmetry patterns will highlights the anomalies. These anomalies could be the result of circulation and metabolic changes which occurred due to the excessive heat produced by the tissues of tumor present in the breast [8]. As the cancerous tissues metabolize faster than the normal tissues raising the temperature of the tumor location on the breast as compare to the surrounding regions. This change in temperature is precisely captured with the help of this thermal imaging technique employing thermal cameras. This technique captures breast thermal images utilizing two acquisition protocols i.e. Static and Dynamic [9, 10]. The Static thermal images are not related to time and they are taken one per patient at different angles. Whereas the Dynamic thermal images are captured during particular interval of time while passing through a cooling step [11]. This technique is having the capacity to detect breast cancer at a very early stage even when the patient is asymptomatic provided that patient is screened at regular intervals. The Static as well as Dynamic protocol based breast thermal images of normal and abnormal patient is presented with the help of Figure 1. The Figure 1 also showcases their corresponding grey scale images (free from image artefacts). The normal Frontal breast thermal images of Static and Dynamic protocol based in Figure 1 are having symmetric heat patterns i.e. uniform temperature throughout both the breasts region of a volunteer. Along with this, the left and right breast thermal images are also showcasing shape symmetry i.e. almost same shape of both the breast regions. Whereas the Static and Dynamic protocol based abnormal frontal breast thermal images are having asymmetric heat patterns, which are highlighted in the Figure 1 and also the left as well as right abnormal breast thermal images of a patient are lacking the shape symmetry quotient i.e. shape of both the breasts are somewhat different with each other.

1.jpg

Figure 1. The normal and abnormal static as well as dynamic breast thermal images

The deep learning and deep transfer learning are actually the two subdomains of machine learning. The machine learning based various Computer aided diagnosis systems (CAD) are developed in the last decade and so, which are accurately detecting as well as classifying various brain, lungs, liver, breast etc. disorders utilizing the clinical images of these organs [12]. These CAD systems research and development are even more important in the context of present scenario, where the whole world is facing severe crisis of medical facilities as well as doctors [13]. So there is a need to develop robust and accurate CAD systems, which could assist radiologist in the interpretation of breast thermal images into normal or abnormal images. At the same time the low diagnostic ability of thermography technique, which is majorly caused due to the limited as well as weak expertise of doctors in the interpretation of breast thermal images to detect breast anomalies can be easily enhanced with the usage of these CAD systems based on deep learning and deep transfer learning. As the deep learning and deep transfer learning based systems are offering advantages like fully automated in nature, less complex to develop (requires hyper parameter tuning), offering optimal training time with highest accuracy. Unlike the traditional machine learning based systems, which requires proper manual selection of segmentation, feature extraction methods along with an efficient classifier for performing classification. These systems also require more training time and offers less accuracy. It is because of all these disadvantages, now the machine learning based systems are getting replaced from the more advanced and accurate deep learning and deep transfer learning based systems. During the last two years, a significant amount of research is carried out for the development of such CAD systems especially based on deep learning and deep transfer learning for automating the entire process of diagnosing the breast cancer utilizing the various medical imaging modalities like Mammography [14-16], Ultrasound [17, 18] and Thermography [19, 20] etc. The present research trend in this field is the use of more accurate and advanced deep transfer learning models, which even delivers good results with limited size datasets for training purpose [12]. But little magnitude of research is conducted for the developments of CAD systems utilizing thermal images as compare to other medical imaging modalities like Mammography and Histopathological images. Considering the fact that there is a need to develop more accurate and robust computer assisted automated approach or CAD system for the automated classification of breast thermal images into either normal or abnormal cases, the following contributions of this research article are stated as follows:

(1) This research article presents an accurate fully automated breast cancer detection system based on the VGG16 deep transfer learning model utilizing both the Static and Dynamic thermal images.

(2) This research article presents a novel idea of training the deep transfer learning models with the help of a 3 sides i.e. Multi-view breast thermal images, which can be fabricated by concatenating the conventional Left, Right and Frontal breast thermal images for the first time. Then experimentation and simulation is done to showcase that these Multi-view concatenated thermal images tends to delivered better results as compare to the conventional single view thermal images.

(3) This research article also presents a brief experimentation utilizing both types of i.e. Static and Dynamic breast thermal images to showcase which type of thermal images delivered better results during training and testing phase.

The proposed approach consists of three major stages as: Preprocessing, Augmentation and finally fine tuning as well as training of VGG16 model for performing accurate binary Breast cancer classification into normal or abnormal cases. The overall proposed approach is well illustrated with the help of the Figure 4 below:

4.jpg

Figure 4. The overall proposed approach based on the VGG16 for the breast cancer detection

3.1 Dataset used

The Database for Mastology Research with Infrared Image (DMR-IR) is a breast thermal images dataset used in this research study [21]. This database is a well-known and considered to be a global dataset for the researchers working in this domain as it consists of both the types of breast thermal images i.e. Static and Dynamic protocol based. This dataset consists of total 287 women thermal images along with their corresponding thermal matrices. From these 287 women around 1345 thermal images are acquired employing the FLIR SC-620 thermal camera at resolution of 640*480 with thermal sensitivity of 45mk. These women volunteers belong to the age group of 29 to 85 years. All these thermal images along with their annotations are well authenticated by the radiologist. Majorly Static thermal images are taken from five different angles, whereas Dynamic thermal images are taken in a patch of 20 each. In this research study, we have taken per patient three lateral views i.e. Left, frontal and right Static as well as Dynamic thermal images as these three views are able to cover the entire breast region. For the augmentation purpose 200 left, 200 frontal and 200 right static and same number of dynamic thermal images are taken. Whereas for the testing purpose 50 left, 50 frontal, 50 right and 50 Multi view static and dynamic breast thermal images are taken.

3.2 Preprocessing

In this stage, both the Static as well as Dynamic thermal images datasets are preprocessed employing cropping, contrast enhancement, normalization and lastly resizing. In this stage, various image artifacts are also removed to further enhance the breast thermal images. Finally with the aid of OpenCv and NumPy functions all the three breast views i.e. left, frontal and right are concatenated in order to generate a more informative Multi view thermal breast images. Post preprocessing all these thermal images are transformed into thermal images of dimensions 224*224 from the 640*480 original dimensions. This reduction in dimensions is imperative as this is the default input size of the VGG16 model and will eventually result in improved computational efficiency of proposed system. Whereas by training the proposed system with less resolution images i.e. less information images eventually helps the proposed system to optimize better in the training phase, so that it can generalize better during the testing phase and hence give accurate results. Now all these images are augmented in the next stage for creating a large size dataset for the training purpose to avoid over fitting.

3.3 Augmentation

Now both the Static and Dynamic thermal image datasets are augmented using the ImageDataGenerator function of Keras. The augmentation is considered to be an important aspect of all the deep learning based classification models as it enhances the accuracy (classification) quotient by reducing the bias factor as now the models has adequate samples to learn [35, 36]. Using the above Keras function, datasets augmentation is done utilizing data generation of 4 types like rotation range, shear range, rescaling and zoom range. After doing a number of experiments, the best values for rotation range=5, shear range=0.02, zoom range=0.02 and rescale=1./255.

3.4 The proposed fine-tuned VGG16 model

The VGG 16 is a very popular pre-trained deep transfer learning model. As the pre-trained models are more accurate and efficient as compare to the newly developed from scratch CNN architectures especially in classification tasks. The VGG16 deep transfer learning model is basically trained over the Image Net dataset consisting of more than 1.4 Crore images and hence achieves a precision of 92.7%. This VGG16 pre-trained model was initially proposed by the Karen Simonyan and Andrew Zisserman [37]. The VGG 16 pre-trained model is considered to be an improved version of its predecessor the Alex Net neural network. In VGG 16 model, there are sixteen layers segregated by five max pooling layers. This pre-trained model also consists of several 3*3 filters for better learning.

5.jpg

Figure 5. The proposed fine-tuned VGG16 model summary

This VGG16 pre-trained model is fine-tuned and hence delivers better results in this research study by utilizing the concept of transfer learning. The last three layers of the conventional VGG16 pre-trained model is removed and new three layers like Flatten, Dropout, and Dense are appended with the required number of neurons in the last fully connected layer. The number of neurons in the last fully connected layer totally depends on the type of classification i.e. binary or degree of multiclass and in our case its two neurons along with the softmax classifier. The rate of learning is set very high for these newly appended fully connected layers. As the remaining model except for these three layers remain unchanged and hence undergoes training with the aid of augmented Static and Dynamic thermal datasets. This model uses an Adam optimizer equipped with a rate of learning of 0.00001, 100 epochs along with batch size of 16. Both the augmented datasets are split into two sets like training (70%) and validation sets (30%). In order to check the model training performance, a validation check is undertaken each after few iterations. The model summary of the fine-tuned VGG 16 pre trained model with three added layers for the breast cancer detection is illustrated with the help of Figure 5.

The algorithm of the proposed fine-tuned VGG16 model with last three added layers as head model for the Breast cancer detection as:

Algorithm 1 (Proposed fine-tuned VGG16 model)

Input:   D₁←Augmented Static breast thermal dataset

               D₂←Augmented Dynamic breast thermal dataset

               µ←Rate of learning

               β←Batch size

               ϵ←Number of epochs

Output: Fine-tuned VGG16 model with updated weights

Steps:

1. Both the augmented datasets are divided into the training as well as validation subsets.

2. The training sets class weights are calculated

3. Basemodel-VGG16 (weights, ImageNet)

4. Layers- VGG16 (weights, none)

5. New model- Output (basemodel)

6. Layers-Set (flatten, dropout, dense) three layers on top of VGG16 layers.

7. The hyper parameters values of µ, β, ϵ are initialized.

8. The output updated weights are stores as the result of training of VGG16 model.

9. Computation of binary cross entropy loss is done in forward propagation and

          Cross Entropy = (1 − z) log (1 − p)) + (− (z log (p))

          Where, z denotes correct class label and p denotes the model predicted probability.

10. As a result, the Adam optimizer is updated in back propagation.

3.5 Comparison approach

For the sake of comparison and evaluation on both the datasets i.e. Static and Dynamic breast Thermal datasets. The three popular and recent deep transfer learning (DTL) models like VGG19 [38], ResNet50 [39] and InceptionV3 [40] are used. These three DTL models are initially fine-tuned and trained using the Static and Dynamic breast thermal augmented datasets. The ultimate object of this comparison is to illustrate that how these commonly used DTL models performs on the above mentioned datasets. The common algorithm 2 for the comparison approach is given as:

Algorithm 2 (comparison approach)

Input: Augmented Static breast thermal dataset consist of normal and abnormal images

             Augmented Dynamic breast thermal dataset consist of normal and abnormal images

Output: The Fine-tuned and trained VGG 19, ResNet50 and InceptionV3 models for breast cancer detection

Steps:

1. Preprocess the breast thermal images of both the datasets in order to remove the noise, symbols and artifacts.

2. Resize these thermal images to the size of 224-by-224-by 3 or 299-by-299-by 3 for the training of these VGG 19, ResNet50V2 and InceptionV3 DTL models.

3. Perform the augmentation of both these datasets using the Keras ImageDataGenerator function of four types rotation range=5, shear range=0.02, zoom range=0.02 and rescale=1./255.

4. The fine tuning and training of these four DTL models over the augmented datasets.

5. The VGG19 models tends to converge at 100 epochs.

6. The ResNet 50 and Inception tends to converge at 200 epochs.

7. Simulation and evaluation of these DTL models over the testing dataset.

The VGG19, ResNet50V2 and InceptionV3 DTL models hyper parameters values are presented with the help of Table 2. All the four deep transfer learning models are trained and evaluated with a learning rate equal to 0.00001, mini-batch size of 16 and Adam [41] is opted as an optimizer technique for the weights adjustment. Whereas the size of input image and number of epochs required to converge varies from model to model.

Table 2. The hyper parameters values of VGG19, InceptionV3 and ResNet50V2

This approach will help in performing the early breast cancer screening. Even there are certain researches carried out in the past that claims that the thermography technique is quite capable of detecting the breast abnormalities prior development of breast cancer [42]. As the proposed system based on VGG16 is trained with both the normal as well as abnormal breast thermal images. It is quite evident from the Figure 1 above that the abnormal breast thermal images are possessing characteristics or features like asymmetric heat patterns in both the breast regions. Another major characteristic or feature persisting in almost all the breast cancer or abnormal breast thermal images is the presence of shape asymmetry quotient in between the left and right Breasts of a patient. Whereas in case of normal breast thermal images, the features like presence of symmetric heat patterns and shape symmetry in between both the left and right breast regions. So the VGG 16 model layers are majorly learning features like asymmetric heat patterns along with shape asymmetry from abnormal breast thermal training images as well as learning features like symmetric heat patterns and shape symmetry from the normal training images in order to perform correct classification.

The Multi-view thermal infrared images are proved to be much more accurate in terms of classifying the breast abnormality as they offer more information to the deep transfer learning models. The proposed approach tends to deliver better results over the Dynamic breast thermal infrared images as compare to the static images. Although the difference is very minute but these dynamic images are more accurate in terms of detecting the breast abnormality as it is clearly showcased by the result section. The VGG16 deep transfer learning model tends to converges at 100 epochs and after that the training accuracy is not improving. The proposed approach based on VGG 16 model offers another advantage as it trained faster as compare to the counterpart ResNet50 and Inception V3. As these two deep transfer learning models tends to converges at 200 epochs and hence consumes more time for training and validation.

Whereas the VGG16 model also tends to converges at 100 epochs just like VGG19 but offers high validation as well as testing accuracy.

The proposed Deep Multi-view Breast cancer Detection system based on VGG16 DTL model in comparison to the conventional machine learning based Breast cancer detection systems are technically much superior because of the following reasons:

(1) The state of the art machine learning based breast cancer detection systems proposed in the fast are computationally very expensive and also needs more training samples in order to deliver better performance. Whereas the proposed VGG16 based system is computationally efficient as it’s converges at just 100 epochs and also delivers high accuracy on the smaller training datasets of both the Static and Dynamic protocol based breast thermal images.

(2) The proposed system based on VGG16 as a pre trained deep transfer learning model over the Image Net dataset [37] and hence uses this knowledge (features, weights, etc.) in order to perform the better classification of breast cancer abnormality in this research study. Whereas the conventional machine learning based approaches are training the classifiers in an isolated manner and are lacking any sort of knowledge.

(3) The proposed Breast cancer detection system based on VGG16 is fully automated and hence less complex to develop and operate. Whereas the conventional machine learning based approaches are complex to develop as these required selection of proper segmentation methods, feature extraction method and finally the type of classifiers to be used for classification. The conventional machine learning approach only tends to delivered good results if these selected methods and classifiers are working well with each other and hence quite complex to develop.

(4) The proposed VGG16 deep transfer learning system is also offering optimal training performance because it leverages knowledge (weights, features) from the previously trained models. Whereas the traditional machine learning based breast cancer detection could have taken more training time.