An Optimized Hybrid Model of Convolutional Neural Networks and eXtreme Gradient Boosting for Enhanced Mammographic Tumor Classification

Breast cancer, recognized as the most prevalent cancer in women and the second leading cause of cancer-related deaths, necessitates an urgent need for robust and efficient diagnostic methods. Mammography, a widely utilized medical imaging technique, has proven instrumental in early diagnosis, thereby improving treatment outcomes. However, the escalating volume of mammography images demands swift and precise classification methods for effective large-scale population screening. This paper seeks to address this pressing need by proposing an advanced image classification model leveraging both deep learning and machine learning algorithms [1, 2].

Contemporary classifiers such as Support Vector Machine (SVM) [3], K-nearest neighbor (KNN) [4, 5], Random Forest (RF) [6], AdaBoost [7], Xgboost [8], and Convolutional Neural Network (CNN) [9] have demonstrated significant efficacy in medical image classification. Nevertheless, the complexity and diversity of medical image data perpetually challenge these methods, underscoring the need for refinement.

Deep Neural Networks (DNNs), capable of intricate function approximation, have shown promise in various domains including facial recognition, object detection, and image classification. Their ability to efficiently extract features and perform classification elucidates their potential applicability in mammographic image classification. However, their complex structure, training difficulty, and high processing costs often impede their efficient implementation.

In conventional medical image classification tasks, feature extraction is a crucial yet time-consuming process, with the quality of extracted features dramatically impacting classification performance. CNN, a type of DNN, has excelled in feature extraction for various applications such as sentiment classification [10], demonstrating its potential for mammographic image classification. However, traditional classifiers linked to CNN often fail to comprehend the extracted features fully, thus limiting classification accuracy.

Addressing these challenges, our study's first contribution is the integration of a CNN model and a machine learning algorithm, both exhibiting exceptional performance in medical image classification. We evaluate several pre-trained CNN models, including VGGNet19 [11], EfficientNetB7 [12], InceptionV3 [13], ResNet152 [14], DenseNet201 [15], and MobileNetV2 [16], in tandem with various classifiers such as KNN, SVM, AdaBoost, Xgboost, and RF for tumor classification in mammography images.

The experimental results, in the validation process of the optimal CNN model, show that the EfficientNetB7 model obtains the highest scores among all the pre-trained models used in combination with the Xgboost algorithm in the automatic medical image classification process. The results of this first study involve a novel image classification model called EfficientNetB7-Xgboost to increase the performance of medical image classification. The proposed model achieves improved results by incorporating EfficientNetB7 as a trainable feature extractor to automatically get features from the input and using Xgboost as a high-level recognition network to generate results. This two-step model ensures efficient feature extraction and classification.

However, in a machine learning model, their parameters affect their effectiveness to a large extent. Indeed, the Xgboost model has many hyperparameters and human experience is required to determine the optimal parameters, which increases the difficulty of the model. Nevertheless, the learning efficiency and generalization ability of Xgboost is related to the selection of appropriate parameters. The parameters have a direct impact on the efficiently of the model predictions. Therefore, evolutionary algorithms may be the solution to this challenge. In fact, particle swarm optimization [17], genetic algorithm [18], and ant colony algorithm [19] have been approved to improve the parameters of Xgboost model. However, the particle swarm optimization model and the genetic algorithm fall quickly into local extremes. The three principal limitations of the algorithm are the exploration ratio, the stagnation stage and the convergence speed of the algorithm. On the other hand, recently, the cuckoo search algorithm (CS cuckoo search) is an evolutionary algorithm proposed [20]; it has the advantage of strong global search capability, few parameters and a nice search path and it is efficient to solve multi-objective problems. Motivated by the above facts, the second contribution of this article is the combination of the CS algorithm and the Xgboost algorithm to optimize the hyperparameters of Xgboost.

In this paper, a tumor mammography prediction model based on EfficientNetB7-CS-Xgboost is proposed. The principal contributions of this paper are as follows:

• To make a comparative analytical study of several pre-trained CNN architectures as feature extractors for mammography image analysis combined with the several classifiers to propose the best classifier in the given classification domain. The new model is called the EfficientNetB7-Xgboost model.
• The development of a new EfficientNetB7-CS-Xgboost model with Xgboost using CS to optimize its parameters. This can practically improve the efficiency of mammography tumor classification, including improving the detection accuracy of various tumor types.
• To evaluate the performance of the EfficientNetB7-CS-Xgboost model, the global set of metrics are used and compared with Xgboost and other ensemble learning models EfficientNetB7-PSO-Xgboost. The evaluation is based on two datasets.

Section 2 of this paper discusses related work, while Section 3 provides a detailed description of the proposed algorithm. Section 4 details the mammography image datasets used, presents the analysis results, and highlights some limitations of our approach. Section 5 concludes the paper and suggests potential future directions.

The objective of this section is to present the novel approach and evaluate the results. This study consists of classification of mammography images that detects abnormalities at an early stage. This section experiments with transfer learning using pre-trained convolutional neural network models and some classifiers to enhance the accuracy of results.

11.png

(a) 1^st Contribution: A comprative study

11b.png

(b) 2^nd Contribution: Optimized cuckoo search

Figure 1. The proposed approach framework

The image dataset and the process used to classify mammograms into two classes (positive and negative) are shown in Figure 1 and are presented in detail in the next sub-sections. In the first sub-section, a description of the images acquired from the Mammographic Institute Society Analysis (MIAS) and Breast Cancer Image (CBIS-DDSM) datasets is provided and the preprocessing applied to these images is described. The objective of the second sub-section is to test some methods and approaches in order to establish a comparative table showing the accuracy rates of each method, which is the first contribution of this paper (see Figure 1 (a)). Finally, the third sub-section presents the Xgboost classifier optimized with the Cuckoo search algorithm which is the second contribution of this study (see Figure 1 (b)).

3.1 Datasets preprocessing

3.1.1 DDSM mammography dataset

The dataset is composed of images from the DDSM [28] and CBIS-DDSM: Breast Cancer Image Dataset [29] databases. The images were preprocessed and transformed to 299×299 images by extracting the ROIs. The data are saved in tfrecords files for TensorFlow. The data set which includes 55,890 training examples (14% positive, 86% negative) are saved in 5 tfrecords files.

The data was split into training and test data. However, this split was done improperly, since the test nimpy files include only masses and the validation files include only calcifications. Therefore, these files must be used together to obtain complete and balanced test data.

- Preprocessing

The dataset is the combination of positive images from the CBIS-DDSM data and negative images from the DDSM data. These data were pre-processed to be converted to 299×299 images. The negative images (DDSM) were divided into 598×598 tiles and resized to 299×299.

The ROIs from the positive images (CBIS-DDSM) were selected using the masks with a small amount of fill to obtain context. Each one of the ROIs was then randomly cropped thrice into 598×598 images with random flipping and rotation, and resized to 299×299 to pass them into the CNN models.

The images are annotated with two stickers:

- Stickers normal 0 for negative and 1 for positive.

- Stickers complete multiclass labels, 0 for negative, 1 for benign calcification, 2 for benign mass, 3 for malignant calcification, 4 for malignant mass.

3.1.2 MIAS

The Mammographic Institute Society Analysis (MIAS) database [30] has mammogram images with a size of 1024×1024 pixels and a resolution of 200 microns. It included 322 mammograms of the right and left breasts of 161 patients, of which 54 were diagnosed as malignant, 69 as benign, and 207 as normal. It included also a file that lists the mammograms in the MIAS database and achieves appropriate details, such as the class of abnormality, the xy coordinates of the image of the center of the abnormality, the approximate radius (in microns) of the abnormality, and the approximate radius (in pixels) of a circle surrounding the abnormality. Site anomalies are classified according to the type of anomaly found (circumscribed masses, calcification, distortions of circumscribed masses, asymmetries, architectural distortions and other well-defined masses). Table 1 shows the column details.

Table 1. MIAS database description

- Preprocessing

Data augmentation: Rotating the images 8 degrees 45 times. These transformations are one of the general principles of training scalable networks with small data sets. Finally, all images were resized to 224×224 to run in the CNN models.

3.2 Comparative study between CNN architectures and classification algorithms

This study consists of two essential parts, starting with feature extraction from mammography images by applying different CNN architectures. Then, the classification of the feature vector obtained in the previous step using different machine learning algorithms to obtain the corresponding class (Figure 2).

2.png

Figure 2. Comparative study of classification algorithms

3.2.1 Feature extraction

This step consists in extracting from the mammography images the features that will be used in the classification step. Indeed, this operation is a difficult task that needs a high expertise level and requires a lot of work and time. In addition, the solutions may not be representative because the characteristics obtained are problem specific. In deep neural network models, feature extraction is done automatically by the layers of the CNN. Indeed, the basic features (such as: edges and contours) are generated by the lower layers, while information about the color and shape of the image are obtained by the intermediate layers. In addition, these networks include a fully connected layer for classification. In this work, this fully connected layer is removed in order to obtain a feature vector as output of the network. Hence, the role of the CNN is limited to extracting features automatically only (see Figure 2). The selected architectures to perform the experiment are EfficientNetB7, vgg19, InceptionV3, ResNet152, DenseNet201 and MobileNetV2. The pre-trained weights that were performed to the networks were trained on the ImageNet database.

3.2.2 Features classification

This experiment consists of six cases.

The first case implements the Xgboost classifier on the mammogram features extracted from the CNN model. The parameters applied in this case are the default parameters.

The second case implements the SVM classifier on the CNN features of the mammograms. The SVM maps the input vector of data to a higher dimensional feature space where a maximum margin hyperplane is built. This case uses the default parameters of SVM.

The third case utilizes the fully connected (dense) layer with the Softmax activation function on the CNN in order to distinguish between positive and negative mammograms.

The fourth case applies the Random Forest (RF) classifier on the mammogram features extracted from the CNN. RF is a learning method that applies the average prediction score of a single tree in a combination of multiple decision trees. The settings applied in RF are the default settings. The fifth case applies the AdaBoost classifier in addition to the CNN features. The AdaBoost algorithm is a sequential boosting ensemble learning method in which each weak classifier is modified based on the instances misclassified by all previous classifiers. Its final Decision is the weighted sum of the scores of the results of a combination of the final classifiers. In AdaBoost algorithm the decision tree is applied as the base classifier and the parameters applied are the default parameters. The sixth case applies the K nearest neighbors (KNN) to classify the target points (unknown class) based on their distance from the points making up a training sample.

3.3 Optimization Xgboost

3.3.1 Training process of Xgboost

After the feature extraction step, several classifiers such as Random Forest, Support Vector Machine, etc. were performed for the classification task. But in this paper, the best results were found with Xgboost model as the classifier for the problem. The description of the XGBoost [31] core is as follows:

Let $\mathrm{D}=\left\{\left(\mathrm{x}_{\mathrm{i}}, \mathrm{y}_{\mathrm{i}}\right)\right\}\left(|\mathrm{D}|=\mathrm{n}, \mathrm{x}_{\mathrm{i}} \in \mathbb{R}^{\mathrm{m}}, \mathrm{y}_{\mathrm{i}} \in \mathbb{R}^{\mathrm{n}}\right)$ describes a dataset with $\mathrm{n}$ examples and $\mathrm{m}$ features.

Eq. (1) consists of a tree boosting model output $\hat{\mathrm{y}}_{\mathrm{i}}$  with K trees [32].

$\hat{\mathrm{y}}_{\mathrm{i}}=\sum_{\mathrm{k}=1}^{\mathrm{K}} \mathrm{f}_{\mathrm{k}}\left(\mathrm{x}_{\mathrm{i}}\right), \mathrm{f}_{\mathrm{k}} \in \mathrm{F}$                 (1)

where, $\left.\mathrm{F}=\{\mathrm{f}(\mathrm{x})) \omega_{\mathrm{q}}(\mathrm{x})\right\}\left(\mathrm{q}: \mathbb{R}^{\mathrm{m}} \rightarrow \mathrm{T}, \omega \in \mathbb{R}^{\mathrm{T}}\right)$  is the spatial regression or classification trees. Each f_k splits a tree into structure part q and leaf weights part ω. T is the number leaves in the tree.

To minimize the following objective function [32], the tree model functions f_k can be learned:

$\mathfrak{L}^{(\mathrm{t})}=\sum_{\mathrm{i}=1}^{\mathrm{n}} \mathrm{l}\left(\mathrm{y}_{\mathrm{i}}, \hat{\mathrm{y}}_{\mathrm{i}}^{(\mathrm{t}-1)}+\mathrm{f}_{\mathrm{t}}\left(\mathrm{X}_{\mathrm{i}}\right)\right)+\Omega\left(\mathrm{f}_{\mathrm{t}}\right)$                      (2)

where, the term l  in Eq. (2) is a training loss function which measures the distance between the prediction $\hat{y}_i$ and the object y_i. And Ω represents the penalty term of the tree model complexity [32].

$\Omega\left(f_t\right)=\gamma \mathrm{T}+\frac{1}{2} \lambda \sum_{j=1}^{\mathrm{T}} \omega_{\mathrm{j}}^2$                      (3)

Xgboost approximates Eq. (2) using the second order Taylor expansion and the final objective function at step t can be simplified as [32]:

$\mathfrak{L}^{(\mathrm{t})} \simeq \sum_{\mathrm{i}=1}^{\mathrm{n}} \mathrm{l}\left(\mathrm{y}_{\mathrm{i}}, \hat{\mathrm{y}}_{\mathrm{i}}^{(\mathrm{t}-1)}+\mathrm{g}_{\mathrm{i}} \mathrm{f}_{\mathrm{t}}\left(\mathrm{X}_{\mathrm{i}}\right)+\frac{1}{2} \mathrm{~h}_{\mathrm{i}} \mathrm{f}_{\mathrm{t}}^2\left(\mathrm{X}_{\mathrm{i}}\right)\right)+\Omega\left(\mathrm{f}_{\mathrm{t}}\right)$                     (4)

where, g_i and h_i are the first and the second order gradient statistics on the loss function [32].

$\mathrm{g}_{\mathrm{i}}=\partial_{\hat{y}_{\mathrm{i}}^{(\mathrm{t}-1)}} \quad\mathrm{l}\left(\mathrm{y}_{\mathrm{i}}, \hat{\mathrm{y}}_{\mathrm{i}}^{(\mathrm{t}-1)}\right), \mathrm{h}_{\mathrm{i}}=\partial_{\hat{y}_{\mathrm{i}}^{(\mathrm{t}-1)}} \quad\mathrm{l}\left(\mathrm{y}_{\mathrm{i}}, \hat{\mathrm{y}}_{\mathrm{i}}^{(\mathrm{t}-1)}\right)$                      (5)

After removing the constant term and expanding Ω, Eq. (4) can be rewritted as [32]:

$\widetilde{\mathfrak{L}^{(t)}}=\sum_{\mathrm{i}=1}^{\mathrm{n}}\left[\mathrm{g}_{\mathrm{i}} \mathrm{f}_{\mathrm{t}}\left(\mathrm{X}_{\mathrm{i}}\right)+\frac{1}{2} \mathrm{~h}_{\mathrm{i}} \mathrm{f}_{\mathrm{t}}^2\left(X_{\mathrm{i}}\right)\right]+\gamma \mathrm{T}+\frac{1}{2} \lambda \sum_{\mathrm{j}=1}^{\mathrm{T}} \omega_{\mathrm{j}}^2=\sum_{j=1}^T\left[\left(\sum_{i \in l_j} g_i\right) \omega_j+\frac{1}{2}\left(\sum_{i \in l_j} h_i+\lambda\right) \omega_j^2\right]+\gamma T$                    (6)

The term weight $\omega_j^*$ of leaf $\mathrm{j}$ for a fixed tree structure $\mathrm{q}(\mathrm{x})$ can be achieved by performing the following equation [32]:

$\omega_{\mathrm{j}}^*=-\frac{\sum_{\mathrm{i} \in \mathrm{l}_{\mathrm{j}}} \mathrm{g}_{\mathrm{i}}}{\sum_{\mathrm{i} \in \mathrm{l}_{\mathrm{j}}} \mathrm{h}_{\mathrm{i}}+\lambda}$                      (7)

After replacing $\omega_j^*$  into Eq. (6) [32]:

$\widetilde{\mathfrak{L}^{(t)^*}}=-\frac{1}{2} \sum_{j=1}^T \frac{\left(\sum_{i \in l_j} g_i\right)^2}{\sum_{i \in l_j} h_i+\lambda}+\gamma \mathrm{T}$                      (8)

where,

T: the number of leaf nodes,

And γ: the coefficient of this term,

Eq. (8) describes the scoring function to measure the tree structure q(x) and to identify the optimal tree structures for classification. However, it is impractical to find all possible q-trees in practice.

3.3.2 Based optimizing process of Xgboost with CS

• The cuckoo search algorithm

Motivated by the peculiar lifestyle of cuckoo species and Levy flight, Yang and Deb [20] presented Cuckoos lay their eggs in the nests of other birds when the host birds abandon the nest unattended. In reality, some of these eggs are similar to the host bird's eggs, hatch, and develop into adult cuckoos. If the host birds detect that these eggs are not their own, they reject the foreign eggs or leave the nest and find a new location to build a novel nest. The cuckoo egg provides a novel solution among all the eggs in a nest. The goal of the CS algorithm is to apply the new potentially better solutions to substitute for the less good solutions in the nests. The CS algorithm is based on the following three rules [33, 34]:

1) Each cuckoo lays only one egg (one solution) at a time and deposits them in a randomly chosen nest.

2) Among these nests, the best nest with high quality eggs will be passed on to the next generation.

3) In case the host bird finds a foreign egg in its nest with a probability, it leaves its nest and builds another nest in another place.

The CS algorithm updates the locations of bird nests based on the above three rules. Its path of research can be expressed as follows:

$\mathrm{X}_{\mathrm{i}}^{\mathrm{t}+1}=\mathrm{X}_{\mathrm{i}}^{\mathrm{t}}+\alpha \oplus \mathrm{L}$                   (9)

where, $X_i^t$ denotes at iteration t the position of the i^th nest. The product ⨁ defines the multiplication by the input, and α is the step size, which follows to a normal distribution. L is the Levy random search path defined as follows:

$\mathrm{L}=0.01 \times \frac{\mu}{|v|^{1 / \beta}} \times\left(\mathrm{g}_{\text {best }}-\mathrm{X}_{\mathrm{i}}^{\mathrm{t}}\right)$                   (10)

where, g_best describes the best nest. When µ, ν follow a normal distribution, $\mu \sim \mathrm{N}\left(0, \delta_\mu^2, \nu \sim \mathrm{N}\left(0, \delta_\mu^2\right)\right.$, where, $\beta=1.5$ :

$\left\{\begin{array}{c}\delta_\mu=\left\{\frac{\Gamma(1+\beta) \sin \left(\frac{\pi \beta}{2}\right)}{\Gamma\left[\frac{(1+\beta)}{2}\right] \beta 2^{\frac{\beta-1}{2}}}\right\}^{\frac{1}{\beta}} \\ \delta_v=1\end{array}\right.$                   (11)

Compared to other metaheuristic approaches, the CS algorithm has two advantages. The first is that it can more effectively maintain stability between the local search strategy and the global search strategy. The second is that it has only two hyperparameters (the egg detection probability p_a and the population size N). For N is fixed, p_a checks the balance between local search and random search. Since the CS algorithm has less parameter than other metaheuristic algorithms, it is significantly better [20].

• Xgboost hyperparameters optimization based on Cuckoo Search (CS)

The categories of the parameters of the Xgboost algorithm are: general parameters, booster parameters, and learning target parameters. In this study, only six key parameters are concerned which are: learning rate (eta), maximum tree depth (max-depth), minimum leaf weight (min-child-weight), gamma, subsample and col-sample-bytree. The details of these parameters are presented in Table 2.

The process of the CS-Xgboost algorithm is described in the flowchart in Figure 3. It is applied to optimize the Xgboost parameters (eta, max-depth, min-child-weight, gamma, sub-sample and col-sample-bytree) as follows:

Set up the cuckoo search algorithm and define the number of nests, N, the probability parameters, pa, the maximum number of iterations, tmax, and the ranges of (eta, max-depth, min-child-weight, gamma, sub-sample, and col-sample-bytree). The initial Xgboost parameters are determined by the following function:

Table 2. Key parameters of the Xgboost algorithm

3.png

Figure 3. The process of the CS algorithm for Xgboost parameter selection

$\mathrm{x}=\mathrm{L}+\mathrm{rand} *(\mathrm{U}-\mathrm{L})$               (12)

The dimensions are calculated according on the number of the optimized parameters and the positions and velocities of the particles are randomly initialized. Each particle is located by the position attribute as a 6-dimensional vector and its range in the search space. Each dimension corresponds to several parameters of Xgboost initialized with different value ranges. At time t the position vector of the i-th particle is defined as follows:

$\begin{aligned} & \mathrm{P}_{\mathrm{i}(\mathrm{t})} \\ & =\left[\mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {eta }}, \mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {max-depth }}, \mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {min }_{\text {weight }}^{\text {chil }}}, \mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {gamma }}, \mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {subsample }}, \mathrm{P}_{\mathrm{i}(\mathrm{t})}^{\text {colsample }_{\text {bytree }}}\right]\end{aligned}$                  (13)

The position vector is then affected to the appropriate parameters of the model and the initial value of the fitness function is initialized by the model performance on the training set. At time t, the fitness value of the i-th particle is:

$\mathrm{F}_{\mathrm{i}(\mathrm{t})}=\left(\mathrm{P}_{\mathrm{i}(\mathrm{t})} \rightarrow\right.$ Xgboost $\left.\left.\right|_{\text {trainingset }}\right)$               (14)

For each nest, compute and verify the fitness value and find the current best solution and save the maximum fitness value and its position. For the i-th particle at time t, its individual optimum is defined as follows:

Pbest $_{i(t)}=\max \left(\mathrm{F}_{\mathrm{i}(\mathrm{j})}\right), \quad 0 \leq \mathrm{j} \leq \mathrm{t}$               (15)

Record the best solution of the last generation and compute the position of the other nests. Then check the fitness value of the novel position. Let m particles at time t, the global optimum is defined as:

Gbest $_{(t)}=\max \left(\right.$ Pbest $\left._{\mathrm{k}(\mathrm{t})}\right), \quad 1 \leq \mathrm{k} \leq \mathrm{m}$               (16)

Calculate the random number that corresponds to the probability of detecting the egg. Verify it with p_a, if random >p_a, calculate the wireless nest position to get a novel set of positions. Identify out the best nest position; stop criterion when the maximum iteration is obtained, and get the best position to reach the optimal value of the parameter; otherwise, go back to step 2.

This research facilitates early diagnosis of mammography to prevent negative impact in these remote areas. The developing of algorithms in this area can be very helpful in achieving better health care services. This paper presents how to use a convolutional neural network with classification algorithms to achieve better medical image identification performance of learning algorithms in the first step, and optimize this new model with the metaheuristic algorithm in the second step. This study observes the performance of different CNN models pre-trained with various classifiers, and then, based on the statistical results, selects the EfficientNetB7 model for the feature extraction step and the Xgboost classifier for classification. Indeed, it is demonstrated that the Xgboost classifier is more suitable for the CNN architecture with a large number of residual layers. The reason is that the Xgboost algorithm performs the weak tree better, which increases the accuracy for each CNN layer. It is more appropriate to combine the CNN architecture that has a low number of layers like VGG-19 and ResNet-50 with the SVM classifier. Nevertheless, it is interesting to achieve the CNN architectures that have many layers like EfficientNetB7 with the Xgboost classifier. In a second step, the CS method is applied with the Xgboost classifier to optimize the hyperparameters. The CS algorithm is applied to choose the appropriate Xgboost parameters to efficiently prevent the phenomenon of "overfitting" or "underfitting" of Xgboost, thus boosting the prediction efficiency. Experiments demonstrate that the proposed model performs well in both datasets. The results of the EfficientNetB7-CS-Xgboost model were compared with those of previous studies, and in particular with the EfficientNetB7-PSO-Xgboost and CNN-Xgboost models over four measures, with the result that the EfficientNetB7-CS-Xgboost model has better accuracy and higher effect than the EfficientNetB7-PSO-Xgboost and EfficientNetB7-Xgboost models. It can be said that this study has improved the efficiency of the model compared to other models (PSO). This novel approach can give, in practice, a novel modeling approach for mammographic tumor classification processing. This model can also be an effective and accurate decision support tool for radiologists and physicians. Nevertheless, the probability parameter in CS is fixed, which can influence the convergence of the algorithm.