A Hybrid Approach for Multiclass Brain Tumor Classification: Ensemble Convolutional Neural Networks and PSO Optimization

Brain tumors are abnormal masses of neoplastic tissue characterized by uncontrolled cell multiplication and growth, unchecked by the mechanisms regulating normal cell division. They can appear at any stage of life [1] and are among the significant diseases affecting the human central nervous system (CNS) [1]. Brain tumors are regarded as among the devastating illnesses that profoundly impact the human body [2]. The majority of brain tumors lack a clear cause [3]. They can be classified as benign or malignant. Benign tumors, which are noncancerous, do not spread and do not invade other parts of the body. In contrast, a malignant tumor is a cancerous tumor, characterized by rapid growth and the possibility of spreading to other areas of the body. Over 120 types of tumors in CNS have been reported by the World Health Organization (WHO) [4].

Three main types of brain tumors exist:

Pituitary tumors, located in the pituitary gland and responsible for hormone production related to growth and other glands. Meningioma tumors, commonly benign and characterized by slow growth, surround the meninges, exhibiting a higher occurrence among women than men. The incidence rates of pituitary and meningioma tumors in clinical practice are approximately 12% and 15-20% respectively [5]. Gliomas, originating from glial cells or the supportive tissue surrounding nerve cells, account for 45% of tumors [6].

Detecting tumors early is key to timely intervention, improving treatment results, and potentially saving lives.

Medical imaging is now a fundamental tool for diagnosis and intervention, offering visual insights into the functionality of organs and tissues. The increasing use of advanced imaging technologies such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) has created a pressing need for automated processing of scanned data [7]. These technologies produce accurate images, allowing doctors to accurately detect tumors and initiate appropriate treatment plans. MRI is preferred over CT scans because it has beneficial qualities and does not impact the human body [8].

Diverse approaches have been applied to medical databases, covering MRI images of brain tumors and tumors in other regions of the human body [9].

Artificial intelligence (AI) has experienced significant progress with the emergence of deep learning techniques, especially Convolutional Neural Networks (CNNs).

In image processing, CNNs stand out as the most commonly utilized and effective algorithm in Deep Learning [10], revolutionizing how computers analyze and interpret visual data. CNNs have shown outstanding effectiveness in various computer vision applications, such as image classification, facial recognition, and object detection. These neural networks use convolutional layers to automatically learn and extract features from images, allowing them to capture diverse structures and patterns. For increased flexibility and customization, ultimately leading to improved performance and outcomes in image analysis, classification, and segmentation tasks, various modified pretrained networks on large datasets like ImageNet have been utilized. These pretrained networks expedite the training process and frequently lead to better performance on subsequent tasks, making them a popular choice in computer vision applications [11]. Modifications to pretrained architectures often involve adding or modifying layers, tuning hyperparameters, or changing activation functions.

A single pretrained CNN model might have limitations in capturing the complete variability and complexity of the data. It may not be easily adaptable to different tasks or domains without extensive retraining or fine-tuning, processes that are often time-consuming and resource-intensive. These limitations may result in the model not encompassing all relevant features or patterns in the data, leading to decreased accuracy in classification and generalization performance on unseen data.

This article proposes an ensemble fusion method that concatenates three different CNN models, to classify brain MR images into four categories: Pituitary tumor, Glioma tumor, Meningioma tumor, and Normal brain. The three pre-trained models serve as deep feature extractors from images. Subsequently, we combine the features extracted from these neural networks to create a synthetic feature, and the dominant features are selected using the PSO algorithm. Finally, a linear kernel SVM classifier is employed for classification.

The major difficulty associated with using linear SVM classifiers is the need to select an optimal value for the penalty parameter C. This tuning parameter C balances the tradeoff between expanding the margin and minimizing errors in the machine learning problem. In this experiment, we chose to use PSO to optimize the parameter C and also to search for relevant features that maximize the score on the testing set.

Our goal is to achieve superior performance in multiclass brain tumor classification by enhancing accuracy and generalization capabilities.

The contributions of this study are briefly:

•Combination of CNN and linear SVM classifier for classifying brain MR images into four categories: Meningioma tumor, Pituitary tumor, Glioma tumor, and Normal brain;

•Fusion of three deep CNNs: VGG19, GoogLeNet, and ResNet50 achieves an even higher accuracy than individual models;

•Employing the PSO algorithm to enhance our ensemble model's performance with a reduced feature subset.

The rest of the document is organized as follows: Sect Ⅱ details the methodology employed to design the ensemble deep CNN model. The results obtained from the proposed method and comparisons with existing studies are presented in Sect Ⅲ. Sect Ⅳ concludes the paper.

3.1 Dataset

The dataset employed in this experiment is comprised of data from three different sources: figshare, Br35H, and the SARTAJ dataset, made publicly available by Masoud Nickparvar [27].

This merged dataset comprises 7023 MRI images of the human brain. For our study, we used 5712 images, classified into four categories: pituitary (1457 images), meningioma (1339 images), glioma (1321 images), and no tumor (1595 images). The no tumor images were sourced from the Br35H dataset. The image sizes vary within the dataset; therefore, the images were resized to 224×224 pixels to match the input size requirements of the three deep learning networks used.

The dataset was split into two groups: 80% allocated for training and 20% reserved for testing. Sample images collected from the database are displayed in Figure 1.

1.png

Figure 1. Sample brain MRI images from the dataset used in this study [27]

3.2 Deep neural network models

This study employs three CNN models: VGG19, ResNet50, and GoogLeNet to sequentially extract feature vectors from brain MR image.

3.2.1 VGG19

VGG19 is a pre-trained CNN model initially trained on 1000 classes from the ImageNet dataset [28]. It accepts 224×224-pixel images as input and comprises of 19 weight layers: 16 convolutional layers followed by 3 fully connected layers, totaling approximately 144 million parameters.

The convolutional layers use small 3×3 convolutional filters, starting at 64 in the first layer and doubling in number after each max-pooling layer, up to a maximum of 512 filters. Its architecture has demonstrated exceptional performance across a various image classification task [29]. Despite its simplicity, VGG19 tends to generalize well when provided with sufficient training data [30].

3.2.2 ResNet50

ResNet50 is a CNN that is 50 layers deep. It was trained on 1.28 million training images across 1000 classes and has an image input size of 224 by 224. The architecture of ResNet50 consists of four key parts: convolution layers for feature extraction, convolution blocks comprising multiple convolution layers with normalization and activation functions for high-level feature extraction, residual blocks that provide shortcut connections to mitigate the vanishing gradient problem, and fully connected layers that make predictions based on the extracted features. The residual layers present in ResNet50 are crucial for transferring large gradient values to their prior adjacent layers [31]. ResNet50 has indeed demonstrated remarkable efficiency in solving the vanishing gradient problem compared to previous methods. This problem occurs due to the iterative multiplication of derivative values in the first layers, which causes these values to decrease and subsequently reduces network accuracy in deep learning [32, 33].

3.2.3 GoogLeNet

GoogLeNet is a deep learning model capable of classifying patterns among approximately 1000 images. It comprises of 22 layers and incorporates 9 inception modules. These modules enable the network to capture features at different scales and resolutions, improving its capability to recognize diverse patterns in images. The 1×1 convolutions at the module's bottom reduce the number of inputs, leading to a dramatic decrease in computational cost [34]. Furthermore, GoogLeNet utilizes the global average pooling layer rather than a fully connected layer, which decreases the number of parameters in the network. In general, it stands out as an efficient and precise deep learning architecture, significantly influencing the development of subsequent models in the field [35].

2.png

Figure 2. Summary of the suggested framework for categorizing brain tumors using an ensemble of VGG19, ResNet50, and Inception V1 with an SVM classifier

Integrating VGG19, GoogLeNet, and ResNet50 leverages their unique feature extraction strengths: VGG19 captures fine-grained textures and tumor edges, essential for identifying subtle tumor patterns; GoogLeNet's inception modules enhance feature diversity by efficiently extracting both local and global information; and ResNet50's skip connections facilitate deeper feature learning, preventing vanishing gradients and improving hierarchical feature extraction. This ensemble approach provides a more robust feature representation, mitigating individual model weaknesses and enhancing classification performance.

The proposed approach involves fusing the outputs of the fully connected layers of VGG19, ResNet50, and GoogLeNet into a synthetic feature set, as illustrated in Figure 2. This results in a set comprising 3000 features, with 1000 features contributed by each model. The dominant features selected using the PSO algorithm are then fed into a linear SVM classifier for classifying 4 classes.

3.3 PSO

In this subsection, we describe the functional principles of the feature selection model included in this study, specifically PSO.

PSO is a robust and efficient algorithm that mimics the behavior of birds searching for food [36]. It has proven successful in addressing search and optimization problems across a range of domains.

In PSO algorithm, each solution is represented as a particle, with the swarm comprising all these particles. Each particle's movement is influenced by three factors: its current velocity, its best position found thus far (pbest), and the swarm’s best position thus far (gbest). These factors guide the particles in exploring the search space effectively and facilitate convergence toward the optimal solution. Initially, particles are randomly positioned and assigned random velocities. Their fitness is evaluated, and pbest and gbest are updated if the new fitness values are better than previously recorded values. Each particle's velocity and position are updated iteratively based on the following equations:

$v_{t+1}=\omega v_t+c_1 r_1\left(\right.$ phest $\left._t-x_t\right)+c_2 r_2\left(\right.$ gbest $\left._t-x_t\right)$      (1)

$x_{t+1}=x_t+v_{t+1}$     (2)

where, $t$ refers to number of iterations. $x_t$ indicates the particle's position at time $\mathrm{t}, v_t$ denotes the current velocity at time $t, p b e s t_t$ is the personal best position, gbest $t_t$ is the global best position in the swarm thus far, $\omega$ denotes the inertia weight, $r_1$ and $r_2$ are uniformly distributed random numbers between 0 and 1 , and $c_1$ and $c_2$ are cognitive and social coefficients. This process is repeated until a predefined stopping criterion, such as a satisfactory fitness level or a maximum number of iterations, is met.

PSO is easy to implement and requires fewer parameter adjustments [37]. It provides stable results in parameter optimization compared to other methods [38]. PSO is a crucial element in ML for SVM parameter adjustment, optimizing weights in back-propagation neural networks, and achieving better results compared to traditional backpropagation methods. PSO effectively balances exploration and exploitation, making it suitable for high-dimensional optimization tasks such as feature selection and hyperparameter tuning [39]. This balance enables PSO to efficiently navigate large search spaces. Its efficiency in navigating search spaces via particle position updates reduces computational costs compared to exhaustive methods such as grid search. PSO also converges more quickly than genetic algorithms by avoiding complex genetic operations such as crossover and mutation. Unlike probabilistic models such as Bayesian optimization, which often face challenges in high-dimensional spaces due to increased statistical and computational complexity, PSO does not rely on gradient information. These features make PSO a valuable tool for various optimization tasks, particularly in complex search spaces.

The role of PSO is to determine the values the penalty parameter C and fs: a real number whose binary representation is used as a mask to select a subset of features which maximize the value of the objective function.

In this experiment, the PSO algorithm parameters are set as follows:

Initial population: In the PSO algorithm, a randomly initialized population of potential solutions was created, with the initial population size set to 30.

Fitness function: The 10-fold-Cross-Validation method will be utilized to evaluate the performance of the SVM model and the wrapper approach will be used to select the most important feature subset that maximize the fitness function value. A vector composed of 3000 bits, which is the binary representation of fs, is used as a mask. This mask includes all bits with a value of 1, representing the features used in the training phase, and ignores all features masked by bits set to 0. Feature subset selection and the penalty parameter C should be optimized simultaneously to find the best solution. In this study, overall classification accuracy is utilized as the fitness function.

Maximum generations: The total number of generations was set to 50.

Termination criterion: Can be either a lack of improvement in the population over a specified number of generations or reaching an upper limit on the number of generations.

In this experiment, the tested ranges of values for parameters C and fs were set to [0.001, 10] and [0.1, 100], respectively.

3.4 SVM based classification

SVM is a supervised learning algorithm used for solving classification and regression problems. It was enhanced by Cortes and Vapnik in 1995 for binary classification. The algorithm was later developed and generalized for multiclass and nonlinear datasets [40].

This classifier seeks to identify the optimal separating hyperplane that achieves the maximum margin. The margin is defined as the distance between the hyperplane and the closest data points, termed support vectors. The hyperplane is defined by the equation:

$w^T x_i+b=0$     (3)

The weight vector w and bias term b are parameters that define the decision boundary. These parameters are calculated as follows:

$\left\{\begin{array}{l}y_i\left(w^T x_i+b\right) \geq 1-\xi_i \text { if } y_i=+1 \\ y_i\left(w^T x_i+b\right) \leq 1-\xi_i \text { if } y_i=-1\end{array}\right.$      (4)

The last two constraints can be combined into:

$y_i\left(w^T x_i+b\right) \geq 1-\xi_i$      (5)

where, $\xi_i$ is the slack variable. The optimal hyperplane can then be found as follows:

$\Phi(w, \xi)=\frac{1}{2} w^T w+C \sum_{i=1}^N \xi_i$     (6)

The parameter C > 0 regulates the balance between maximizing the margin and minimizing classification errors.

It significantly impacts the efficiency and performance of the SVM classifier [41].

Using the method of Lagrangian multiplier, the solution the optimal classification hyperplane can be formulated as follows:

$Q(\alpha)=\sum_{i=1}^N \alpha_i-\frac{1}{2} \sum_{i=1}^N \sum_{j=1}^N \alpha_i \alpha_j y_i y_j K\left(x_i, x_j\right)$     (7)

where, $\alpha_i=\left(\alpha_1, \alpha_2, \ldots, \alpha_N\right)$ is the vector of Lagrange multipliers, most multipliers satisfy the condition $\alpha_i=0$ with only the sample for which $\alpha_i \neq 0$ being considered a support vector. $K\left(x_i, x_j\right)$ is the kernel function in SVMs that maps data, which is not linearly separable, into a higher-dimensional feature space where linear separation may become possible. Currently, the most widely used SVM kernel functions are linear, polynomial, radial basis function (RBF), and sigmoid, among others [42]. In this study, the linear SVM classifier is integrated into the final layer of the fully connected CNN to enhance efficiently fit the date length to turn the kernel [33]. Figure 3 provides a detailed depiction of the process employed by the proposed model.

3.png

Figure 3. Flowchart of the proposed ensemble approach for brain image classification

3.5 Performance metrics

In this research, we evaluated the performance of each model using accuracy, precision, recall, and F1-score metrics. These classification performance measures are derived from the four values of the confusion matrix: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). Additionally, we plotted the AUC (Area Under the Curve) for the ROC (Receiver Operating Characteristics) curve. The formulas for these evaluation metrics are given in Eqs. (8) to (12).

Accurracy $=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$     (8)

Sensitivity (Recall) $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$      (9)

Specificity $=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$      (10)

Precision $=\frac{T P}{T P+F P}$     (11)

F1 - Score $=2 \times \frac{\text {Precision} \times \text {Recall}}{\text {Precision}+ \text {Recall}}$     (12)