Divya Mohan*
| Ulagamuthalvi Venugopal | Nisha Joseph | Kulanthaivel Govindarajan
© 2023 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Medical Image Segmentation is a process of segmenting abnormalities from normal tissues. Due to the increasing growth of deep learning, there are various deep network models used to segment 3D medical images. Recently, U-Net and V-Net are used to segment 3D medical images. But these networks suffer from high computation burden. The objective of this paper is minimizing the computation time by reducing the input data. Initially, the 3D slices are reduced by taking the average of few slices (Inter-slice reduction). Then, only the tumor area is segmented using detection window (Intra-slice reduction). The reduced 3D Magnetic Resonance Imaging (MRI) data was fed as input to UNet with Long Short Term Memory (LSTM) layers for segmentation and classification. BRATS 2017 and BRATS 2018 are tested by proposed method of dataset. It achieves 96.24% accuracy, 90.84% Dice Score Coefficient (DSC) on BRATS 2017 dataset and 92% accuracy and 88.88% DSC on BRATS 2018 dataset in 12 and 10 seconds respectively. The proposed method is compared with some recent methods. It achieved reasonable gain in computation time with negligible loss in other metrics.
tumor, magnetic resonance imaging, convolutional neural network, dense block
Since several decades brain tumor has become one of the most dreadful diseases in the world. There are several aids for curing brain tumor. The abnormal cells in brain are termed as Gliomas. It has been splited into Low Grade Glioma (LGG) and High Grade Glioma (HGG). Initially tumor is in less area (LGG) and grows severe (HGG) which leads to death. Though the disease is severe, it can be diagnosed when it is treated earlier. Medical tools are growing day by day which increases the life of person. An effective tool to scan brain is the MRI which scans using magnetic field and computer-generated radio waves. This helps the physician in identifying tumor region and its severity.
The MRI images are available in three dimensions similar to our brain. The BRATS dataset consists of four different modalities for a single patient. Classifying tumor in the multimodal 3D data has several challenges. The position, volume and contour of Tumor vary brain to brain. Automatic segmentation of tumor should identify the Tumor area very accurately. The computer aided automatic segmentation depends on deep learning nowadays. As deep learning is a supervised learning, the system has to be trained with more number of data.
As already mentioned, the brain image of a single patient consists of four modalites and each modality has three dimensions. CNN suffers high computation time to learn about such huge data. Due to two important reasons the deep learning model used for segmentation suffers from high computation time. They are:
A novel method is developed for reducing MRI data dimension and for improving the efficiency of the deep learning models. The inter-slice and intra-slice reduction are done to reduce the dimension. The major innovations regarding this work are:
The structure of the paper is: Section 2 explains very recent methods of segmentation, classification of brain Tumor. Section 3 elaborates the method proposed with all its phases. Section 4 discusses the datasets used for experiments. Section 5 demonstrates the proposed method with some experiments and analysis. It also analyses the results with some comparison. Section 6 concludes the work.
Computer has its important footsteps in many healthcare applications [1]. As brain Tumor segmentation has many of its researches based on deep learning, this section discusses some recent researches related to brain Tumor segmentation and classification. For automated segmentation of granular prostate cancer magnetic resonance images, a 3D AlexNet approach [2] has been implemented; network performance is compared by generic network ResNet 50, Inception -V4. 3D MRI pictures are first sliced, with roughly 10% of the top and bottom slices being normalized since they do not include any important features [3]. Furthermore, tumor enhancement is carried out on regularized segment employing convolutional processing in various assessments with variable sizes of kernel. For glioma extraction from normalized slices, the kernel-enhanced glioma areas are subjected to a local adaptive thresholding approach.
Based on encoder-decoder architecture, 3D MRIs creates [4] a semantic segmentation network for tumor sub-region segmentation. Unlike previous NAS (Neural Architecture Search) methods, which typically searched the optimal operators in each network layer, missed a good strategy to search for feature aggregations, UXNet [5] is a novel NAS method used for 3D medical image segmentation that searches both aggregation strategies of scale-wise features and block wise operators in the encoder-decoder network.
An algorithm [6] is designed to search the optimal augmentation strategies automatically using potential data augmentation. The 3D Context Residual Network (ConResNet) [7] was developed to provide error-free three dimensional medical image segmentation.
Self-attention between nearby image patches is used in the majority of deep neural network designs. More accurate segmentations than CNNs may be produced without any convolution procedures [8].
A novel differentiable search structure has been devised using a versatile network topology for facilitating a rapid gradient-based search [9]. As a result of it, to address this type of issue a topology loss has been implemented. Furthermore, due to budget limits, the usage of GPU memory for searching three dimensional model is restricted, machine-aided predictions were used to assess the initial BraTS labels' compliance with radiologic criteria [10].
A deep learning model for glioma and stroke lesion identification is reported in Ref. [11]. This application is developed with segmentation, registration and visualization. It is referred to as AIMIS3D [12]. With sufficient training, the prediction of the prostate organ is done using YOLOV3 algorithm. The types of cancers are separated using the U-Net developed from MRI data.
The BraTS dataset was trained using Support Vector Machine (SVM) classifier. The training was performed by utilizing World Health Organization (WHO)-based radiomic features. The classifier’s behaviour is analyzed using the prediction score histogram on the lower grade population [13]. Five professional radiologists were asked to interpret BraTS images between lower and higher grade gliomas for creating a new ground truth.
The more frequent primary brain tumors key diagnostic indicators have been reviewed in clinical background information [14]. It provides the review of all resources and datasets that are publicly available for developing the novel computational tools and image biomarkers and it has a focus on the BraTS Challenge.
The AGSE-VNet [15] framework is developed for automatically segmenting brain tumor from MRI data. For each encoder the SE is added. For decoder AG is added.
A unique technique for splitting the 3D segmentation process [16] among numerous remote workstations has been developed. The principles underpinning distributed multimedia network segmentation were used to reduce the training Hidden Markov Model (HMM) segmentation computational time.
Four modalities are present in multi modal 3D input in BRATS 2018 dataset. They are T1, T1c, T2 and FLAIR. The size of each input is 240 x 240 x 155. In the phase of tumor segmentation, the four modalities are frequently referred as single modality of MRI. The whole tumor segmentation is performed better with Flair is used in whole tumor segmentation to provide better performance whereas T2 is used for segmenting core in better way. There are two ways to fuse these modalities: Early Fusion and late fusion. In early fusion, low-level characteristics are combined. In late fusion, all modalities are merged in a deep layer after CNN. Processing these images takes very long time.
The input size is minimized using preprocessing techniques. This paper identifies the important area of interest and then it is provided to the UNet with LSTM model for segmentation. Figure 1 shows the proposed system architecture. The proposed architecture comprises of Slice Trimming (Inter-slice reduction) and Region of Interest (ROI) Segmentation (Inter-slice reduction). The average processing time of LSTM is 5x times faster than CNN model [17]. Hence, LSTM layer is included to UNet model in this work.
Slice Trimming (Inter-Slice Reduction)
The correlation between nearer slices paves the way to this section. The number of slices in the BRATS dataset is 155.
The slices are reduced by taking the average of nearby 5 slices. Let the no.of.slices in the three dimensional input be:
$I=\left\{S_1, S_2, \ldots, S_n\right\}$ (1)
where, $n=155$.
Now the trimmed slice will look like:
$I_t=\left\{S_{a 1}, S_{a 2}, \ldots, S_m\right\}$ (2)
where,
$\begin{gathered}S_{a 1}=\frac{S_1+S_2+S_3+S_4+S_5}{5}, S_{a 2}=\frac{S_6+S_7+S_8+S_9+S_{10}}{5}, \ldots, \\ S_m=\frac{S_{151}+S_{152}+S_{153}+S_{154}+S_{155}}{5}\end{gathered}$
The value of $m=\frac{155}{5}=31$. The number of slices is reduced from 155 to 31. The value of 5 is taken, because 155 is divisible by 5 and 31.
Figure 1. System architecture
ROI Segmentation (Intra-slice Reduction)
The next step is to find the ROI. In brain tumor segmentation, the ROI denotes the tumorous area in the brain. The two important challenges in identifying ROI are:
1. The location
2. The size of the tumor
The first challenge is resolved by identifying the center pixel location of the tumor area from the FlAIR image. The second challenge is resolved by having common detection window for all images. The algorithm for identifying ROI is given in Algorithm 1.
|
Algorithm 1: ROI Segmentation |
|
Input: Trimmed 3D MRI Data, Detection Window Output: ROI Steps: 1. For each Flair in {$T_1, T_{1 c}, T_2$, Flair} 1.1 Find the histogram equalized image $I_{\text {hist_eq }}$. 1.2 For each pixel I in $I_{\text {hist_eq }}$ $D=\left\{\begin{array}{cc}1 & i>250 \\ 0 & \text { Otherwise }\end{array}\right.$ 1.3 End 1.4 Find the minimum ( $x_{\min }, y_{\min }$ ) coordinate whose pixel value is 1 in D. 1.5 Find the maximum ( $x_{\max }, y_{\max }$ ) coordinate whose pixel value is 1 in D. 1.6 Center $c=\left\{\frac{x_{\min }\,\,+ \,x_{\max }}{2}, \frac{y_{\min }\,\,+\,y_{\max }}{2}\right\}$ 1.7 Crop $T_1, T_{1 c}, T_2$, Flair to the size of detection window around the center. 2. End |
Identifying Center Pixel Location: To find the center pixel location of the tumor area, the histogram equalized image $I_{\text {hist }_{e q}}$ of flair is calculated. From $I_{\text {hist }_{e q}}$, a binary image D is created using
$D=\left\{\begin{array}{cc}1 & i>T \\ 0 & \text { Otherwise }\end{array}\right.$ (3)
In the above equation, i is the pixel value and T is a threshold. By looking the values in $I_{\text {hist }_{e q}}$, it is observed that the pixel values of tumor area are above 250. Hence the threshold value is set to 250. The center pixel location is identified by calculating the minimum and maximum location of white pixels in D. The average of these locations is the center location.
Feedback UNet with Convolutional LSTM Layer
Mainly for brain tumor segmentation UNet provides efficient results. In UNet model convolutional LSTM is used as a replacement of recurrent convolutional layer [18], which dealt with sequential data. The feedback UNet with convolutional LSTM layer architecture is shown in Figure 3. In the first round the cell preserves all the features. In second round features are retrieved using maintained features.
Three major changes are made in UNet. The resultant UNet is fed as the input. It is first modification. Output concatenation is second modification. The third modification is the utilization of convolution.
The region of interest extracted from the intra-slice reduction is the input to U-Net. The probability map for all the classes are obtained from the final layer of the softmax function. The probability maps produce the results of segmentation. In this network model, the inputs are the image and its probabilistic maps of the network. These probabilistic maps are acquired for each class at the final layer.
Once again, the inputs at the first round are fed into the network. The probability maps generated during second round are used as the result of final segmentation.
Hence the ROI size is made common for all images. But the location depends on the input image. The full workflow of the proposed method is shown in Figure 2.
After finding the ROI in each modal of 3D MRI, it is given LSTM network model. It segments and classifies the labels of tumors.
Figure 2. Flow of the proposed method
The performance and efficiency of the proposed method is tested and analyzed on BRATS 2017 and 2018 datasets [19]. There are 431 brain tumor cases in 2017 dataset and 476 brain tumor cases in 2018 dataset. There are 146 training samples in both the datasets. The testing samples of 2017 and 2018 dataset are 146 and 191 respectively. Some examples of BRATS 2017 and 2018 are shown in Figure 3.
As already mentioned, the BRATS dataset consists of 4 MRI modules: T1, T1c, T2 and Flair. Annotations include 3 types of tumor sub-regions: The enhancing tumor, the necrotic, the peritumoral edema, and non-enhancing tumor core are the tumor sub regions. Annotation is done on these tumor sub regions. Whole Tumor (WT), Enhancing Tumor (ET) and, Tumor Core (TC) are the nested sub regions after annotation. Figure 4 shows some sample MRI images from BRATS dataset.
Figure 3. Feedback UNet with convolutional LSTM architecture
Figure 4. Examples of BRATS dataset
This section analyses the proposed method with some experiments. Initially, performance measures are defined and the results are analyzed. Finally, the proposed method is compared with recent methods and various analyses are done with ablation study.
The most common metrics used for measuring the performance of brain tumor classification are sensitivity and DSC. In this paper, in addition to these measures, accuracy ( $A c_r$ ) and specificity are also used. The formulas for the above-mentioned metrics are defined in Table 1. The proposed method is tested on BRATS 2017 and 2018 datasets.
The whole implementation is done in Matlab 2020a. The proposed method results for both the datasets are mentioned in Table 2. For training UNet, we adopted the Adaptive Moment Estimation (Adam) optimizer, accompanied using batch size 2 and learning rate 10-4. The experiments are carried out for 100 epochs.
Table 1. Performance measures
|
Measure |
Formula |
|
Accuracy |
$A c_r=\frac{T P+T N}{T P+T N+F P+F N} \times 100$ |
|
Sensitivity |
Sensitivity $=\frac{T P}{T P+F N}$ |
|
Specificity |
Specificity $=\frac{\mathrm{TN}}{(\mathrm{FP}+\mathrm{TN})}$ |
|
DSC |
$D S C=\frac{2 \times T P}{F P+(2 \times T P)+F N}$ |
*TP - True Positive, TN - True Negative, FP - False Positive, FN – False Negative
Table 2. Results obtained by the proposed method on both datasets
|
Dataset/ Measure |
BRATS 2017 |
BRATS 2018 |
|
Accuracy (%) |
97.24 |
92.8 |
|
Sensitivity (%) |
97.45 |
91.25 |
|
Specificity (%) |
96.87 |
90.4 |
|
DSC (%) |
97.84 |
88.88 |
Table 3. Performance comparison of proposed method with recent methods
|
Method/Measure |
Dataset |
Accuracy (%) |
Sensitivity (%) |
Specificity (%) |
DSC (%) |
|
Ranjbarzadeh et al. [11] |
2017 |
97.78 |
97.89 |
97.7 |
97.54 |
|
RescueNet [20] |
- |
91.05 |
- |
91.26 |
|
|
Khan et al. [21] |
96.9 |
- |
- |
- |
|
|
Zhou et al. [22] |
- |
- |
- |
81.33 |
|
|
BrainSeg-net [23] |
- |
- |
- |
87.47 |
|
|
AResU-Net [24] |
- |
- |
- |
79.33 |
|
|
Proposed Method |
97.24 |
97.45 |
96.87 |
97.84 |
|
|
Rani et al. [3] |
2018 |
- |
- |
- |
88.46 |
|
Zhou et al. [22] |
92.5 |
- |
- |
- |
|
|
BrainSeg-net [23] |
- |
- |
- |
81.17 |
|
|
Proposed Method |
92.8 |
91.25 |
90.4 |
88.88 |
From the above table, it is observed that the high accuracy is attained by the proposed method, sensitivity, specificity and DSC (which are above 90%) on both datasets.
The proposed brain tumor segmentation technique is analyzed by comparing it with recent methods [2, 11, 20-25]. All these methods use deep learning for segmentation and classification. Table 3 shows the results obtained for the proposed brain tumor segmentation method.
It is better in its performance than the existing methods on both datasets. But the proposed method is not able to achieve the results obtained by Ranjbarzadeh et al.’s [11] method. This is due to the fact that Ranjbarzadeh et al.’s [11] method uses stroke lesion identification.
The goal of this work is in reducing computation burden of the deep learning technique. In this section, the computation time is analyzed with and without inter-intra slice reduction. The proposed method is tested on Intel Core i7 2.8 GHz machine with a GPU NVIDIA GeForce GTX 1050. Table 4 displays the computation time of proposed method.
Table 4. Computation time comparison of proposed method with other methods
|
Dataset/Method |
Computation Time (s) |
|
|
BRATS 2017 |
BRATS 2018 |
|
|
Ranjbarzadeh et al. [11] |
38 |
- |
|
U-Net |
35 |
30 |
|
Proposed method (Inter-intra-slice reduction) |
12 |
10 |
The time needed for computation of the proposed method in the machine with the above-mentioned configuration is 12s for BRATS 2017 dataset and 10s for BRATS 2018 dataset. This speed is 3 times faster than simple U-Net model. Thus, the proposed method is efficient in computation time with satisfiable performance metrics values.
Knowledge of researchers should be useful to human. One major problem that humans face is the brain tumor. There are several researches in this field that uses deep learning. In this paper, research is made in brain tumor segmentation. Though the deep learning suffers from high computation time, this paper tries to reduce the time by reducing the three-dimension MRI data. The reduced slice is given to UNet for segmentation and classification. The proposed brain tumor segmentation method is tested and analyzed on 2017 and 2018 BRATS dataset. It has 5 labels including non-tumor area. It is evaluated using accuracy, sensitivity, specificity and DSC. The comparison of proposed brain tumor segmentation method is done with existing methods. It is proved that there is a negligible loss in performance of existing methods. But the computation time is very much reduced when compared to using only UNet. In future, a new network model can be designed according to the size of the tumor.
[1] Sowmyayani, S. (2022). A handbook on healthcare applications. BPI Publications, pp. 1-42. https://doi.org/10.9734/bpi/mono/978-93-5547-947-1
[2] Chen, J., Wan, Z., Zhang, J., Li, W., Chen, Y., Li, Y., Duan, Y. (2021). Medical image segmentation and reconstruction of prostate tumor based on 3D AlexNet. Computer Methods and Programs in Biomedicine, 200: 105878. https://doi.org/10.1016/j.cmpb.2020.105878
[3] Rani, N.S., Karthik, U., Ranjith, S. (2020). Extraction of gliomas from 3D MRI images using convolution kernel processing and adaptive Thresholding. Procedia Computer Science, 167: 273-284. https://doi.org/10.1016/j.procs.2020.03.221
[4] Myronenko, A. (2018). 3D MRI brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop, Springer, Cham., pp. 311-320. https://doi.org/10.1007/978-3-030-11726-9_28
[5] Ji, Y., Zhang, R., Li, Z., Ren, J., Zhang, S., Luo, P. (2020). UXNet: searching multi-level feature aggregation for 3D medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham., pp. 346-356. http://dx.doi.org/10.1007/978-3-030-59710-8_34
[6] Xu, J., Li, M., Zhu, Z. (2020). Automatic data augmentation for 3D medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham., pp. 378-387. https://doi.org/10.48550/arXiv.2010.11695
[7] Zhang, J., Xie, Y., Wang, Y., Xia, Y. (2020). Inter-slice context residual learning for 3D medical image segmentation. IEEE Transactions on Medical Imaging, 40(2): 661-672. https://doi.org/10.1109/TMI.2020.3034995
[8] Karimi, D., Vasylechko, S.D., Gholipour, A. (2021). Convolution-free medical image segmentation using transformers. In International Conference on Medical Image Computing and Computer-Assisted Intervention Springer, Cham, pp. 78-88. http://dx.doi.org/10.1007/978-3-030-87193-2_8
[9] Zhang, D., Lui, L.M. (2021). Topology-preserving 3D image segmentation based on hyperelastic regularization. Journal of Scientific Computing, 87(3): 1-33. https://doi.org/10.48550/arXiv.2103.16768
[10] He, Y., Yang, D., Roth, H., Zhao, C., Xu, D. (2021). Dints: Differentiable neural network topology search for 3d medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5841-5850. https://doi.org/10.48550/arXiv.2103.15954
[11] Ranjbarzadeh, R., Bagherian Kasgari, A., Jafarzadeh Ghoushchi, S., Anari, S., Naseri, M., Bendechache, M. (2021). Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images. Scientific Reports, 11(1): 1-17. https://doi.org/10.1038/s41598-021-90428-8
[12] Jia, G., Huang, X., Tao, S., Zhang, X., Zhao, Y., Wang, H., He, J., Hao, J., Liu, B., Zhou, J., Li, T., Zhang, X.L., Gao, J.L. (2022). Artificial intelligence-based medical image segmentation for 3D printing and naked eye 3D visualization. Intelligent Medicine, 2(01): 48-53. https://doi.org/10.1016/j.imed.2021.04.001
[13] Dequidt, P., Bourdon, P., Tremblais, B., Guillevin, C., Gianelli, B., Boutet, C., Cottier, J.P., Vallée, J.N., Fernandez-Maloigne, C., Guillevin, R. (2021). Exploring radiologic criteria for glioma grade classification on the BraTS Dataset. IRBM, 42(6): 407-414. https://doi.org/10.1016/j.irbm.2021.04.003
[14] Menze, B., Isensee, F., Wiest, R., Wiestler, B., Maier-Hein, K., Reyes, M., Bakas, S. (2021). Analyzing magnetic resonance imaging data from glioma patients using deep learning. Computerized Medical Imaging and Graphics, 88: 101828. https://doi.org/10.1016/j.compmedimag.2020.101828
[15] Guan, X., Yang, G., Ye, J., Yang, W., Xu, X., Jiang, W., Lai, X. (2022). 3D AGSE-VNet: an automatic brain tumor MRI data segmentation framework. BMC Medical Imaging, 22(1): 1-18. https://doi.org/10.1186/s12880-021-00728-8
[16] Al-Zu’bi, S., Hawashin, B., Mughaid, A., Baker, T. (2021). Efficient 3D medical image segmentation algorithm over a secured multimedia network. Multimedia Tools and Applications, 80(11): 16887-16905. https://doi.org/10.1007/s11042-020-09160-6
[17] Menze, B.H., Jakab, A., Bauer, S., et al. (2014). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10): 1993-2024. https://doi.org/10.1109/TMI.2014.2377694 0
[18] Alom, M.Z., Hasan, M., Yakopcic, C., Taha, T.M., Asari, V.K. (2018). Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation. arXiv preprint arXiv:1802.06955. https://doi.org/10.48550/arXiv.1802.06955
[19] Nema, S., Dudhane, A., Murala, S., Naidu, S. (2020). RescueNet: An unpaired GAN for brain tumor segmentation. Biomedical Signal Processing and Control, 55: 101641. https://doi.org/10.1016/j.bspc.2019.101641
[20] Amin, J., Sharif, M., Yasmin, M., Fernandes, S.L. (2018). Big data analysis for brain tumor detection: Deep convolutional neural networks. Future Generation Computer Systems, 87, 290-297. https://doi.org/10.1016/j.future.2018.04.065
[21] Khan, M.A., Ashraf, I., Alhaisoni, M., Damaševičius, R., Scherer, R., Rehman, A., Bukhari, S.A.C. (2020). Multimodal brain tumor classification using deep learning and robust feature selection: A machine learning application for radiologists. Diagnostics, 10(8): 565. https://doi.org/10.3390/diagnostics10080565
[22] Zhou, T., Canu, S., Ruan, S. (2020). Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation. Computerized Medical Imaging and Graphics, 86: 101811. https://doi.org/10.1016/j.compmedimag.2020.101811
[23] Rehman, M.U., Cho, S., Kim, J., Chong, K.T. (2021). Brainseg-net: Brain tumor MR image segmentation via enhanced encoder–decoder network. Diagnostics, 11(2): 169. https://doi.org/10.3390/diagnostics11020169
[24] Zhang, J., Lv, X., Zhang, H., Liu, B. (2020). AResU-Net: Attention residual U-Net for brain tumor segmentation. Symmetry, 12(5): 721. https://doi.org/10.3390/sym12050721
[25] Magrini, A., Lazzari, S., Marenco, L., Guazzi, G. (2017). A procedure to evaluate the most suitable integrated solutions for increasing energy performance of the building’s envelope, avoiding moisture problems. International Journal of Heat and Technology, 35(4): 689-699. https://doi.org/10.18280/ijht.350401
