Cervical Cancer Classification and Region Segmentation in Cervigram Images Using a Convolutional-Enhanced Vision Transformer Network

Cancers affecting women include breast cancer, cervical cancer, and ovarian cancer, as per the records stated by the World Health Organization (WHO) in 2022. As per WHO statement, every year around 270,000 women patients lose their lives due to cervical cancer. Hence, the detection of cervical cancer in women has received considerable attention worldwide. Cervical cancer occurs in the cervix region of women patients due to Human Papillomavirus (HPV) infection [1]. The symptoms of cervical cancer are continuous bleeding in vagina region; severe pelvic pain, foul smell in the blood and fatigue. There are two types of cervical cancer physical screening available in the current medical system. They are stated as the Pap smear cell test and the HPV test. The pre-cancerous cells in the cervix region of the female patient can be identified using the Pap smear cell test [2, 3]. In the case of the HPV test, HPV in the cervix region has been identified. If any abnormalities in the cervix have been found, then a gynecologist uses colposcopy equipment, which is inserted inside the cervix region, and cervigram images are captured [4-6]. These images are called cervigrams or colposcopy images. Over the last decade, researchers in this field have used machine and deep learning techniques to detect cervical cancer at an earlier stage. In this article, the cervigram images are used to detect and segment the cancer region of the pixels using these predictive modeling algorithms. These algorithms create significant potential for cervical cancer detection systems in a fully automated method [7-9]. Figure 1(a) depicts the cervigram image in a non-cancerous case where there are no abnormalities in this image, and Figure 1(b) depicts the cervigram image in a cancerous case where there is an abnormal region in this image.

The contributions of this research article are stated in the following points.

•The spatial-frequency image transformation has been obtained by the Gabor transform (GT) to improve the spatial relationships between the pixels in the cervigram images.

•The Convolution feature extractor module has been proposed in this work to produce the attention-based feature maps from the cervigram images, which have both temporal and local feature values to improve the cervigram image identification rate.

•The Convolutional Incorporated Vision Transformer (CIVT) model has been constructed from the Vision Transformer Model (VTM) by implementing the Convolutional layers in the transformer encoder blocks to improve the classification performances of the proposed cervical cancer detection system.

•The cancerous region of pixels in the cancerous cervigram images is segmented using a segmentation algorithm with the detection of all regions of cancer pixel boundaries.

•The extensive experimental setup is constructed with the performance evaluation parameters to validate the experimental results of the cancer region segmentation accuracy with other recent models.

1.png

Figure 1. Representative cervigram images for cervical cancer classification, (a) non-cancerous case cervigram image, (b) cancerous case cervigram image

This research article is split into various sections. Section 1 introduces cervical cancer and its detection methodologies in detail. Section 2 addresses the traditional methodologies used for detecting cervical cancer with various modeling approaches; Section 3 explains the proposed method to detect cervical cancer, and the results are completely demonstrated in Section 4 with recent state-of-the-art methods. The conclusion along with the future scope of this article has been given in Section 5.

The detection and segmentation of cancer regions in cervigram images are proposed using a Deep Attention Convolution Feature Extractor (DACFE) based CIVT model. This proposed system consists of pre-processing, Gabor transformation, a feature extractor along with the proposed CIVT model. The cancer case and non-cancer case cervigram images are individually processed to produce the feature matrix during the training stage of the proposed system. The test cervigram image is pre-processed, and then spatial transformation is achieved using GT. From this spatially transformed image, the deep edge features are computed using the proposed DACFE, and the computed deep edge features are used by the proposed CIVT model to produce the classification outputs as either a cancer cervigram image or a non-cancer cervigram image. Finally, the heuristic segmentation algorithm is proposed to identify and segment the cancer pixels in the cancer cervigram image. Figure 2(a) shows the proposed DACFE-CIVT-based cervical cancer training system, and Figure 2(b) shows the proposed DACFE- CIVT based cervical cancer testing system.

2a.png

(a)

2b.png

(b)

3.1 Data augmentation

The expansion of the used dataset is done by Data Augmentation (DA) [17]. Through the DA process, the image count in the cervigram imaging dataset has been increased by modified versions of the source images in the same dataset. DA is used for eliminating the over fitting and improve the proposed model's performance. The diversity and generalization of the proposed model has been improved by DA. Here, flipping with respect to horizontal and vertical and rotation with respect to left and right have been used for DA and all these DA methods are individually applied to all the cervigram images in the dataset. The DA method is used only in the training phase of the proposed system and it is not required in the testing phase of the proposed cervical cancer classification system.

3.2 Discrete Gabor Transform

It is a multi-scale transformation model which is used in many image processing applications such as analysis of texture, computation of Gabor features and filtering on the images. In this work, the two-dimensional Discrete Gabor Transform (DGT) is used for the spatial-frequency transformation process. It is the extended version of the short-time Fourier Transform. The DGT uses a Gabor kernel that converts the spatially processed cervical image into a spatial-frequency image through the linear convolution process. The Gabor kernel of the DGT is designed with respect to multiple scales and orientations. During the cervical image transformation using DGT, the texture and edge pixels are preserved. In this work, the scale values are set between 0 and 4, and the orientation values are set between -30° and +30°. Now, the preprocessed cervical image is linearly convolved with the 1D kernel of DGT with respect to the specified scales and orientations. Through the different scales and orientations, a total of 300 kernels are generated, which are individually convolved with the preprocessed cervical image. Hence, 300 GT images are produced and all these images are combined into a single GT image by taking the maximum pixel value in all the 300 GT images at the same coordinate position. The pixels in the final GT image belong to the spatial and frequency and it is used to compute the features by the proposed feature extractor. Table 2 shows the DGT design specifications.

Table 2. Discrete Gabor Transform (DGT) design specifications

3.3 Deep attention convolution feature extractor

The feature extractor module is the main block of the proposed cervical cancer image detection system, which is used to produce the deep feature maps [18-21]. These feature maps are individually produced for non-cancer cervigram images and cancer case cervigram images during the training phase of the proposed system. The feature maps represent the correlation and pattern values that are essential for differentiating the cervigram images belonging to either a non-cancer case or a cancer case. The traditional feature map production system used algorithms which areshowing the variations between the pixels and the region of pixels in the cervigram images. The feature maps produced by the traditional algorithms, such as binary pattern algorithms [21], energy-based Grey Level Co-occurrence Matrix (GLCM) algorithm [20] and statistical feature production algorithms [19], do not generate enough features to discriminate the cervigram images if they belong to a low-resolution format. The cervigram images mostly belong to a low-resolution format, and hence these traditional feature computation methods through the algorithms are not enough to produce significant feature maps. Hence, the attention based Convolution feature extractor module is proposed in this work to produce attention-based feature maps that have both temporal and local feature values. This proposed DACFE module is derived from the basic deep convolutional network LeNet, which is depicted in Figure 3(a). This Traditional LeNet feature extraction module receives the GT cervigram image and produces the 2Ddeep feature map (2D-DFM). This feature extractor module contains two Convolution layers (CLL1 and CLL2) and two Max-Pooling Layers (MLPs), as illustrated in the following Eqs. (1)–(4).

ConvolutionLayers: CLL1 andCLL2; Internalstructureof(CLL1):32 convolutionfilters        (1)

with 3 * 3 kernel size Internalstructureof(CLL2): 64 convolutionfilters        (2)

with 5 * 5 skernel size        (3)

$\begin{aligned} { Max }- { Pooling }- &  { Layer }:  { MLP }  \rightarrow 2 * 2  { Max }-  { Poolingalgorithm }\end{aligned}$        (4)

The GT cervigram image is fed through a series of two convolutional layers. The purpose of this convolutional layer is to generate the local feature values from the cervigram image. The CLL1 has been constituted using 32 filters with a 3 × 3 kernel size. Hence, 32 feature maps are produced, and all of these are combined into a unique feature map. The feature map generated by CLL1 is fed into MPL to produce the shrunk feature map, and it is passed through CLL2. This CLL2 has been constituted using 64 filters with a 5 × 5 kernel size. Hence, 64 feature maps are produced, and all of these are combined into a unique feature map. The feature map generated by CLL2 is fed into MPL to produce the shrunk two-dimensional deep feature map, as illustrated in Figure 3(a). This produced deep feature map contains spatial feature values alone, and hence the classified outputs by the classification algorithm using these generated spatial feature maps are reduced. This also affects the cancer region segmentation process. To improve the cancer region segmentation output along with the cervigram image classification accuracy, the feature map containing both spatial and local feature values is required. Therefore, the proposed DACFE feature extraction modules are developed to produce the two-dimensional feature map which consists of both spatial and local feature values. Figure 3(b) shows the proposed DACFE feature extraction module from the Gabor cervical image.

3a.png

(a)

3b.png

(b)

Figure 3. Comparison of conventional and proposed feature extraction architectures for Gabor-transformed cervigram images, (a) Traditional LeNet-based feature extraction module from the Gabor cervical image, (b) proposed Deep Attention Convolution Feature Extractor (DACFE) feature extraction module from the Gabor cervical image

This proposed DACFE feature extraction module receives the GT cervigram image and also produces the 2D-DFM, which contains both spatial and local feature values. This proposed feature extractor module contains four Convolution layers (CLL1, CLL2, CLL3, and CLL4) and four MLPs, as illustrated in the following Eqs. (5)–(10).

ConvolutionLayers: CLL1, CLL2, CLL3 andCLL4;        (5)

Internal structure of (CLL1):512 convolution filters  with 5 × 5 kernel size        (6)

Internal structure of (CLL2):1024 convolution filters  with 7 × 7 kernel size        (7)

Internal structure of (CLL3):512 convolution filters  with 3 × 3 kernel size       (8)

Internal structure of (CLL4):512 convolution filters  with 7 × 7 kernel size       (9)

$\begin{aligned} &{Max}-{ Pooling }- {Layer}: M L P \rightarrow 2 * 2\ { Max }-  { Poolingalgorithm }\end{aligned}$       (10)

The GT cervigram image is fed through a series of two convolutional layers CLL1 and CLL2 through the MLP layer. The CLL1 has been constituted using 512 filters with a 5 × 5 kernel size. Hence, 512 feature maps are produced and all these are combined into a unique feature map. The feature map generated by CLL1 is fed into MPL to produce the shrunk feature map, and it is passed through CLL2. This CLL2 has been constituted using 1024 filters with a 7 × 7 kernel size and hence 1024 feature maps are produced and all these are combined into a unique feature map. The feature map generated by CLL2 is fed into MPL to produce the two-dimensional local feature map, as illustrated in Figure 3(b).

The main advantage of this proposed DACFE feature extraction module is to produce the local feature map values to improve the cervigram image classification rate by the classification algorithm. The local feature map has been produced by CLL1 and CLL2. In order to produce the final local feature map from the Gabor cervigram image, the Modulated Attention Block (MAB) is proposed. This MAB receives the local feature maps that are generated by CLL1 and CLL2 and produces the final local feature map through the attention approach. This MAB splits the CLL1 and CLL2 feature map into n × n, and the first block feature values of CLL1 feature map are self-attended by the all split blocks in CLL2 feature map. Then the second block feature values of CLL1 feature map are self-attended by the all split blocks in CLL2 feature map. This process is carried out till the last split block in CLL1. The attention module generates the local feature map and this local feature map is fed into CLL3, which has been constituted using 512 filters with a 3 × 3 kernel size and hence 512 local feature maps are produced, and all these are combined into a unique local feature map. The local feature map generated by CLL3 is fed into MPL to produce the local feature map. The produced local feature maps are now concatenated by the Feature Concatenator (FC). The concatenated feature map is finally passed through CLL4, which has been constituted using 512 filters with a 7 × 7 kernel size. Hence, 512 local feature maps are produced and all of these are combined into a unique deep-local feature map. The feature map generated by CLL4 is fed into MPL to produce the deep-local feature map. This feature map may contain certain negative values, and these negative values reduce the cervigram image classification rate. Hence, the negative values in the produced spatial-local feature are removed by the Linear Rectification Unit (ReLU) and finally two-dimensional deep feature maps are produced. Table 3 shows the configuration settings of the proposed DACFE.

Table 3. Configuration settings of the proposed Deep Attention Convolution Feature Extractor (DACFE)

3.4 Proposed Convolutional Incorporated Vision Transformer model

The generated two-dimensional deep feature maps from the proposed DACFE feature extraction module are trained and classified through the transformer model in this work. This research work proposes a CIVT model which is an improved and modified version of the traditional VTM. The proposed CIVT is a kind of transformer model which applied deep learning principles for object detection and image classification. It uses a self-attention mechanism to deliver the global relationships between the various parts of the image. The traditional CNN computed local features alone through convolutional layers (filters) and it is not adequate to handle large image datasets with various patterns. Hence, to improve performance efficiency during the handling of large datasets, VTM is used, which computes global features through the self-attention mechanism. Through this self-attention mechanism, the optimal performance for handling a large dataset can be obtained. Hence, the VTM is identified as the alternative solution for the traditional CNN model. The proposed CIVT model using transformer encoder blocks with an attention mechanism is illustrated in Figure 4.

4.png

Figure 4. Proposed Convolutional Incorporated Vision Transformer (CIVT) model for cervical image classification process

The CIVT architecture for the proposed cervical image classification system contains four functional modules as stated below.

•Patching and Embedding (PaE) module;

•Positional Encoding (PE) module;

•Multiple Transformer Encoder Module;

•MLP Head Module.

The PaE module converts the two-dimensional feature map matrix into a one-dimensional flattened feature map with its positional coordinates. This module performs input patch splitting, patch flattening and patch embedding. In input patch splitting block, the input 2D-feature map is split into a number of non-overlapping patches. For the 2D-feature map of size W × H, the number of generated patches are about $\frac{W}{8} \times \frac{W}{8}$. In this work, the 2D feature map size is 256 × 256; then, 32 × 32 = 1024 patches are generated through the patch splitting block. Each generated patch is also called as tokens. Each generated two-dimensional patch is converted into a one-dimensional vector using the patch flattening block. Then, the linear projection algorithm has been used in the patch embedding block to project the one-dimensional feature vector into a high-dimensional vector. The linear projection is performed through the Convolutional layer and the one-dimensional feature vector generated by the patch flattening block is now passed through the convolutional layer to generate the two-dimensional feature vector. The kernel size of the convolutional layer is equal to the generated feature patch size.

The PE module in the proposed CIVT classifier is used to add or insert the spatial coordinates or positions of each generated patch into the two-dimensional feature vector which is generated through the previous patch embedding block in PaE module. There are numerous positional embedding algorithms available to generate and add the patch coordinates. They are listed as learnable positional embedding, sinusoidal PE, coordinate PE, and relative PE. Among them, the coordinate PE is used as the PE algorithm in this work to generate the coordinates of each generated patches. This encoding algorithm is chosen over the other encoding algorithms due to its lower complexity and stable performance with respect to its size. This encoding algorithm finds the horizontal and vertical coordinates (x,y) and adds this coordinate or positional information to the 2D-feature map, which is generated through the PaE module of the proposed CIVT classification architecture.

The proposed Convolutional Transformer Encoder (CTE) module (depicted in Figure 5) is designed with Transformer Encoders (TE) and Convolutional layers with different kernel sizes. This TE module is designed with TE layers which is used to learn global relationships between the generated patches. This uses Multi-Head Self-Attentio (MHSA) mechanism to determine and derive the global relationships with the aid of Feed Forward Network (FFN) process. The architecture of the TE is depicted in Figure 6.

5.png

Figure 5. Proposed Convolutional Transformer Encoder (CTE) module

6.png

Figure 6. Internal architecture of the Transformer encoder components used in the proposed CIVT model, (a) Architecture of Transformer Encoders (TE), (b) internal design of Multi-Head Attention (MHA)

The TE consists of a Normalization block, MHA block and MLP block as depicted in Figure 6(a). In the Norm block, the generated 2D-feature-embedded patches are applied to the Normalization layer (Norm), which is used to prevent covariate shift and also used for fast convergence. It performs scaling and shifting on the 2D-feature-embedded patches. Due to this normalization process, the training time is significantly reduced which improves the proposed classifier architecture's reliability. The normalized feature-embedded patches are given to MHA. The long-range dependency relationships between each normalized output are computed using MHA. The internal architecture of MHA with query (q), key (k) and value (v) is depicted in Figure 6(b).

This MHA architecture uses a self-attention mechanism to find long-range dependency relationships. The entire operation of the MHA is depicted in the following Eq. (11).

$MHAOutput (Q, K, V)=SoftMax \left(\frac{Q . K^T}{\sqrt{d_k}}\right) \cdot V$        (11)

where, $d_k$ is the key dimension and $Q . K^T$ is the attention score.

The ‘matmul’ is used to generate the attention score, which matches every patch feature with the other patch features. Then, the stable gradients are achieved between the patch features. Then, the stable gradients are achieved between the patch features using the scaling function through the key dimension. Then, the softmax layer performs the normalization of the generated attention score into the weights. Finally, the normalized weights are dot product with the value (v) to generate the final output.

Table 4. Interfered design specifications of the proposed Convolutional Incorporated Vision Transformer (CIVT) model

This output is passed to the MLP, which is the key element in the TE. It can be constructed using an FFN. This block provides feature refinement on the output of the MHA. It consists of two linear layers, where the first layer is used to expand the input feature dimension from 512 to 1024, and the non-linear activation function is applied to remove the uncertainty in the output. This output is now passed through the second linear layer, which performs the downscaling of the generated feature dimension from 1024 to 512. The MLP output is finally passed to MLP head, which is used to produce the desired output using Fully Connected Neural Networks (FCNN). The desired output is produced by class probability. It consists of three dense layers with 4096, 2048, and 512 neurons in each layer. The output of the final dense layer is passed through the softmax layer to produce the desired classification outputs. Table 4 is the interfered design specifications of the proposed CIVT model for the classifications of the non-cancer and cancer cervigram images.

7a.png

(a)

e065bc6e-55d3-4937-842d-900f1b40ccc6.png

(b)

Figure 7. Performance evaluation of the proposed CIVT model under different numbers of Transformer Encoders, (a) Impact of the number of Transformer Encoders(TE) on the GC dataset, (b) impact of the number of Transformer Encoders (TE) on the KI dataset

8.png

The number of TE in the proposed CIVT creates the greatest impact on the cervigram identification accuracy with respect to cancer and non-cancer cases on both GC and KI datasets. The cervigram identification accuracy has been improved on both datasets when the number of TE is increased. Figure 7(a) shows the impact of the number of TE on GC dataset, and Figure 7(b) shows the impact of the number of TE on KI dataset.

Figure 8(a) shows the proposed DACFE-CIVT based cervical cancer classification outputs on GC dataset and Figure 8(b) shows the proposed DACFE-CIVT based cervical cancer classification outputs on KI dataset.

Figure 9 shows the overall cancer region segmentation output images on the GC dataset. Figure 9(a) shows the source cervigram images, Figure 9(b) shows the cancer region segmented images by the proposed method and Figure 9(c) shows the gold standard cancer region segmented images.

9.png

Figure 9. Cancer region segmentation output images on GC dataset, (a) source cervigram images, (b) cancer region segmented images by proposed method, (c) gold standard cancer region segmented images

3.5 Segmentation

This research article uses the Morphological Segmentation (MS) framework which uses morphological functions for segregating the cancer pixels in the classified cancer case cervical image. The morphological filters uses morphology functions for locating the cancer pixels through opening and closing. The following steps are used in the MS framework for locating the cancer pixels.

Step 1: The ‘erosion’ with 0.2 mm structuring element is applied to the cancer case image which is used to remove the small protrusion pixels in the image.

Step 2: The opening function (which follows erosion by dilation) has been applied on step 1 output image with 0.1 mm and 0.2 mm structuring element values, respectively.

Step 3: The closing function (which follows dilation by erosion) has been applied to the step 1 output image with 0.2 mm and 0.5 mm structuring element values, respectively.

Step 4: The image obtained from step 3 is subtracted from the image obtained from step 2 to identify the abnormal pixels with few miss-leading pixels.

Step 5: The miss-leading pixels in the step 4 output image is detected and removed using Top-Hat transform.

Figure 10 shows the overall cancer region segmentation output images on KI dataset, Figure 10(a) shows the source cervigram images, Figure 10(b) shows the cancer region segmented images by the proposed method, and Figure 10(c) shows the gold standard cancer region segmented images.

10.png

Figure 10. Cancer region segmentation output images on KI dataset, (a) source cervigram images, (b) cancer region segmented images by proposed method, (c) gold standard cancer region segmented images

In this article, the cancer regions in cervigram are detected and segmented using the proposed DACFE-CIVT classification model. This proposed work combines the deep learning model and the transformer model to improve the cancer case cervigram image detection and classification accuracy. The deep learning method is used in this article for computing the local and statistical features from the cervigram images, and the proposed transformer model has been used to perform the classification process more effectively than conventional transformer models. The proposed DACFE-CIVT-based cervical cancer detection system is applied and tested on two datasets, GC and KI, and the results are reported with detailed demonstration.

Even though the proposed DACFE-CIVT classification model, it exhibits certain limitations in the cancer segmentation process in abnormal cervigram images, as illustrated in the following points.

•The proposed DACFE-CIVT classification method has been validated only on the online GC and KI cervigram imaging datasets and not verified on the clinical real-time cervigram imaging datasets, where bias is required to verify the significance of the proposed cancer segmentation method.

•The cancer region boundary of pixels is important to estimate the severity levels of the cancer-case cervigram images. In this article, the segmented boundary of pixels is not used as the criterion for severity estimation.

The main future expansion of this research work is stated in the following points:

•The cancer case cervigram images will be diagnosed in the future to estimate and analyze their severity levels using Generative Adversarial Networks (GAN) based transformer model.

•The graph cut-based deep learning model will be used to segment the cancer regions in the cancerous case cervigram images to improve the external boundary of cancer regions more accurately than the current study.

•The clinical validation will be done in the future with larger cervigram imaging datasets to validate the experimental results of the proposed model.