Fusion of Ensembled UNET and Ensembled FPN for Semantic Segmentation

The annotation of digital images is one of the most essential tasks in computer vision, as this is the process that allows systems to gain deeper insights into digital images. Adding an annotation to images is essential for the machines to properly identify and interpret the objects. The annotation of images involves labeling the objects that a model is expected to recognize in them. The images, along with tags representing the objects in them, are used to train the system, enabling it to recognize the objects in the new image. To enable computers to learn digital images, several types of annotation are available. Below is a list of the different annotation techniques available in computer vision:

·Image Classification: It is a form of annotation in which an image is analyzed, and based on the analysis; the image is classified into a specific category.

·Object Detection: It is a technique that enables computers to recognize objects in an image.

·Segmentation: Every pixel in an image is classified into a specific class through the segmentation process.

Segmentation is the advanced form of annotation. There are different types of segmentation, including semantic segmentation, panoptic segmentation, and instance segmentation. Semantic image segmentation is the process of classifying each pixel in the image. Semantic segmentation involves assigning the same values to pixels corresponding to a particular class. Multiple objects of the same class are treated as a single entity in semantic segmentation. Semantic segmentation is used to understand the presence, location, and sometimes, the size and shape of objects present in the image. The semantic segmentation process simplifies the analysis of the images by aiding in precise object recognition. Semantic segmentation is the most efficient one because it segments images more precisely and, therefore, makes it easier for systems to identify the desired objects in images.

Nizam et al. [1] have implemented Faster RCNN for the segmentation process. The meliponine family consists of bees that are very small in size. The meliponine family is very difficult to identify in the natural environment. Meliponine image is segmented from its background using Fast R-CNN, an object detection method. As a result, faster RCNN produces 74% accuracy, which is a reasonable level of accuracy.

Another promising architecture used for semantic segmentation is FAST FCN (Fast Fully Convolutional Neural Network). The satellite images are classified using FCN [2] and all the images are grouped into the classes of water, farmland, and field. Thus, by implementing FAST FCN, the researchers were able to achieve an accuracy score of 0.93 and a precision score of 0.99.

The iris pattern is obscured by eyelids, eyelashes, and reflections, which can lead to segmentation errors because the iris lies within a small, damp, and dynamic area. Researchers are increasingly using convolutional neural networks (CNNs) to improve the accuracy of existing iris segmentation techniques. Lozej et al. [3] highlights several deep learning models implemented in UNET and applied to the CASIA dataset to provide precise results.

Mask RCNN algorithms [4] have been used to detect and segment tooth forms which help in achieving reasonable pixel accuracy. Researchers are also exploring the underwater environment. Semantic image segmentation is used to explore objects underwater [5]. Underwater fish are segmented accurately based on employing UNET as the semantic segmentation framework. The architecture proposed, thus resulted in an IOU score of 0.8583.

To speed up and simplify the detection of liver cancer, this research intends to create a UNET architecture for segmenting the liver and tumor from abdominal CT scan pictures. In this work, the number of convolutional filters per block was decreased in a contracting path to improve the accuracy of the model [6]. Segmenting blood vessels has become very popular among researchers. This research proposes a novel approach for segmenting blood arteries using U-net Convolutional Neural Networks (CNNs). The suggested method is more precise, focused, and sensitive when compared to existing methods [7].

The detection and classification of tumors can benefit substantially from the use of medical pictures. Brain tumors are segmented into multiple classes using cascade CNN and distance wise attention mechanism. A multi-modality brain MRI image dataset called BRATS-2018 is utilized to train the suggested architecture. The proposed architecture performed better than other architectures, scoring 87.45 on the dice score [8].

For the segmentation three approaches are suggested: automatic region growth, fully automatic mask selection-based active contour techniques, and mass segmentation dependent on iterative active contour [9]. The experiment proves that the automatic mass segmentation based on the active contour technique results in segmentation with high accuracy. On the Cityscapes Dataset, the suggested approach delivers Panoptic Segmentation. Utilizing DeepLabV3 cutting-edge's encoder-decoder architecture, MobileNet-V2 has been customized and optimized. The proposed method also makes use of Atrous convolution and Spatial Pyramid Pooling to increase its accuracy and sturdiness. Very positive and encouraging results have been obtained, demonstrating the potential of the suggested strategy for rapid and accurate robust scene interpretation [10].

Kim et al. [11] proposed two models and three model ensembling to improve the performance of portrait semantic segmentation by employing soft voting and weighted soft voting. To evaluate the faculty performance of the Teaching assistant evaluation dataset authors employed stacking and voting ensemble methods to improve the performance of the classifier [12].

A multi-task model that executes pixel-based defect segmentation and severity estimate of the flaws in a single two-branch network was proposed [13]. First, two single-task models were developed and trained to perform the fault segmentation and severity estimation tasks independently. Then, they contrasted this to a multi-task model that executes the two tasks at hand simultaneously. Both segmentation tasks increased by 2.5% and 3% mIoU when the tasks were combined into a single model. The author presented a joint attention-guided feature fusion network (JAFFNet) based on the encoder-decoder network for the saliency identification of surface faults. The JAFF module gains the ability to highlight faulty features and reduce background noise during feature fusion for finding low-contrast faults [14].

In the existing literature, the fusion of heterogenous ensembled architecture is not carried out/implemented. In this paper, we propose a novel approach that fuses the ensembled UNET architecture with the ensembled FPN architecture. Ensembled UNET and ensembled FPN is formed by ensembling the three best pretrained deep learning models. The main goal of the proposed architecture is to improve the performance of semantic segmentation. By using the proposed architecture, a very high IOU has been achieved, and dice loss is very minimal when compared with the existing state of art methods in the literature.

The proposed methodology as shown in Figure 3. fuses the ensembled UNET and ensembled FPN. First, the encoder of the UNET architecture is replaced by a single pretrained network (VGG, INCEPTION, RESNET, DENSENET, MOBILENET). Each model is evaluated with the IOU score, F1 score, and Loss function. We select the best three models (InceptionV3, DenseNet, and ResNet) based on their evaluation metrics. Second, the encoder of the FPN architecture is replaced by a single pretrained network (VGG, INCEPTION, RESNET, DENSENET, MOBILENET). Each model is evaluated with the IOU score, F1 score, and Loss function. We select the best three models (VGG16, InceptionV3, and ResNet) based on their evaluation metrics. The selected models are ensembled to form the segmented mask. To improve the performance of segmentation, the ensembled models are fused. The model results with the best IOU score of 98.68%.

3.png

Figure 3. Proposed methodology

3.1 Segmentation using UNET and FPN

UNET is an architecture designed to facilitate the semantic segmentation process. UNET architecture consists of deep learning layers such as convolutional and maximum pooling layers, which are organized in such a way that it results in the segmentation process. Two paths are involved in the UNET architecture. The path on the left is known as the contracting path or encoder and the path on the right is known as the expansion path or decoder. In the left path, the encoder will perform the Maxpool Downsampling to extract the features. In the right path, the decoder will perform upsampling and concatenation to reproduce the image.

A feature extractor known as a Feature Pyramid Network or FPN generates proportionally scaled feature maps at several levels in a completely convolutional manner from a single-scale image of any size. A Top-down and a bottom-up pathway are used to build the pyramid. The feedforward computation of the backbone ConvNet's bottom-up pathway computes a feature hierarchy made up of feature maps at various scales with a scaling step of two. One pyramid level is established for each stage in the feature pyramid. As a reference set of feature maps, the output of each stage's final layer is used. We employ the feature activations produced by each stage's final residual block for ResNet.

The top-down pathway creates the illusion of greater resolution features. Through lateral connections, these features are then improved with features from the bottom-up pathway. Each lateral link combines feature maps from the top-down and bottom-up pathways that are the same spatial size. The bottom-up feature map has lower-level semantics, but because it was subsampled less frequently, its activations are more precisely localized.

3.1.1 Basic operations in UNET and FPN

Operations in UNET

Convolution:

In a convolution operation, features of the input image are extracted. It is the most essential and basic operation conducted in the UNET model.

(N×N)*(F×F)=(N-F+1)*(N-F+1)                    (1)

where, N×N=Size of image and F×F=Size of filter.

Pooling:

Pooling is used to reduce the size of the feature maps thus making the computation faster.

The first step in the segmentation process is to resize the images to the standard size, so the images in the two datasets have been resized to 128*128. All the images are colored images; hence they include 3 channels, resulting in the image shape being 128*128*3.

In UNET architecture, the image is given to the input layer, and to this input image a feature space of 16 is added, resulting in a 128*128*16 result. Convolution is carried out with 3*3 kernel size and padding is modulated as same, which means additional pixels are added on the edges, making the output image the same size as the input. Following this operation is a max pooling operation, upsampling and a few convolutions are performed.

Operations in FPN

Botton-Up pathway:

ResNet is used to build the bottom-up pathway in this method. It is made up of numerous convolution modules (convi for I=1 to 5), each with numerous convolution layers. The spatial dimension is halved as we ascend (i.e. double the stride). Each convolution module's output is tagged before being used in the top-down pathway.

Top-down pathway:

We use the nearest neighbor upsampling to upsample the previous layer by 2 as we proceed down the top-down path. In the bottom-up pathway, we once again apply a 1×1 convolution to the relevant feature maps. We then add them element by element. To all merged layers, a 3×3 convolution is applied. When combined with the up-sampled layer, this filter minimizes the aliasing effect.

3.2 Segmentation carried out by replacing the encoder with the single pre-trained model in UNET and FPN

UNET architecture has two paths, one being the contracting path or encoder, while the other is the expansion path or decoder. The encoder part of UNET architecture can be replaced with any pre-trained model. In the UNET architecture, "backbone" refers to the pretrained model to be used as the encoder. This pre-trained network as a backbone facilitates the use of pre-trained weights (example: ImageNet). The decoder part can be built up programmatically based on the encoder part. FPN architecture has two paths, one being the Bottom or encoder, while the other is the Top-down or decoder. The encoder part of UNET and FPN architecture can be replaced with any pre-trained model. In the UNET and FPN architecture, "backbone" refers to the pretrained model to be used as the encoder. This pre-trained network as a backbone facilitates the use of pre-trained weights (example: ImageNet). The decoder part can be built up programmatically based on the encoder part. The Segmentation Model library contains UNET and FPN architectures as well as several backbones. All backbones have weights trained on the IMAGENET database. This library constructs the decoder automatically based on the backbone used.

A specific pre-trained model is imported based on the backbone used. Data is pre-processed using the get preprocessing module based on the backbone used as the encoder block. As semantic segmentation is simply a multiclass classification, SoftMax activation is used. As the activation is SoftMax, the output is the probabilities for each channel. The modified UNET and FPN model is compiled using Adam optimizer along with a learning rate of 0. 0001. The loss function used in the compilation is a combination of dice loss and focal loss. These two losses are used for semantic segmentation purposes to treat unbalanced data. The metrics used in the compilation process are the F1 score and the IOU score. In this way, the compiled model is then trained in batches of 8 for about 50 epochs, making it more effective, so that when a test image is provided, every pixel is correctly classified.

In this paper, the UNET's and FPN's encoder block is replaced with five pre-trained networks VGG16, RESNET, INCEPTIONV3, DENSENET, and MOBILENETV2. Following are the reasons behind choosing each of these backbones:

·VGG16 - It reduces the number of parameters in convolutional layers thus improving the training time.

·RESNET - It is used as a backbone because it eliminates the vanishing gradient problem.

·INCEPTIONV3 - Choosing a fixed kernel size for image analysis can be difficult since features may vary in size. For global features, larger kernels are preferred over a larger area of the image. Area-specific features are best detected with smaller kernels. InceptionV3 helps in achieving this by going wider with layers. Within the same layer, kernels of different sizes are used.

·DENSENET-DenseNet reduces the number of parameters and thus motivates the reuse of features.

·MOBILENETV2-MobilenetV2 has very small computational complexity and is one of the fastest models, resulting in very high accuracy.

3.3 Semantic segmentation via an ensemble of pre-trained networks in UNET and FPN

Figures 4 and 5 show the ensembling of three pretrained networks in the UNET and FPN architectures. Based on the IOU score of single pretrained networks, the best three models were selected for ensembling. The UNET architecture uses three pre-trained networks (InceptionV3, ResNet, and DenseNet) for ensembling.

The FPN architecture uses three pre-trained networks (VGG16, InceptionV3, and ResNet) for ensembling. These pre-trained models are trained independently and then predictions from these pre-trained models are taken as outputs.

The three outputs, which are mainly probabilities based on replacing the encoder part of UNET and FPN with three pre-trained networks individually, are stored in a list and converted to an array. A set of weights is assigned, and the probabilities of all three pretrained models are multiplied by those weights, and then the summed probabilities are calculated.

4.png

Figure 4. Ensemble of pre-trained networks in UNET

5.png

Figure 5. Ensemble of pre-trained networks in FPN

Segmentation models are a pre-defined library that has UNET architecture and FPN architecture that supports multiple backbones. Based on the backbone specified in UNET and FPN architecture, the segmentation models library will define the decoder. As these backbones were trained on huge datasets such as IMAGENET, defining the backbone results in defining the weights of the pre-trained networks.

As part of the proposed architecture, four pre-trained networks will be utilized as encoders – VGG16, InceptionV3, ResNet, and DenseNet. The reason behind using these pre-trained networks is stated below:

·VGG16

The VGG-16 model is trained based on the extracted features as input data. This relatively small model on low-dimensional data does well. The layers of the model are 13 convolutional layers, 5 pooling layers, and 3 dense layers.

·ResNet

As layers get increased, the error rate also increases but with the use of ResNet, the percentage of error taken during training gets reduced. Using ResNet, the vanishing gradient issue gets solved using skip connections in ResNet.

·DenseNet

DenseNet is used for its ability to reduce the number of parameters, as well as the need to memorize duplicate feature maps. A DenseNet is split into Dense Blocks, where the dimension of the feature maps is the same within a block, but the number of filters differs between them. The output feature maps of the layer are not added to the incoming feature maps by DenseNets but are concatenated. Hence the equation is represented as:

$x_l=H_l\left[x_0, x_1, \ldots \ldots x_{l-1}\right]$                    (2)

·InceptionV3

InceptionV3 architecture is used due to its huge performance gains. Using different convolutional filter sizes, the Inception network can extract features from input data at varying scales. Compared to other architectures, it is less expensive. Below are the steps that are used in training these networks independently as an encoder in UNET.

Images and masks are resized to the standard size of 128*128. Oxford pet dataset contains three classes, every pixel in the mask can be categorized into any one of the three classes. The resized images and resized masks are therefore stored in a list, which is then converted to an array. To analyze the images in the dataset, images are read in RGB format, so they have 3 channels, while masks are read in rescale format. The class values are converted to numeric values (0, 1, 2) using a label encoder. In SD saliency 900 mask dataset pixel in the mask can be categorized into any one of the 189 classes. To analyze the images in the dataset, images are read in RGB format, so they have 3 channels, while masks are read in rescale format. The class values are converted to numeric values (from 0 to 188) using a label encoder. The labels are then one hot encoded using the to_categorical method.

The parameters which are essential for the model construction are defined as follows:

·Activation

SoftMax activation is used as semantic segmentation a form of multiclass classification.

$\sigma(Z)=\left(e^Z\right) / \sum_{j=1}^K e^{z j}$                    (3)

where, σ denotes the softmax activation.

·Loss function

The summation of dice and focal loss is used as these losses are best for the semantic segmentation process.

·Optimiser

Adam optimizer is used and has a faster computation time and also requires fewer parameters for tuning.

$w_t=w_{t-1}-n\left(m_t\right) /\left(\sqrt{V_t}\right)+\epsilon$                    (4)

where, n=learning rate.

·Metrics: F1 score and IOU score.

Pre-processing is done using the get preprocessing module for input images depending on the backbone used. The compilation of the model is then based on the specified backbone, optimizer, and metrics defined. Afterward, the model is trained in batches of 8 for about 50 epochs. The above procedure is thus repeated by defining three pre-trained networks (ResNet, InceptionV3, and DenseNet) as UNET'S backbone and three pre-trained networks (ResNet, InceptionV3, and VGG16) as FPN backbone.

The two outputs, which are mainly probabilities based on replacing the encoder part of UNET with three pre-trained networks individually, and probabilities based on replacing the encoder part of FPN with three pre-trained networks individually are stored in a list and converted to an array. A set of weights is assigned, and the probabilities of all two ensembled models are multiplied by those weights, and then the summed probabilities are calculated. Based on the fused output, the predictions are made on the test image thus resulting in precise and more accurate results.