Deep O-Net Model for Medical Image Segmentation and its FPGA Implementation

In the domain of healthcare, the medical image is an essential tool in controlling the quality of segmentation, and it depends on high-quality medical scanning units. Computed tomography (CT), and X-ray are useful medical imaging types that provide high-quality details to track early the development of diseases and pathological cases [1].

Lung disease is one of the health problems that individuals suffer from and leading to a malfunction in the functionality of the lungs. Segmentation is considered as the first procedure in understanding and interpreting the common lung disease types, such as water density [2], pulmonary fibrosis, and pneumonia [3].

In the last years, the COVID-19 pandemic has swept the world, and its repercussions are still ongoing until now. COVID-19 is a respiratory infection that can be recognized through segmentation of CT lung imaging.

Lung segmentation uses some mechanism to transform a lung image into several meaningful regions or sub-images. It detects the class for every pixel in the lung scan image.

In the real world, the lung regions are delineated visually using medical images by qualified doctors. However, manual segmentation is tedious, not scalable for large image data, and a time-consuming process. Additionally, some errors may appear because the results of delineation details are related to low image contrast, which has an unfair distribution of gray levels and creates noise issues. Moreover, lung scanning is influenced by neighboring tissues and organs, and it is difficult to take multiple medical images due to the effects and harms of the types of radiation. To limit and cope with the spread of viruses and infections, several conventional and automated experiments and solutions have been considered and presented. The conventional methods or extraction features are employed with the help of the developer. Usually, these methods include several procedures related to the threshold, region of interest (ROI), and the convex hull method [4]. Thus, it can be difficult to be general when dealing with multisource images. Today, modern accurate systems are dependent on the integration of Deep learning models. They have as much imitation in data processing as processing takes place in the structure of the human brain. A limited dataset cannot emphasize accurate assessment results, and may not cover all situations. However, the deep segmentation architectures are often constructed using encoding and decoding units with variant scaling levels, like the U-Net and its modified versions [5, 6], SegNet [7], SP-V-Net [8], Anam-Net [9], GFNet [10], DW-UNet [11]. The promise solution merges deep learning models and image algorithms, as addressed in the work [12, 13]. Thereby, cost design is a common issue facing performance; the running speed will be low when applying deep levels. Additionally, the mismatch between the cost design and the dataset’s size will contribute to creating an overfitting state and false errors. Consequently, the developments and scientific research have gone towards Hyperparameter Optimization (HPO) strategies that support the functionality of Training Parameters (TPs) together with Design Parameters (DPs). Usually, TPs minimize the score of the loss function for trial design by utilizing one of the numerical Gradient methods like Adam. Another approach improves accuracy by integrating two or more types of losses, but this will require more inclusive refining of the hyperparameters. To resolve this issue, model-free paradigms like Random Search (RS) are leveraged. In return, the DPs can be solved by building light designs based on human tuning settings [14], but this degrades accuracy. Thus, some of the HPOs apply bioinspired natural algorithms to determine the structure components, such as Particle Swarm Optimization (PSO) [15]. These algorithms mitigated the cost of computations compared to traditional concepts of RS.

In general, the nature optimization concepts add a delay to the running time and may result in deep structure parameters. To complete the implementation, some general works utilized high-performance computing, for example, Graphics Processing Units (GPUs) and Field-Programmable Gate Arrays (FPGAs) [16-18]. The time performance of the diagnosis COVID-19 is compared over several heterogeneous processors, involving multicore CPU, GPU, and FPGA [19].

This work focuses on segmenting the lung and infections in parallel. The major contributions that is to be achieved in this work can be summarized as follows:

1. O-shaped Network (O-Net) attempts to collect lung and infection data by applying the symmetric concept with the encoder-decoder approach.
2. The main property of the suggested O-Net extracts the lung and infection feature from the input images using the same number of layers per level as the previous works based U-Net model. The infrastructure uses absolute difference operations to find the feature maps of the decoding blocks along all levels, instead of using the concatenation operator. These modifications enable the O-Net to offer comparable results; at the same time, the number of parameters will be reduced compared to the hybrid systems.
3. The suggested O-Net is built using two identical new C-shaped networks called C-Nets, then implemented according to the trial method for finding the model and training parameters.
4. RS and PSO are embedded to enhance the accuracy and shorten the selection time of hyperparameters (number of filters, learning rate, kernel size, epochs) of the O-Net infrastructure.
5. The inference for O-Net will be accelerated by conducting it with the help of a high-performance computing FPGA Zynq board and GPU.

This paper is structured into 6 sections: Section 1 gives an introduction to the application of lung segmentation. Section 2 reviews how the researchers deal with the challenges. Section 3 demonstrates the explanations of the suggested methodology. Section 4 describes the experimental details. Section 5 presents the practical results from different views. Also, it discusses and analyzes the main achievement results, comparing them with other prior works. Finally, Section 6 includes a conclusion for the main points of the work. Additionally, offers some future directions for improving the use of the proposed method.

This section describes the optimal framework designed to solve the problem of finding the maximum performance of the suggested segmentation architecture in the lung application. Moreover, it explains the major components of the framework, which are a combination of O-Net, PSO, and RS. Figure 1 illustrates the schematic representation of the framework.

Figure 1. A schematic representation of the overall framework

3.1 Problem definition

The objective is to select the optimal search value for Hyperparameters (HPs) that maximizes the evaluation metrics to consider the model successful in performing CT segmentation. Eq. (1) represents the mathematical objective function with the best solution (Bs).

$Bs=argma{{x}_{HPs\epsilon \wedge }}~f\left( HPs \right)$          (1)

The factor ∧ refers to the search space for the defined HPs, and f(HP) is the evaluation performance metric. The f is not differentiable and has computational complexity. Thus, both RS and PSO will be considered.

3.2 Optimized O-shaped Network

Optimized O-Net is an extended and optimized variant model that will be applied to boost the success of producing accurate and fast delineation. It is utilized as a solution for segmenting the lung CT imaging into four regions: infections, right lung, left lung, and background.  It takes a small target dataset for CT, then learning through pixel-to-pixel and end-to-end approaches. O-Net is extended to the previous works, which combine the important properties of the basis 2D U-Net with the Siamese or symmetric concept, as illustrated in Figure 2(a). The O-Net segments lung and infection areas simultaneously using two identical low-cost new C-Net architectures, dictating one C-Net for each. Their structure is constructed with the help of diverse units shown in Figure 2(b) and (c), then the information outputs are passed to a channel reduction to allow the O-Net to learn and improve feature correlations among variant channels. Finally, the segmented features for both lung and infections are overlapped using an addition function, followed by a channel reduction of linear 1x1 padded convolution embedded with a Sigmoid activation function, assigning the data vector to the specified classes as depicted in Figure 2(d).

The symmetry concept supports the capturing of more details across various multi-scale levels. Mathematically, all data flow over the O-Net infrastructure can be formulated as follows.

$L{{F}_{Encoder1}}=DS\left( {{A}_{ReLU}}\left( SC\left( L{{F}_{input}} \right) \right) \right)$     (2)

$I{{F}_{Encoder1}}=DS\left( {{A}_{ReLU}}\left( SC\left( I{{F}_{input}} \right) \right) \right)$       (3)

$L{{F}_{Encoderi}}=DS\left( {{A}_{ReLU}}\left( SC\left( L{{F}_{Encoderi-1}} \right) \right) \right)~1\le i\le N$     (4)

$I{{F}_{Encoderi}}=DS\left( {{A}_{ReLU}}\left( SC\left( I{{F}_{Encoderi-1}} \right) \right) \right)~1\le i\le N$    (5)

$L{{F}_{Decoderi}}={{A}_{ReLU}}\left( SC\left( \left| L{{F}_{Encoderi}}-US\left( L{{F}_{Encoderi}} \right) \right| \right) \right)~1\le i\le N$       (6)

$I{{F}_{Decoderi}}={{A}_{ReLU}}\left( SC\left( \left| I{{F}_{Encoderi}}-US\left( I{{F}_{Encoderi}} \right) \right| \right) \right)~1\le i\le N$         (7)

$L{{F}_{CR}}={{A}_{Sigmoid}}\left( con{{v}_{1\times 1}}\left( L{{F}_{Decoder5}} \right) \right)$    (8)

$I{{F}_{CR}}={{A}_{Sigmoid}}\left( con{{v}_{1\times 1}}\left( I{{F}_{Decoder5}} \right) \right)$      (9)

$F_{CO}^{Classes\times H\times W}={{A}_{Sigmoid}}(con{{v}_{1\times 1}}\left( L{{F}_{Decoder5}}+I{{F}_{Decoder5}} \right)$          (10)

where, SC represent the standard convolution, A_ReLU and A_Sigmoid referrer to ReLU and sigmoid functions, CR means channel reduction, DS is down sampling based maxpooling, F_CO is fully classification layer, LF means lung feature, IF means infection feature, i level number, and N is count of filters.

Figure 2. O-shaped Network (O-Net) architecture. (a) encoder and decoder block diagram; (b) components of each encoder block; (c) components of each decoder block; (d) represents the contents of the channel reduction block

3.3 C-Net

The C-Net is a C-shaped topology takes a convolutional neural network (CNN) as a fundamental basis. It is designed with light infrastructure, considering the encoding and decoding approach for learning the input samples in an end-to-end way. The major properties of this architecture are embedding one convolution per level and replacing the concatenation operation with an absolute difference. Also, both the encoder and decoder are designed with five levels where their number of feature filters is determined two times, either using a fixed method or a dynamic method. The fixed method follows the same approach as U-Net, the number of feature filters is times by 2 at the encoding side, while the number of feature filters is divided by a factor of 2 at the decoding side. On the other hand, in the dynamic method, the filter numbers are irregular and specified according to PSO and RS algorithms.

The encoding section consists of down-resolution blocks shown in Figure 2, where the image size is shrunk by a stride of 2 for each block. Figure 2(b) shows a building encoding block, which is a chain of one convolution with the same padding, Rectified Linear Unit (ReLU) activation function, and Maxpooling layer.

In correspondence, the decoding section consists of up-resolution blocks (see Figure 2), where the image size is enlarged by 2 per block.

Figure 2(c) illustrates the building decoding block, which uses transpose convolution to enlarge the incoming encoded data by 2. After that, the corresponding data of the encoding-decoding level undergoes an absolute difference layer, followed by a stack of one convolution layer that applies same-padding, and ReLU.

3.4 Loss function

In various segmentation applications, the Dice loss (L_DSC) and Binary Cross-Entropy (L_BCE) are combined as a weighted sum to utilize the best possible properties for improving accuracy. The merged loss function is denoted by Eq. (11). This combination helps to speedup the network convergence, but will lead to a need for more computations.

$L={{L}_{BCE}}+{{L}_{DSC}}$       (11)

The L_BCE loss is selected to perform binary classification functions and work at the pixel level. It measures the divergence between the true (y) and predicted (p) labels for every pixel i according to Eq. (12). Although it introduces stable, probabilistic learning and handles the issue of gradient hiding, it is ineffective for imbalanced classes and does not immediately measure spatial overlap. Moreover, it is inappropriate for tiny object structures because of the dependence on the quality of image pixels.

${{\text{L}}_{\text{BCE}}}\text{=-}\frac{\text{1}}{\text{N}}\underset{\text{i=1}}{\overset{\text{N}}{\mathop \sum }}\,\left( {{\text{y}}_{\text{i}}}\text{log}\left( {{\text{p}}_{\text{i}}} \right)\text{+}\left( \text{1-}{{\text{y}}_{\text{i}}} \right)\text{ }\!\!\times\!\!\text{ log}\left( \text{1-}{{\text{p}}_{\text{i}}} \right) \right)$         (12)

On the contrary, the L_DSC loss mitigates the effect of the unbalanced category. It finds the overlaps between true and predicted areas using Eq. (13). Nonetheless, the L_DSC has less stability during the training; the unsuitable weights will create inaccurate results. Additionally, it is more complicated and may need soothing methods.

${{\text{L}}_{\text{DSC}}}\text{=1-2}\frac{\mathop{\sum }_{\text{i=1}}^{\text{N}}{{\text{y}}_{\text{i}}}{{\text{p}}_{\text{i}}}}{\mathop{\sum }_{\text{i=1}}^{\text{N}}{{\text{y}}_{\text{i}}}^{\text{2}}\text{+}\mathop{\sum }_{\text{i=1}}^{\text{N}}{{\text{p}}_{\text{i}}}^{\text{2}}}$          (13)

The value of N denotes the overall number of image pixels, yi represents the segmented value labeled by the expert, while pi represents the segmented value predicted by the suggested network. Both of y_i and p_i are computed for every pixel and belong to the set {0,1}.

3.5 Particle Swarm Optimization

PSO is an efficient meta-heuristic algorithm that simulates the behavior of a swarm in nature, such as birds flocking, and was developed by Vanneschi and Silva [33]. The PSO is utilized to optimize complex problems concerning a defined criterion, known as the objective function. It is simple and does not differentiate the specified objective function, also containing a few parameters. PSO includes a group of candidate solutions referred to as swarm particles(m), where every particle(i) is expressed in the Eq. (14) as a position vector of a collection of different parameters ($X_{i}^{d})$ that needs to be optimized to realize a maximum or minimum objective function f(HPs) given in Eq. (1). 

${{X}_{i}}=[x_{i}^{1},x_{i}^{2},x_{i}^{3},\ldots x_{i}^{k}]$          (14)

All the particles move through the search space given in Eq. (15) and change their location according to the variables of velocity, personal best, and the global best. Besides, their velocity and position are updated using Eqs. (16) and (17). The particle continues searching the dedicated space until it finds the optimum and best solution (d) for the problem.

$\mathrm{x}^{\mathrm{k}} \in\left[\mathrm{x}_{\min }^{\mathrm{k}}, \mathrm{x}_{\text {max}}^{\mathrm{k}}\right]$           (15)

where, $\text{x}_{\text{min}}^{\text{k}}$ and $\text{x}_{\text{max}}^{\text{k}}~$are minimum and maximum limitations for K-th parameter(particle).

$V_{i}^{k+1}=w.V_{i}^{k}+{{c}_{1}}.{{r}_{1}}.\left( pbes{{t}_{i}}-X_{i}^{k} \right)+{{c}_{2}}.{{r}_{2}}.\left( gbes{{t}_{i}}-X_{i}^{k} \right),$ $i\in \left[ 1,2,3,...,m \right]$          (16)

$\text{X}_{\text{i}}^{\text{k+1}}\text{=X}_{\text{i}}^{\text{k}}\text{+V}_{\text{i}}^{\text{k+1}}$          (17)

where, d denote the number of parameters, i represent particles index, X_i^k and V_i^k referred to the vector position and velocity for the particle i at some iteration k, pbest_i denotes to a personal best position that by a particle i detected, gbest represent the best position that all swarm detected, w denotes to the inertia weigh, c1 and c2 represent the acceleration coefficients, and finally both r1 and r2 are some random values within a range of [0,1].

3.6 Random Search

RS is a simple numerical approach that solves the complicated problem efficiently when searching several hyperparameters[34]. It boosts fast implementations, and can use one of the techniques to stop at an appropriate time to preserve available computational hardware. In RS, the maximum or minimum objective function f(xi) can be obtained by picking out a random group of configurations(xi) uniformly from a dedicated range of hyperparameters R over N iterations. Mathematically, this is formulated as in Eq. (18).

${{\text{X}}_{\text{i}}}=[\text{x}_{\text{i}}^{1},\text{x}_{\text{i}}^{2},\text{x}_{\text{i}}^{3},\ldots \text{x}_{\text{i}}^{\text{k}}],\text{ }\!\!~\!\!\text{ X}\subseteq \text{R}$         (18)

For each iteration(i), the objective function is calculated by Eq. (19) and compares if the new value (Bi) is more reliable than the current best function factor (Bs), then both of will be updated as in Eqs. (20) and (21).

$\text{Bs=f(}{{\text{X}}_{\text{i}}})$       (19)

$\text{Xbest=}{{X}_{i}}$       (20)

$\text{Bs=}{{B}_{i}}$        (21)

The random search for the best solution is repeated until the specified trial number (N) is reached or there is no improvement, and the nearest factor for the desired objective is generated.

This section explores all the results obtained from the practical implementations. It contains visual interpretations and the overall mathematical scores for the entire test dataset. The key factors that are contribute in success of the suggested models are aggregating complementary features using the duplicated C-Net designs, and integrating PSO and RS natural algorithm to obtain the optimal training and model parameters.

5.1 Visualization results of segmentation

Table 4 gives an immediate visual comparison of the manual and automated delineations for selecting one CT sample.

As demonstrated, both the O-Net including C-Net has good segmented images in focusing on infection parts within both lungs at different positions, especially O-Net, which presented high differentiation between the background, lungs, and infections. Although there is diversity among optimizing model approaches, the O-Net model provided accurate segmentation results with minimum mean squared error (MSE). According to the visual images and MSE factor, the first experiment with trial O-Net has the least error and yields accurate results. On the other hand, the approaches of PSO and RS is affected by reducing the number of parameters and the maximum iteration value, but still reliable and approximately matched the reference images with the other experiments, the fourth experiment is considered the moderate approach. This indicates the suggested models have been able to introduce satisfactory segmentation results.

Table 4. Results of random sample using expert and automated models of O-Net for together infection and lungs when the maximum iteration for PSO and RS is equal to 20 and 8

Original CT	Expert			Predicted
	Lung	Infection	Lung + Infection	Trial O-Net	O-Net with PSO	O-Net with RS	O-Net with PSO and RS
MSE				0.0104	0.0213	0.0178	0.0199
Maximum iteration equal to 20
MSE				0.0104	0.0195	0.0179	0.0200
Maximum iteration equals to 8

To quantify the effective performance of the adopted approach, six metrics are reported as shown in Table 5. Accuracy (Acc), DSC, precision (Prec), specificity (SPE), sensitivity (SEN), and Intersection of Union (IOU). Table 6 presents the practical values of measurements using random dataset spliting. As noted, the C-Net and O-Net based on PSO and RS optimizations realizes remarkable performance in terms of DSC, and IOU in segmenting infection areas, while the model tuned by trail approach has the highest results when segmenting both lung regions and introduced a good performance when segmenting infection parts. The diversity is a reflection of the configuration of MPs and TPs. Alongside, applying the absolute value instead of the concatenation approach, which decreases the number of features at each level, and the segmentation combination of lung and infection. 

Table 5. Performance metrics are used to assess the quality of segmentation

Note: TP = True Positive; TN = True Negative; FP = False Positive; FN = False Negative; Acc = Accuracy; DSC = dice similarity coefficient; Prec = precision; SEN = sensitivity; IOU = Intersection of Union.

Table 6. Practical scores based on C-Net and O-Net architectures with random data spling and different implementations for lung segmentation, infection segmentation, lung and infection segmentations

Note: Acc = Accuracy; DSC = dice similarity coefficient; Prec = precision; SEN = sensitivity; IOU = Intersection of Union; O-Net = O-shaped Network; PSO = Particle Swarm Optimization; RS = Random Search; MI represent the maximum number of iterations

Table 7. Dice rates resulted from validation C-Net and O-Net model using four fold protocol, Particle Swarm Optimization (PSO), and Random Search (RS) at maximum iteration of 20

To validated generality of the presented models C-Net and O-Net, the experiment Table 7 gives a report for the obtained results using 4-fold cross validation. As seen good scores are achieved and the fold 3 results the top scores and its weights are used to assess models using test set(fold4).  However, the dice score decreased compared to random split approach because the number of trained pattern is also reduced.

5.3 Cost analysis

As shown in Table 8, the software cost for the Trial O-Net is the highest, due to the different nature of the structure parameters. On the other hand, the combination of PSO and RS represents the balanced solution. To speed up the testing phase and work within the available range of hardware resources, all experimental models are structured units based on fixed symmetric C-Nets designs with a semi-parallelism component per unit level and leveraging 9 DSP units to design parallel convolution, but this has led to some delay, although each one contains a semi-parallelism component per unit level. However, the proposed design is flexible and can be accommodated for diverse tasks by changing the distribution of workload with respect to the capabilities of the available FPGA platform.

Table 8. Software and Hardware costs for implementing the optimised O-Net design

Note: O-Net = O-shaped Network; PSO = Particle Swarm Optimization; RS = Random Search.

5.4 Time analysis

Table 9 compares the execution time of different experiments conducted on variant speeds of platforms. As illustrated, the tuning time for the second experiment based on the PSO method is less than the tuning time of both RS and hybrid PSO and RS at the same number of iterations. This is because the PSO’s searching mechanism is guided and performed according to the details of prior iterations obtained from a sharing group of particles, while the RS does not depend on a specified direction point. In addition, the approximated HPs produced from PSO have created a model with a smaller number of parameters compared to the RS strategy.

Table 9. Running time for variant stages and using different processors

Note: O-Net = O-shaped Network; PSO = Particle Swarm Optimization; RS = Random Search.

Another observation, the training time for experiment 1 using the trial approach consumes the largest implementation time compared to both PSO and RS versions, which are approximately equal, and takes the least time. The cause of equaling PSO and RS is due to having an approximate total count of operations that is decreased by the GPU optimization matrix and run-to-run variations implemented. Besides the size of memory, restrict the epoch period.

Finally, the table demonstrates that the testing time of implementing all experiments on an FPGA is more suitable than a GPU for real-time applications when using 16-bit precision. The reason is that the two C-Nets are designed to work in parallel.

5.5 Discussion and comparison with benchmark architectures

Although the related researchers' work produces remarkable success in terms of accuracy, it suffers from a long time, computation overheads and memory threshold. On the other hand, the suggested O-Net aims to yield a balance point between the cost and the accuracy criteria. This section discusses the most significant improvement implications for the presented model (O-Net) to work within resource-limited environments with good performance results. Tables 10 and 11 abstract all the advancements of the suggested concepts.

In general, the reported results stand out the optimising the model design's effect on the accuracy and time, where deep models give promising scores.

Table 9, the current work-based trail method achieved the best acc and IOU scores when segmenting lung areas, and the work [1] has the highest DSC score. However, the results of the C-Net with PSO and RS are considered a good enhancement. This variation is due to the used machine and HPs. On the other hand, Table 10 reveals the diversity between the time and mode size across variant models. The ECA-Attention with DANet developed with a large model size and consumes the largest amount of memory when compared with current models at the same image resolution. In contrast, the Anam-Net needs the smallest memory size when compared with the combination of PSO and RS at a maximum iteration of 20 when applied on the same image size of 512 × 512. The increase in model parameters, together with image resolution, is reflected in the training and testing time as seen in Table 10. The infrastructure of ECA-Attention with DANet takes the smallest testing time, and the Anam-Net consumes the smallest training time when compared to the suggested models. The suggestion gives moderated measures compared to the related works and can be developed for the future by applying and implementing more advanced optimizing for the hardware design.

Table 10. Comparison of DSC and IOU scores between C-Net design and baseline works for lung segmentation

Note: Acc = Accuracy; DSC = dice similarity coefficient; IOU = Intersection of Union; The bold values are considered the highest results.

Table 11. Comparison of evaluation time and cost between the optimised O-Net design and baseline works

Note: O-Net = O-shaped Network; PSO = Particle Swarm Optimization; RS = Random Search; The bold values are the best reported results.