Utilizing Mask R-CNN for Automated Evaluation of Diabetic Foot Ulcer Healing Trajectories: A Novel Approach

The study was conducted at a diabetic foot clinic of a tertiary care health facility in North India. The protocol was approved (Ref. No-2020/000170) by the Research Ethics Committee of the PGIMER Chandigarh, and infirmed, written consent was obtained from all the participants. The method for evaluating the healing trajectory of a diabetic foot ulcer consisted of the following steps: image acquisition (visible and thermal image), image fusion, ROI segmentation, automatic measurement of the ulcer’s size (weekly) and its thermal parameters, and finally prediction of the ulcer’s healing status as depicted in Figure 1.

2.1 Data acquisition

The imaging setup consists (Figure 2) of three main components i.e., infrared (IR) camera, a thermally insulated foot holding box and subjects (patients). As per the approved protocol, all trials were conducted in a controlled hospital setting. In order to reduce the impact of temperature variations, subjects were instructed to fast for two hours before the study [24]. This assured that the digestive system would not affect the temperature. Also, prior to data collection, adequate rest time (minimum 15 minutes) was given to the subjects to normalise blood circulation. A FLIR E-60 infrared thermo-imaging camera (which gives thermal as well as RGB images) was used to photograph all of the ulcers. This is a non-contact tool for assessing the surface temperature of the targeted ROI. The thermal camera, having an image resolution of 320×240 pixels, thermal sensitivity ¡ 0.05℃ @ 30℃ and ±2℃ or 2% of reading, for ambient temperature of 10℃ to 35℃, field of view (FOV) / minimum focus distance 25°×19° / 0.4 m (1.31 ft.), was lodged in a room space (minimum 2 square meter), and distance (fixed 1 m) between the subject and the IR camera. The observation area was appropriately protected from outside infrared radiation, and the background was kept non-reflective.

image003.jpg

Figure 1. Process flow diagram of proposed methodology

image004.png

Figure 2.​ Image acquisition setup

2.2 Thermal and visible image fusion

The process of integrating two or more images into one composite image that combines the information contained within the individual images is referred to as image fusion. The purpose of the merging of thermal and visual images in this study was to obtain the finest features to localize the ROI (foot ulcer) [25]. The primary objective of the fusion approach is to effectively overlap the colour value from one image onto the resulting fused image. In this case, the thermal image serves as the overlapping candidate. We used the HSV fusion method (Figure 3) to fuse the visible and thermal images. The Structural Similarity Index (SSIM) was calculated to figure out how well these fusion methods work for each fused image [26].

$\operatorname{SSIM}=\frac{\left(2 \times \bar{x} \times \bar{y}+c_1\right)\left(2 \times \sigma_{x y}+c_2\right)}{\left.\left(\sigma_x^2+\sigma_y^2\right)\right) \times\left((\bar{x})^2+(\bar{y})^2+c_1\right)}$                       (1)

The HSV-based fusion was performed on both the thermal and visible images, ensuring that both the visual and thermal properties are intact. HSV-based fusion is described as the HSV color space, which represents how humans see and interpret color using three main components: hue (H), saturation (S), and value (V). The HSV colour space distinguishes between chromatic and achromatic information (H and S, respectively) and expresses them separately. Hue is defined as a 0° to 360° angle that represents the type of colour (red, green, blue, etc.). Saturation is the percentage of gray in a colour that ranges from 0% to 100%. The value of a colour describes its brightness or intensity. The appropriate hues get gradually brighter as V ranges from 0 to 100% [27]. This approach allows for easy interpretation of both thermal and visual information; it preserves the visual information and enhances the thermal information with improved contrast. Many studies have used the HSV fusion method in medical images from different modalities such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), Single Photon Emission Computed Tomography (SPECT), and Positron Emission Tomography (PET) for better visualization of anatomical and functional information, improved accuracy, and reduced radiation exposure [28-30].

pi_zhu_2023-08-31_095358.png

Figure 3. HSV fusion method

2.3 Expert labelling and data annotation

One of the clinical investigators annotated the wound area (ulcer) and the full foot via the annotation tool makesense.ai. The annotations are in Coco format, and both the dataset and the annotations are given to the R-CNN model for training. Data labelling is shown in Figure 4.

image007.png

Figure 4. Data labelling process

2.4 Features extraction and ROI segmentation

We employed the Mask R-CNN technique for the segmentation of foot ulcers and the whole foot from the fused images [23, 31]. Mask R-CNN relies on region proposals, which are generated via a Region Proposal Network (RPN) (Figure 5). Mask r-CNN follows the Faster r-CNN model by having a feature extractor followed by the RPN. This is then supplemented by an operation known as ROI-Pooling, consisting of three important modifications to produce standard sized outputs suitable for input to a classifier [32]. First, Mask R-CNN replaces the somewhat imprecise ROI-Pooling operation used in Faster R-CNN with an operation called ROI-Align that allows very accurate instance segmentation masks to be constructed; and second, Mask R-CNN adds a network head (a small fully convolutional neural network) to produce the desired instance segmentation. Finally, mask and class predictions are decoupled. The mask network head predicts the mask independently from the network head predicting the class. This entails the use of a multi-task loss function. The loss function [33] of Mask R-CNN is defined as:

$L=L_{\text {class }}+L_{\text {box }}+L_{\text {mask }}$                       (2)

where, L_class + L_box  are identified the same as in Faster R-CNN [34, 35], L_class + L_box are defined as:

$\begin{gathered}L_{\text {class }}+L_{\text {box }}=\frac{1}{N_{\text {cls }}} \sum_i L_{\text {cls }}\left(p_i, p_i^*\right)+  \frac{1}{N_{\text {box }}} \sum_i p_i^* L_1^{\text {smooth }}\left(t_i-t_i^*\right)\end{gathered}$                       (3)

$L_{\mathrm{cls}}\left(\left\{p_i, p_i^*\right\}\right)=-p_i^* \log p_i^*-\left(1-p_i^*\right) \log \left(1-p_i^*\right)$                       (4)

And the L_mask is the average binary cross-entropy loss.

$\begin{gathered}L_{\mathrm{mask}}=-\frac{1}{m^2} \sum_{1 \leq i, j \leq m}\left[y_{i j} \log \circ y_{i j}^k+(1-\right.   \left.\left.y_{i j}\right) \log \left(1-\circ y_{i j}^k\right)\right]\end{gathered}$                       (5)

where, $p_i$ = predicted probability of anchor i being an object, $p_i^*$ = ground truth label (binary) of whether anchor I is an object, $t_i$ = predicted 4 parameterized coordinates, $t_i^*$ = ground truth coordinates, Ncls = Normalization term, set to be mini-batch size (~2) in the paper, Nbox = normalization term, set to the number of anchor locations (~256).

image016.jpg

Figure 5. Mask r-CNN architecture

Interventions for DFU require periodic assessment for their efficacy, as it is critical to understand whether the applied therapy is beneficial or requires modification, especially in wounds that are visibly not responding to the offered treatment. The wound healing trajectory is one way to assess the efficacy of treatment. We proposed an imaging-based technique that involves the fusion of thermal and visual images to determine the healing trajectory of a DFU. The present study found a good correlation between fused thermal images and ground truth obtained from automated camera based wound estimation for the assessment of the healing trajectory of a diabetic foot ulcer.

Image fusion, which is extensively used in medical imaging, sensor feed analysis, and satellite imaging, tries to generate new images with an enriched feature map that are more suited to human visual perception. We explored different fusion methods for our dataset, such as DCT Fusion [25], DWT Fusion [39], Laplacian Pyramid Fusion [40], PCA Fusion [37], and HSV Fusion [38, 41]. However, frequency-based techniques extract redundant features, and there is a slight decrease in the colour contrast of the fused image. For PCA fusion, the eigenvectors provide the thermal candidate more weightage since the principal components are more distinct, resulting in the fused image losing desirable features of the visual image. On the other hand, the HSV fusion precisely extracts the desired features, and these features blend with the colour map obtained from the thermal image. In Table 2, the performance of each fusion method was evaluated using the Structural Similarity Index (SSIM), and based on the SSIM score, the HSV fusion approach was selected for our dataset. An instance segmentation method, Mask R-CNN (Mask Region-based Convolutional Neural Network), was used to perform the DFU segmentation, which significantly improved the segmentation result. Basically, Mask R-CNN and CNN are both deep learning models used for image analysis and processing, but their architecture and functionality differ significantly. CNN is a form of deep neural network used for image classification, object detection, and segmentation. It consists of convolutional, pooling, and fully connected layers. Whereas, the Mask R-CNN is an extension of the Faster R-CNN model that involves the addition of a mask prediction branch to the region proposal network. It is used for object detection, segmentation, and instance segmentation tasks. It includes a backbone network, a network for region proposal, and two parallel branches for object detection and mask prediction. The main reason of choosing Mask R-CNN for this study was to segment the foot ulcer properly from the entire foot. It is a good choice for segmentation because it uses a region-based approach to object detection and segmentation, which allows it to accurately identify and segment objects of interest within an image. Unlike traditional CNNs, which require the entire image to be fed through the network, it first generates region proposals and then selectively processes only the regions that are likely to contain objects. The performance of the segmentation technique was validated by the loss function (Figure 6). Several other studies have proposed to segment the DFU from the foot and to know the healing trajectory of the DFU. Cui et al. [16] introduced method of automatic segmentation using CNN model. In this study, author generated a probability map with the help of CNN and the U-Net model on color images. Alzubaidi et al. [42] propose a novel CNN based model, i.e., DFU-QUTNet, to segment the DFU. The focus of the novel CNN model was to increase the architecture’s width rather than the depth of the network to reduce the computational cost and increase efficiency. On the other hand, Brem et al. [43] describes how the healing of foot ulcer is affected by taking into the consideration of pressure point. So far, most of the studies have used color-based image segmentation to segment the DFU without considering the inflammation factor as one of the prime features. In the present study, we used the fusion of colour and thermal features to segment the DFU so that the ulcer boundary could better delineate to assess the healing trajectory of the ulcer.

The authors acknowledge CSIR- Central Scientific Instruments Organisation, Chandigarh (India) and Postgraduate Institute of Medical Education and Research (PGIMER) Chandigarh (India) for their support and encouragement in carrying out this research work.