Multi-Modal Medical Image Matching Based on Multi-Task Learning and Semantic-Enhanced Cross-Modal Retrieval

Within the medical domain, both images and descriptive texts serve as important data forms, offering detailed insights into patient conditions. Typically, medical images convey visual information about diseases, while descriptive texts elucidate diagnoses, treatment recommendations, and other pertinent textual descriptions. Thus, the effective matching and categorization of medical images and descriptive texts is deemed paramount.

The learning of inter-modal associations primarily delves into establishing connections between two distinct data modalities, such as images and texts. Such a learning approach assists in comprehending the mutual relationships between modalities, enabling cross-modal data matching and retrieval. Conversely, the learning of intra-modal associations is chiefly concerned with data relationships within a single modality. For instance, within the image modality, classification tasks might be centred on categorizing medical images into various disease types. Within the text modality, classification endeavours might focus on categorizing diseases into distinct classes based on descriptive texts. By concurrently engaging in both inter and intra-modal associative learning, knowledge can be garnered from two different perspectives, potentially enhancing the model's generalization capabilities and reducing the risk of overfitting. Figures 1 and 2 respectively depict the convolutional processing of medical images and their corresponding descriptive texts.

1.png

Figure 1. Convolutional processing of medical images

2.png

Figure 2. Convolutional processing of descriptive texts for medical images

Lower-level features closely resemble the original data and contain a plethora of detailed information, such as textures and colours in images, and vocabulary and basic semantics in texts. This information is deemed pivotal in preliminary classification and matching. For these lower-level features, the intrinsic structure of data can be better captured through self-supervised methods, facilitating effective representation learning even in the absence of explicit labels. In image-text matching tasks, a more meticulous capture of the correlations between the two modalities for these lower-level features becomes necessary. Soft bidirectional ranking loss ensures that both similarities and differences between the two modalities are aptly considered. Thus, for the low-level features of images and texts, self-supervised clustering loss and soft bidirectional ranking loss are respectively utilized for image-text classification and matching tasks.

Higher-level features predominantly capture abstract and higher-order information, like structures and object relations in images, and advanced semantics and context in texts. Such information proves crucial for intricate classification and matching tasks. For these higher-level features, the uniqueness and categorical characteristics of each instance become more vital during classification tasks. Instance loss ensures that, at this abstract level, each instance is correctly classified. Moreover, for these higher-level features in matching tasks, comparisons need to be made on a broader semantic scale, ensuring accurate matching of the advanced semantics of both modalities. Hence, for the high-level features of images and texts, instance loss and soft ranking loss are respectively employed for image-text classification and matching tasks. Figure 3 presents a schematic diagram of the image-text classification and matching task learning processes.

Learning visual and linguistic representations for medical images and their descriptive texts is both a complex and pivotal endeavour. Utilising a ResNet-50 pre-trained on ImageNet as the image encoder ensures a robust feature extraction capability right from the model's initiation, especially in capturing fundamental concepts and object characteristics in images. Extracting both high and low-level features ensures that information is harvested at various scales. While higher-level features tend to capture an image's holistic and abstract information, the lower-level features reflect the image's details and texture. This complementarity is believed to enhance the model's matching and classification accuracy. The selection of the output from the third residual module of text convolution as the low-level textual feature indicates an emphasis on mid-level textual data. Such features capture certain key nuances of textual data, vital for ensuring text richness and integrity.

3.png

Figure 3. Learning processes for image-text classification and matching tasks

In medical imaging, minute detail variations might signify entirely distinct medical conditions. Emphasis is placed on the individuality of instances by the instance loss function, ensuring that these subtle yet pivotal differences are captured. Such a function ensures that instances within the same category are drawn closer while those from different categories are kept distinct. Effective learning in the presence of noisy data is facilitated by this loss since it prioritizes the characteristics of individual instances over entire categories. Additionally, the distinction between instances that might appear similar but belong to different categories is enhanced, an aspect that becomes particularly salient in medical imaging where numerous conditions might bear striking visual resemblances. Within the instance loss function framework, medical images and their accompanying descriptive texts are treated as cross-modal data pairs, and an individual label, denoted by uf_g, is assigned. Assuming the image concept representation of the medical image and its descriptive text is represented by M^g_u, and the textual concept representation is denoted by M^g_y, the criteria "1{}" must satisfy 1{true}=1 and 1{false}=0. The transformation matrix of the fully connected layer is represented by Q^gÎE^v×2048, the number of all classification possibilities and that of uf_g is represented by O=[O¹, ..., O^v], and the weights of the fully connected layer are denoted by Q^g. The following definitions are given:

${{O}_{u,g}}=SOFTMAX\left( {{Q}^{g}}{{c}^{g}} \right)$            (1)

$M_{u}^{g}=-\sum\limits_{b=1}^{v}{1\left\{ u{{f}_{g}}=b \right\}\log O_{u,g}^{b}}$         (2)

${{O}_{y,g}}=SOFTMAX\left( {{Q}^{g}}{{y}^{g}} \right)$            (3)

$M_{y}^{g}=-\sum\limits_{b=1}^{v}{1\left\{ u{{f}_{g}}=b \right\}\log O_{y,g}^{b}}$                  (4)

Annotation of medical data is both costly and time-consuming. Utilization of unlabelled data for learning is facilitated by the self-supervised clustering loss, optimizing the usage of available data. Efforts are made to ensure that similar instances are clustered together, while different instances are separated. Reliance on labelled data is not required by this loss function, making it especially valuable in the realm of medical imaging where extensive labelled datasets might not be accessible. It is adept at capturing the inherent structure of data, which becomes particularly significant given the intricate and latent relationships potentially present in medical images and their descriptive texts. The k-means algorithm is employed by the self-supervised clustering loss function. This algorithm divides the sample data into j(j≤v) datasets, represented by A={A₁, ..., A_k}, with the volume of training set data denoted by v. Post encoding of medical images using ResNet-50, the resulting feature dataset is denoted as Z={z₁, ..., z_v}, where z_u$\in$E²⁰⁴⁸(u∈[1,v]). For the initialization of the clustering model, random initialization for the j cluster centers ω={ω₁, ..., ω_j} is necessitated, where ω_k$\in$E²⁰⁴⁸(k$\in$[1,j]). Assuming the cluster index assigned to sample z_u is represented by x_u, the optimization objective function for clustering is given as follows:

$K=\frac{1}{v}\sum\limits_{u=1}^{v}{{{\left\| {{z}_{u}}-{{\omega }_{{{x}_{u}}}} \right\|}^{2}}}$                  (5)

${{x}_{u}}=\underset{k}{\mathop{ARGMIN}}\,{{\left\| {{z}_{u}}-{{\omega }_{k}} \right\|}^{2}}$                 (6)

In order to minimize the previously mentioned objective function, the cluster center i_kmust be updated, and the update formula is:

${{\omega }_{k}}=\frac{\sum\limits_{u=1}^{v}{1\left\{ {{x}_{u}}=k \right\}{{z}_{u}}}}{\sum\limits_{u=1}^{v}{1\left\{ {{x}_{u}}=k \right\}}}$                     (7)

Specifically, an iteration termination condition of K≤1e^-5 was set in the experiments conducted. Constraints are also established for the visual feature class labels of medical images and their descriptive text modalities, resulting in the cluster labels X={x₁, ..., x_v}, also referred to as appearance labels uf_l. Analogous to the structure of the instance loss function, the image self-supervised clustering loss function is denoted by M^l_u, while the text self-supervised clustering loss function is denoted by M^l_y. Assuming the appearance features are represented by c^l and y^l, and the weight matrix of the fully connected layer is represented by Q^l$\in$E^j×2048, this layer is tasked with categorizing c^l and y^l into j categories. The probabilities of c^l and y^l belonging to different categories are represented by O_u,l$\in$E^j and O_y,l$\in$E^j respectively, yielding the following calculation formulas:

${{O}_{u,l}}=SOFTMAX\left( {{Q}^{l}}{{c}^{l}} \right)$             (8)

$M_{u}^{l}=-\sum\limits_{b=1}^{j}{1\left\{ u{{f}_{l}}=b \right\}\log O_{u,l}^{b}}$                (9)

${{O}_{y,l}}=SOFTMAX\left( {{Q}^{l}}{{y}^{l}} \right)$            (10)

$M_{y}^{l}=-\sum\limits_{b=1}^{j}{1\left\{ u{{f}_{l}}=b \right\}\log O_{y,l}^{b}}$             (11)

The hyper-parameter is represented by η, and the cosine similarity function is denoted as A(.,.). Positive sample pairs and negative sample pairs are denoted by (c^g_1o, y^g_1o) and (c^g_1b, y^g_1o) respectively. Traditional bidirectional ranking loss functions are:

$\begin{align}  & M_{RA1}^{g}\left( {{c}^{g}},{{y}^{g}} \right)=MAX\left[ 0,\eta -A\left( c_{1o}^{g},y_{1o}^{g} \right)+A\left( c_{1b}^{g},t_{1o}^{g} \right) \right] \\ & \begin{matrix}   \begin{matrix}   {} & {} & {} & {}  \\ \end{matrix} & {} & +MAX  \\  \end{matrix}\left[ 0,\eta -A\left( c_{1o}^{g},y_{1o}^{g} \right)+A\left( c_{1p}^{g},y_{1b}^{g} \right) \right] \\  \end{align}$                          (12)

$\begin{align}  & M_{RA1}^{l}\left( {{c}^{l}},{{y}^{l}} \right)=MAX\left[ 0,\eta -A\left( c_{1o}^{l},y_{1o}^{l} \right)+A\left( c_{1b}^{l},y_{1o}^{l} \right) \right] \\ & \begin{matrix}   \begin{matrix}   {} & {} & {} & {}  \\  \end{matrix} & {} & +MAX  \\    \end{matrix}\left[ 0,\eta -A\left( c_{1o}^{l},y_{1o}^{l} \right)+A\left( c_{1o}^{l},y_{1b}^{l} \right) \right] \\   \end{align}$                            (13)

However, within the actual dataset of matching medical images with descriptive texts, a significant majority of sample combinations might be mismatches. The difficult-negative mining strategy focuses on those negative samples currently deemed "difficult to differentiate" rather than all negative samples. Such a strategy can effectively prevent models from overly focusing on easily distinguishable samples, thus mitigating the risk of overfitting. For matching tasks between medical images and descriptive texts, models must consider the profound connections of both modalities. Such tasks involve not only recognizing and learning the correlation between images and texts but also managing a large number of negative samples and ambiguous matches. Against this backdrop, the use of the aforementioned loss function is deemed unsuitable. Assuming the mean value of all concept feature negative samples within the cluster where y^g_2o resides is represented by y^-g_2b, and the mean value of all appearance feature negative samples within the cluster where y^l_2o resides is represented by y^-l_2b, the improved bidirectional ranking loss functions are given as:

$\begin{align}  & M_{RA1}^{g}\left( {{c}^{g}},{{y}^{g}} \right)=MAX\left[ 0,\eta -A\left( c_{2o}^{g},y_{2o}^{g} \right)+A\left( c_{2b}^{g},y_{2o}^{g} \right) \right] \\ & \begin{matrix}   \begin{matrix}   {} & {} & {} & {}  \\   \end{matrix} & {} & +MAX  \\   \end{matrix}\left[ 0,\eta -A\left( c_{2o}^{g},y_{2o}^{g} \right)+A\left( c_{2o}^{g},\bar{y}_{2b}^{g} \right) \right] \\   \end{align}$                       (14)

$\begin{align}  & M_{RA2}^{l}\left( {{c}^{l}},{{y}^{l}} \right)=MAX\left[ 0,\eta -A\left( c_{2o}^{l},y_{2o}^{l} \right)+A\left( c_{2b}^{l},y_{2o}^{l} \right) \right] \\ & \begin{matrix}   \begin{matrix}   {} & {} & {} & {}  \\   \end{matrix} & {} & +MAX  \\    \end{matrix}\left[ 0,\eta -A\left( c_{2o}^{l},y_{2o}^{l} \right)+A\left( c_{2o}^{l},\bar{y}_{2b}^{l} \right) \right] \\   \end{align}$                      (15)

The BR loss function aims to minimize the distance between matched image and text pairs while maximizing the distance between unmatched pairs. Bidirectional ranking loss considers both Image-To-Text and Text-To-Image sorting relationships, ensuring bidirectional consistency in matches. By combining the two, not only can match accuracy be improved, but also the stability and robustness of the model's matching results can be enhanced. A soft ranking loss function was constructed by integrating the BR loss function and bidirectional ranking loss. Unlike traditional classification loss functions, the soft ranking loss function priorities the relative relationships between samples rather than the absolute classification of each sample. This function provides a larger margin of error for models, allowing a certain level of errors while still promoting correct ordering. It also enables models to recognize and learn varying degrees of matches, ranging from complete matches to complete mismatches, rather than a simple binary match/non-match decision. For concept and appearance features, the corresponding loss function expressions are provided below:

$M_{RA}^{g}=M_{RA1}^{g}+M_{RA2}^{g}$          (16)

$M_{RA}^{l}=M_{RA1}^{l}+M_{RA2}^{l}$            (17)

A medical image matching model based on multi-task learning has been proposed. By incorporating multi-task hierarchical convolutional networks, the algorithm can handle multiple modalities of medical images simultaneously, ensuring that their associative information is thoroughly explored, leading to more precise medical image matching. In Table 1, the performance of different image-text matching models on the task of matching medical images with descriptive text is displayed. The metrics R@1, R@5, R@10 represent the retrieval performance of the models, with higher values indicating superior performance. It can be seen that DeViSE exhibits moderate performance on the task of matching descriptive text and medical images, with an mR value of 63.5, indicating its overall average performance. VSRN, being a visual-semantic reasoning model, demonstrates improved performance on the text matching task, but it only achieves 9.6 for R@10 in the medical image, suggesting a potential outlier. On the whole, its average performance is recorded at 74.2. DPC is a dual-path convolutional image-text embedding model. It is observed that it exhibits commendable performance on both the text and medical image matching tasks, with an mR of 77.9, illustrating its balanced cross-modal performance. The model proposed in this research achieves the best results across all metrics. Notably, the R@10 for descriptive text reaches 96.6, suggesting that there's a 96.6% chance of finding the actual matching medical image among the top 10 retrieval results. Moreover, the R@1 for medical images is also notable at 51.8, implying that over half the top-matching retrieval results are correct. Overall, its average performance stands at 81.4, outperforming the other models.

From Figure 5, a clear contrast in performance between the single-task learning model and the multi-task learning model proposed in this research is evident on different retrieval metrics (R@1, R@5, R@10). It is observed that the proposed model, when tasked with matching medical images to descriptive text, outperforms the single-task learning model in both image-text retrieval and text-image retrieval. This strongly indicates the effectiveness and superiority of the proposed model. Outstanding performance is witnessed on the R@1 metric, suggesting high accuracy within the most relevant search results. As previously discussed, the model incorporates multiple loss functions, difficult-negative mining strategies, and a multi-modal multi-granularity semantic enhancement network. These design elements may be crucial factors in its outperformance over the single-task model.

In the Table 2, various cross-modal retrieval models and their performance in image-text matching tasks are presented. Two retrieval tasks, "Image-To-Text" and "Text-To-Image", are listed, with three evaluation metrics each: R@1, R@5, R@10. These metrics represent the hit rates in the top 1, 5, 10 search results, respectively. The highest score of 78.1 on the R@1 metric is achieved by the proposed model. Additionally, leading scores of 93.6 and 96.1 are recorded on R@5 and R@10 respectively. Similarly, for the "Text-To-Image" task, the highest scores on R@1, R@5, and R@10 are 57.9, 83.8, and 91.2 respectively. Thus, it can be concluded that the proposed model clearly excels when compared with other cross-modal retrieval models. This further attests to the advantages of the multi-modal multi-granularity semantic enhancement network in handling cross-modal retrieval tasks for medical images.

Table 3 displays the performance comparison of ablation models in cross-modal retrieval. The importance of each component of the model is evaluated by removing them one by one. It is observed that the model integrates various techniques for cross-modal retrieval, each playing a pivotal role in enhancing the overall performance. Performance deterioration is noted upon the removal of any individual component. Essential roles of CNN, Bi-GRU, and the similarity computation module in capturing the deep semantic relationships between images and text are identified. Although some metrics occasionally show a slight improvement in certain ablation experiments, it does not undermine the importance of the corresponding components. Instead, it suggests their significance in handling specific types of data or tasks. Collectively, the proposed model offers an efficient and robust cross-modal retrieval solution. The ablation experiments further confirm the significance of each component, establishing the rationale and efficacy behind the model's design.

5.png

Figure 5. Comparison of performance between the single-task learning model and the proposed model

6-1.png

(1) Txet-To-Image

6-2.png

(2) Image-To-Txet

Figure 6. Recall@1 for different retrieval sample sizes

Table 4 showcases the Recall@1 performance of the Image-To-Text and Text-To-Image tasks at different training epochs and retrieval sample sizes. As the training epochs increase, an improvement in the model's Recall@1 performance across different retrieval sample sizes is noted, confirming the model's adaptability and effectiveness over continuous training. High performance at specific retrieval sample sizes underscores the model's ability to maintain excellence across varying dataset sizes. Even though there's a slight drop in performance after certain training epochs, the model consistently showcases superior performance in most scenarios. This data further emphasizes the model's effectiveness and robustness across different retrieval tasks, particularly with varying sample sizes and training epochs.

Figure 6 depicts the change in Recall@1 performance for Text-To-Image and Image-To-Text tasks across training epochs. Both tasks display remarkable Recall@1 performance, particularly in the initial training epochs. While some decline in performance for certain sample sizes is noticed as training progresses, the model consistently delivers stable results across most scenarios. This reiterates the model's robustness and efficiency across datasets of different sizes.