Centric Multi-Feature-Based Retrieval Approach for Fast Handling Large Medical Image Datase

Imaging techniques are a fundamental part of modern medicine; they are used to medically image the human body from the inside and outside. The first technique that comes to mind is X-ray, also called computed radiography (CR). While X-rays are the oldest and still the most widely used method, many different scanners, such as magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET), generate large amounts of data [1].

Regardless of the imaging modality, medical images can be digitally stored, accessed, and shared through an integrated system known as PACS (Picture Archiving and Communication System). This system includes software capable of aggregating data from various imaging technologies and supplementing it with descriptive metadata—such as the physician's name, patient ID, and diagnostic reports—providing a structured and efficient method for annotating, organizing, and retrieving image-related information for both clinical and research applications [2]. For instance, magnetic resonance imaging (MRI) files often contain detailed parameters like pulse sequence timing, flip angles, and acquisition counts [3].

Furthermore, modern PACS solutions are often integrated with Radiology Information Systems (RIS), enabling the management of complex, multidimensional image data. These systems utilize the standardized DICOM (Digital Imaging and Communications in Medicine) protocol to store and transmit both the raw image files and all relevant metadata concerning image acquisition procedures [3]. However, despite the advanced infrastructure, the images stored in PACS are not inherently useful for clinical interpretation or decision-making unless properly structured and annotated. For example, considering a repository of over 900,000 images lacking metadata or organization, without contextual information, extracting meaningful insights would require manually reviewing the entire dataset, which is highly impractical [4]. Since recent biomedical image datasets [5] contain multi-dimensional images (3D+time), analysis by computer vision and biomedical image processing algorithms may take longer. This dimensionality of medical images is the real performance issue, as multidimensional computational complexity increases with each additional dimension since the processing of image data must be performed accurately with all available computational power. Therefore, the biomedical image data generated daily requires algorithms that can efficiently process and analyze this data to produce meaningful insights.

Many retrieval algorithms and architectures have been adapted for 3D medical image retrieval, e.g. Deep Convolutional Neural Networks (CNNs) [6], Deep Similarity Learning for multimodal images [7], the algorithm developed [8] or the registration algorithm based on Compressive Sensing and Local Scale-Invariant Feature Transform (SIFT) [9]. However, a major limitation of the pairwise method, even when based on these algorithms, is the high computational cost incurred by exhaustive comparisons that scale poorly with increasing dataset size. Traditional retrieval pipelines often calculate similarity scores between a query and each entry in the dataset, resulting in significant delays that make such methods impractical for use in real-time or on a large clinical scale. Attempts to reduce these costs through downsampling, segmentation or subsampling [10-12] often result in a loss of accuracy and diagnostic precision.

Consequently, there remains a need for a retrieval framework that avoids the computational inefficiencies of pairwise comparisons, eliminates the need for data reduction techniques, and scales effectively with the growing volume of 3D medical image data. Our approach addresses these challenges through a centralised, multi-feature-based pipeline that supports efficient, accurate and timely retrieval without compromising data quality.

1.1 Problem formulation

Despite the advancements in medical processing, efficient retrieval of relevant medical images from large-scale datasets remains a significant challenge due to the high computational cost associated with traditional methods. Conventional approaches rely on direct image-to-image comparisons, which are inefficient for handling large volumetric medical datasets. Additionally, existing methods often lack an optimized, memory-efficient strategy for managing and indexing extracted features, leading to scalability limitations and may not fully utilize the potential of 3D and 4D data, leading to suboptimal feature extraction.

1.2 Major contributions

This study presents an effective retrieval framework based on 3D SIFT feature extraction [13] in combination with FAISS [14], which was chosen over other deep learning techniques and key-point methods such as HOG, HDFS [15] and SuperPoint [16] due to its feature descriptor based on a 64-dimensional descriptor, this descriptor provides higher efficiency than classical 128-dimensional and also for 3D medical image search, 3D SIFT is a GPU-optimised implementation that achieves about 7x speed improvement while maintaining robustness in MRI scans, making 3D SIFT both computationally efficient and robust to noise and intensity changes in 3D volumes resulting in similar image matching performance with a much larger memory footprint [17] and less time-consuming when dealing with large medical datasets to enhance both the storage and indexing of volumetric medical images. By replacing traditional image-to-image comparisons with a more efficient feature-to-feature matching strategy, the framework substantially reduces computational demands while preserving high retrieval. Tailored for real-time diagnostic use, the architecture supports high-performance querying and is well-suited for large-scale clinical and research imaging environments.

By tackling the key challenges associated with big medical data, the proposed framework delivers a robust and scalable solution for efficient image retrieval. Leveraging ranked 3D SIFT features for optimized storage and indexing, it significantly improves data organization and search precision, all while minimizing processing overhead. Its feature-driven architecture, combined with a high-capacity database, ensures fast and reliable access to relevant medical imagery, making it highly applicable to real-time clinical workflows, biomedical research, and secure data-intensive applications.

The remainder of this paper is organized as follows: Section II reviews related work and current strategies for improving medical image storage and retrieval. Section III details the design and components of the proposed framework. Section IV presents the experimental evaluation of the system, analyzing its performance in terms of retrieval speed, precision, and scalability. Section V concludes the paper by outlining limitations and suggesting directions for future enhancements.

The proposed method presents a lightweight, feature-centric biomedical storage and retrieval system aimed at improving the efficiency of feature extraction algorithms such as 3D SIFT for large-scale medical image queries. Designed to enhance both speed and accuracy, the framework also addresses the challenge of data scarcity in biomedical imaging by supporting scalable processing without requiring massive training datasets. It integrates three core components - feature extraction, indexing, and real-time matching- while optimizing overall processing time as a key performance indicator. Additionally, the system leverages features from both 2D and 3D deep learning models, enabling accurate and cost-efficient retrieval, making it highly suitable for advanced medical imaging and research applications.

Most machine learning and computer vision workflows explored in this study rely on a core computational framework for processing image queries against reference datasets, summarized by the following equation:

Processing Time=Fr Extraction Time + Comparison Time

Feature extraction time: the period needed to generate meaningful descriptors from medical images using techniques like SIFT, CNN-based embeddings, or hybrid methods.

Comparison time: the duration required to match these extracted features against those in the dataset, typically using similarity metrics such as Euclidean distance, cosine similarity, or domain-specific medical measures.

The proposed Centric Feature-Based Retrieval Approach introduces a two-stage pipeline—Indexing (offline) and Query (online/real-time)—to optimize medical image retrieval. By extracting and storing features only once during the indexing stage, the system eliminates the need for repeated dataset processing at query time. This design follows the principle of Separation of Concerns, enabling real-time performance by decoupling expensive computations from the user-facing query phase. Unlike traditional full pairwise methods that perform full pairwise comparisons at every query, the Centric strategy drastically reduces processing overhead while maintaining high retrieval accuracy. It is particularly well-suited for large-scale medical datasets, offering scalable, storage-efficient, and fast retrieval performance ideal for real-time clinical applications.

The main difference between the two approaches is explained in Figure 1.

1.png

Figure 1. Separation of concerns of centric vs full pairwise approach

This enhanced formulation ensures that the system not only extracts and compares features efficiently but also structures and stores extracted features in an optimized manner, facilitating rapid and scalable real-time retrieval. By incorporating indexing mechanisms, such as KD-trees, FAISS, or Using vector database, the proposed framework significantly improves query performance and scalability for large-scale medical imaging datasets as shown in Figure 2.

2.png

Figure 2. Bloc diagram for building an end-to-end medical image based on multi-features approach

The diagram above shows a typical workflow for indexing medical image features extracted from many mid-range and high-end biomedical datasets. Our process consists of two main stages line-dashed which are The Offline and Real-time data processing to speed up processing historically and archived medical data stored in original datasets.

The Offline processing stage consists of two blocks, the first one is crowling and importing images from one or more datasets (e.g., ultrasound, MRI, CT, and radiography) provided to Feature extraction algorithms (the 3D SIFT to extract keypoint descriptors) from the corpus of parsed images.

To go through the scarcity of biomedical datasets, medical images from TCIA dataset (e.g., from ultrasound, MRI, CT, and radiography), are provided to the 3D SIFT extraction method. As shown in the diagram, multiple public datasets, e.g., OASIS [34], ADNI [35], and HCP [36], can be used simultaneously for feature extraction to feed our feature database. As mentioned earlier, TCIA is a service that provides a large archive of medical cancer images that can be publicly downloaded. The data are organized as “collections”; typically, patient images are organized by a common disease (e.g., lung cancer), image modality, or image type (MRI, CT, digital histopathology). DICOM is the primary file format used by TCIA for radiologic imaging. Supporting data for images such as patient outcomes, treatment details, genomics, and expert analysis are also provided when available.

The second block addresses feature extraction using 3D SIFT and vectorization, a technique for converting raw data into numerical features that can be processed to produce what is known as a feature vector.3D SIFT -Rank is designed for feature extraction from 3D volumes. The main format is Nifti, but (.hdr,.nii.gz) or raw data (IEEE 32-bit float, Little Endian) are also supported.

The third block of the process is indexing. As the name suggests, indexing data refers to the process of creating a structured and organized reference system to efficiently retrieve and access specific information from a data set. Simply put, indexing helps to improve the speed and efficiency of data queries by allowing systems to quickly find and retrieve the desired information without having to search through the entire data set.

This improvement is achieved through a Java-based multi-feature wrapper that plays a central role in managing the flow between raw medical image ingestion, feature extraction and vector database indexing. This wrapper performs several automated tasks, such as error handling by automatically filtering out unsupported or corrupted medical images based on format and size constraints, as well as batch crawling, which allows the system to process medical volumes in optimized batches and what is more important is feature aggregation, where multiple feature vectors are encapsulated into a unified metadata object per image. These aggregated feature vectors are then passed in bulk to the FAISS-based indexing engine, allowing for a fast build-up of the similarity search index. This modular approach not only improves fault tolerance and scalability but also enables parallelism and pre-fetching mechanisms that significantly reduce latency and indexing time.

This modular and scalable structure allows for dynamic index updates each time a new image is processed, enabling real-time retrieval tasks such as classification, object recognition and clustering with significantly reduced latency, as illustrated in Figure 2.

5.1 Evaluating centric feature query-based stage

To rigorously evaluate our approach to medical image retrieval and similarity assessment, we conducted a series of experiments using benchmark datasets commonly employed in clinical research. These included the LIDC-IDRI dataset, consisting of 1609 2D thoracic CT scans from 214 patients for nodule detection, and two well-established 3D brain MRI datasets -OASIS and ²- which contain tumor annotations across multiple institutions. Additionally, the TCIA brain metastasis dataset was used, comprising 560 multimodal MRI volumes from 412 patients, annotated with over 5000 brain metastases, and available in both DICOM and NIfTI formats. These diverse, multimodal, and publicly available resources ensured clinical relevance and scientific reproducibility for our evaluation.

Our experimental study focused specifically on 3D volumetric medical data, performing controlled tests using both the 3D SIFT and 3D HOG feature extraction algorithms. The datasets were scaled progressively, ranging from as few as 1 image to as many as 500 NIfTI volumes, totaling approximately 93 GB in raw (.nii) format, or 15.9 GB when compressed (.nii.gz). For each configuration, we compared the performance of two retrieval strategies: the Centric Feature Query-Based approach, which separates offline indexing from real-time querying, and the traditional method, which performs full pairwise comparisons at query time. Across all configurations, we evaluated the feature extraction time, indexing overhead, and similarity search latency, under conditions designed to reflect realistic clinical use cases as shown in the Tables 1-12.

Our demonstration was primarily conducted on 3D medical imaging datasets, where the Centric Feature-Based Retrieval Approach consistently exhibited real-time query performance, even at the largest tested scales. Using 3D SIFT, the total processing time—including a one-time offline indexing step—was approximately 4807.67 seconds, with the query stage executing in under 10 seconds. Similarly, when applying 3D HOG, the system achieved even faster results: 3026.59 seconds for offline processing and ~5.10 seconds per query. In both cases, the traditional full pairwise method required significantly more time—over 10,721 seconds for SIFT and 7,322 seconds for HOG—underscoring the computational burden and poor scalability of full pairwise comparison strategies Tables 7-12.

Table 1. Traditional full pairwise comparisons at query time using ResNet50 model

Table 2. Indexing stage of centric feature-based extraction using ResNet50 model

Table 3. Query stage of centric feature-based extraction using ResNet50 model

Table 4. Traditional full pairwise comparisons at query time using VGG16 model

Table 5. Indexing stage of centric feature-based extraction using VGG16 model

Table 6. Query stage of centric feature-based extraction using VGG16 model

Table 7. Traditional full pairwise comparisons at query time using 3D SIFT

Table 8. Indexing stage centric feature-based extraction using 3D SIFT

Table 9. Query stage of centric feature-based extraction using 3D SIFT

Table 10. Traditional full pairwise comparisons at query time using 3D HOG

Table 11. Indexing stage of centric feature-based extraction using 3D HOG

Table 12. Query stage of centric feature-based extraction using 3D HOG

Table 13. Computing accuracy and consistency of traditional full pairwise strategy based on 3D SIFT

Table 14. Computing accuracy and consistency of centric similarity search strategy based on 3D SIFT

Table 15. Computing accuracy and consistency 3D HOG traditional full pairwise strategy

Table 16. Computing accuracy and consistency of centric similarity search strategy based on 3D HOG

A key advantage of the Centric architecture is the explicit separation between indexing and querying. The indexing phase, which is executed offline, is remarkably efficient (e.g. ~0.02 seconds), so that only the lightweight query phase needs to be executed in real-time. This decoupled design ensures low latency and scalable performance, making the method ideal for high-throughput clinical environments. After validating our approach for large 3D volumes, we extended our evaluation to 2D image datasets using deep learning-based feature extraction models (VGG16 and ResNet50) Tables 1-6 to demonstrate the generality and robustness of our framework for different modalities and architectures.

Having analysed the computational advantages of each approach, we now focus on another crucial aspect: the accuracy and consistency of the search results presented in the Tables 13-16.

5.2 Evaluating efficiency of feature search queries

To evaluate the accuracy and retrieval consistency of the two approaches, we conducted a comparative assessment on the same datasets of high-resolution 3D MRA brain scans. Feature extraction was performed using 3D SIFT and 3D HOG descriptors as shown in Tables 9 and 10, chosen for their robustness in preserving spatial and structural features. The Traditional method employed full pairwise comparisons with cosine similarity, while the Centric approach relied on approximate nearest neighbor (ANN) search using L2 distance for 3D HOG and cosine similarity for 3D SIFT. Despite differences in similarity metrics and numerical score scales, both methods retrieved the exact same Top 5 most similar images for each query. This validates that the Centric Feature-based approach maintains 100% retrieval fidelity when equivalent feature representations and comparable metrics are used.

While score values varied due to the nature of the distance measures—L2 distance in Centric queries producing lower-is-better values (e.g., 0.0000 to ~1.91), and cosine similarity in Traditional comparisons yielding higher-is-better results (e.g., 1.0000 to ~0.9905)—the relative ranking of retrieved results remained identical. This ranking consistency is crucial for content-based medical image retrieval, where diagnostic decisions may depend on precise match ordering. Consequently, the Centric approach offers not only superior computational efficiency but also retrieval accuracy equivalent to full pairwise methods, making it highly suitable for real-time clinical applications.

To consolidate the insights gained from the results presented above, we now provide a comprehensive summary of the experimental studies conducted across both 2D (ResNet50 and VGG16) and 3D (3D SIFT and 3D HOG) feature extraction models across a wide range of dataset sizes to assess scalability, efficiency, and retrieval performance focusing 3D algorithms like 3D SIFT for Summarizing experimental studies and compared two strategies: a traditional approach, which performs real-time pairwise comparisons for every query, and a Centric Approach, which decouples the pipeline into offline indexing and fast real-time querying stages. Among all models and settings, the 3D SIFT-based evaluation on a large 500 files of 6.5 GB volumetric brain MRA dataset highlighted the most significant performance distinctions between the two paradigms illustrated on Tables 13-16, Using 3D SIFT features, the Traditional approach required over 10,700 seconds per query (5081.41s for feature extraction + 5640.18s for comparison), making it impractical for real-time applications. In contrast, the Centric pipeline processed the same dataset in 4820.26 seconds total, with the majority (4807.64s) spent during a one-time offline indexing phase. At query time, only 12.59 seconds were needed for feature extraction, and FAISS similarity search executed in just 0.0002 seconds, enabling truly responsive queries. This design as shown in Figure 3 yields more than 55% reduction in total processing time, but more importantly, an 850× speedup in online execution—demonstrating that separating concerns between indexing and querying not only optimizes resource use but transforms the retrieval system into a scalable, real-time solution for medical imaging applications.

3.png

Figure 3. 3D SIFT accuracy and retrieval consistency comparison of traditional full pairwise and centric similarity search strategies

Experimental results highlight the effectiveness and practicality of the proposed Centric feature-based method. Its core strength lies in the architectural separation between offline indexing and online querying, which allows for rapid, real-time retrieval without the need to reprocess the entire dataset for each query. This design ensures seamless responsiveness, making the system highly suitable for real-time applications, including clinical and machine learning environments. The indexing process, performed only once and offline, enables the system to handle queries almost instantly, even on large-scale datasets.

In addition to its performance benefits, the method is both flexible and resource-efficient. It does not require prior image segmentation [10] and runs effectively on standard computing hardware, avoiding reliance on costly GPU setups Table 17. The indexing structure supports both visual and textual queries and can easily accommodate various feature extraction algorithms, whether replacing 3D SIFT with HOG or integrating GPU-accelerated methods for more demanding applications. Offline processing tasks can also be scheduled during low-demand periods, further optimizing system efficiency.

Table 17. Requirements and resources for experimental study

Importantly, the Centric approach also offers a significant advantage in terms of data privacy and security. Because it stores only extracted features rather than raw medical images, it minimizes the risk of exposing sensitive patient information. This feature-centric design reduces the need to handle or share original data directly, thereby enhancing compliance with privacy regulations and contributing to more secure medical data management. Altogether, the Centric approach presents a scalable, secure, and adaptable solution for managing and retrieving large volumes of medical imaging data in support of precision medicine and biomedical research.