Xianghe Zhu
© 2024 The authors. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
With the global promotion and application of Traditional Chinese Medicine (TCM), the identification and management of TCM materials have become critical issues that need to be addressed. Traditional methods for identifying TCM materials rely on manual experience and expert knowledge, leading to low efficiency and a high likelihood of errors. With the development of image processing technology, image-based classification and retrieval of TCM materials have gradually become a research hotspot. However, existing methods often encounter challenges such as insufficient classification accuracy and low retrieval efficiency when faced with the diversity and complexity of TCM material images. Therefore, how to effectively extract image features and improve the accuracy of classification and retrieval has become the central challenge in current research. Traditional image features, such as color, shape, and texture, are commonly used in the classification and retrieval of TCM materials. However, these features are often unable to fully reflect the diversity and detail of the materials, especially when distinguishing between morphologically similar materials. Although deep learning techniques have made breakthroughs in the field of image processing, the application of deep learning in TCM material image classification still faces many challenges due to insufficient data and annotation. A combination of technologies, including superpixel segmentation, feature point extraction, and clustering encoding, provides an effective approach to improving classification and retrieval performance and warrants further research. A kind of feature enhancement-based method for the classification and retrieval of TCM material images was proposed in this study, consisting of four main components. First, fine image segmentation was performed using the Simple Linear Iterative Clustering (SLIC) superpixel segmentation technique to extract features; second, an initial classification method based on feature points was used to perform coarse classification of the TCM material images; third, clustering algorithms were employed to encode features and perform initial sorting; and finally, the image retrieval results were optimized through reordering based on the initial sorting. Experimental results demonstrate that the methods effectively enhance the classification accuracy and retrieval efficiency of TCM material images.
TCM materials, image classification, image retrieval, superpixel segmentation, feature extraction, clustering algorithm, reordering
TCM, as an integral part of Chinese cultural heritage, has gradually attracted widespread attention and research globally [1-6]. The variety and diversity of TCM materials, combined with their numerous medicinal forms and complex traditional knowledge backgrounds, make their identification, classification, and retrieval particularly challenging [7-11]. With the rapid development of digital technologies, image processing has become an increasingly important tool in the study of TCM materials, especially in the field of image classification and retrieval. Effectively extracting useful features from a large number of TCM material images for accurate classification and rapid retrieval has become a pressing issue in this domain.
The research on TCM material image classification and retrieval holds significant academic and practical value. First, accurate image classification aids in improving the efficiency of automated identification and diagnosis of TCM materials, providing crucial support for clinical applications in TCM [12-19]. Second, image retrieval technologies can facilitate the efficient management of large-scale TCM material databases, enabling academic researchers, medical professionals, and general consumers to quickly locate the required medicinal material information [20-24]. Furthermore, with the global dissemination of TCM, cross-lingual and cross-cultural image retrieval technologies will also contribute to the international spread of TCM culture, providing technological support for the modernization and globalization of TCM.
However, existing image classification and retrieval methods for TCM materials still exhibit some limitations [25-28]. Most current research focuses on image classification through traditional feature extraction methods, such as color, texture, and shape features. These methods, however, often fail to provide sufficient discriminative power when dealing with the complex and varied images of TCM materials, especially when the materials have similar morphological features, resulting in low classification accuracy. Additionally, the reordering performance of existing image retrieval methods is suboptimal, as they often rely on simple similarity measures and fail to fully consider the multi-level features of image content, leading to inaccuracies and imprecision in retrieval results. Therefore, how to combine multi-dimensional features, improve feature extraction and classification strategies, and enhance the performance of image classification and retrieval remains a critical challenge in current research.
This study proposes a kind of feature enhancement-based method for the classification and retrieval of TCM material images, aimed at improving the accuracy and efficiency of image classification and retrieval through innovative image processing and feature enhancement techniques. The research consists of four main parts. First, a feature extraction method based on SLIC superpixel segmentation was proposed to refine image processing and enhance the accuracy of feature representation. Second, an initial classification method based on feature points was employed to perform coarse classification of medicinal material images, laying the foundation for subsequent fine classification. Third, clustering algorithms were used to encode image features and perform initial sorting, providing a preliminary reference for retrieval. Finally, based on the initial sorting results, further image reordering optimization was conducted to achieve more precise image retrieval for TCM materials. Through the integration of these methods, this study can fully exploit the deep features of images in TCM material classification and retrieval, improving classification accuracy and retrieval efficiency. The approaches hold significant academic value and promising application prospects.
Figure 1 presents examples of the classification of TCM material images. The objective of this research is to enhance the classification accuracy and retrieval efficiency of TCM material images, thereby addressing the challenges associated with the identification and management of TCM materials. To achieve this goal, four primary steps were undertaken to progressively improve image processing, ensuring that each step contributes effectively to the subsequent one. First, the SLIC superpixel segmentation technique was applied to provide refined segmentation of TCM material images. Superpixels partition the image into regions of uniform size with clear boundaries, effectively reducing computational complexity while better capturing the local structural features of the image. This lays a solid foundation for subsequent feature extraction and classification tasks. Through this step, more representative local features can be extracted, enabling a more detailed and discriminative feature description. Furthermore, based on the extracted superpixel features, an initial classification method based on feature points was employed. The purpose of this step is to achieve coarse classification of the material images by utilizing these prominent feature points. This approach effectively reduces category confusion, allowing subsequent fine classification and retrieval to be performed within a smaller candidate space, thereby improving the overall efficiency of the system. Next, clustering algorithms were applied to encode the image features and perform initial sorting based on the feature points and preliminary classification results. This step further optimizes the organization of image features, providing more reliable initial sorting results for efficient retrieval. Finally, a retrieval reordering step was conducted based on the initial sorting results, further enhancing retrieval accuracy. This step adjusts the final retrieval results by considering deeper image features and sorting strategies according to the specific requirements of user queries.
Figure 1. Classification examples of TCM material images
The images of TCM materials often exhibit rich color variations and complex texture features, and morphological similarities frequently exist between different types of materials. To address this, the SLIC superpixel segmentation algorithm was employed to enhance the feature extraction process of the images. Specifically, in this study, a five-dimensional feature vector was constructed in the LAB color space and the xy coordinate space, using color, spatial position, and texture information to measure the similarity between pixel points. This approach ensures both the accuracy and efficiency of the superpixel segmentation under multidimensional features. The specific steps of SLIC superpixel segmentation include initializing seed points, calculating similarity metrics, and iterating the optimization process. In TCM material images, the segmentation regions were first initialized by selecting representative seed points. The selection of seed points was based on the local color and texture information of the image to ensure that each superpixel accurately reflects the local features. Then, a five-dimensional feature vector was used to measure the similarity between adjacent pixels. The pixels were assigned to corresponding superpixels based on the color and spatial location differences. This clustering-based approach effectively groups similar regions within the same superpixel, thereby enhancing feature compactness and distinguishability. Finally, after multiple iterations of optimization, the superpixel boundaries gradually converged, achieving an ideal segmentation result.
Given the complexity and diversity of TCM material images, two feature extraction methods, Red, Green, and Blue (RGB) and Local Binary Pattern (LBP), were utilized to comprehensively describe the color, texture, and other information of each superpixel block. RGB features, as fundamental color features in image processing, represent the color characteristics of a region by calculating the average RGB values of each superpixel. In TCM material images, the materials often display varying layers and shades of color, such as differences in the color of the skin, interior, and edges of the materials. These subtle color variations are crucial for the identification of TCM materials. By extracting the average RGB values of each superpixel region, these color features can be captured to some extent, and the three-dimensional RGB feature vector provides direct color information for subsequent classification and retrieval tasks.
In contrast to the RGB features, which describe color, LBP features focus on extracting local texture characteristics of the image, which are crucial for capturing surface texture, details, and morphological variations of the materials. The appearance of TCM materials typically presents complex texture structures, such as different texture features in the bark, leaves, and roots of the materials. These subtle texture details play a key role in distinguishing between different types of TCM materials. LBP features are generated by locally comparing the pixels within each superpixel block, creating a binary pattern, which is then converted into a numerical value, ultimately forming a high-dimensional texture feature histogram. In this study, 59 distinct LBPs were selected, and the LBP values for each superpixel block were statistically processed and normalized, resulting in a 59-dimensional LBP feature vector. This feature vector effectively captures the local texture variations in TCM material images, enhancing the description of surface details.
In the classification and retrieval processes of TCM material images, the combination of RGB and LBP features not only improves the representation of color information but also enhances sensitivity to texture details. Through this multi-feature extraction method, a more comprehensive description of the multidimensional information in TCM material images can be achieved, providing a richer feature foundation for subsequent material classification and retrieval tasks.
In this study, the initial classification step based on feature points plays a crucial role, particularly when handling complex and diverse TCM material images. The objective of this phase is to effectively cluster the local features (RGB and LBP) of the images, grouping similar regions together to lay the foundation for subsequent image retrieval reordering and fine classification. Specifically, the k-means clustering method and Fuzzy C-Means (FCM) clustering algorithm were adopted to cluster the extracted RGB and LBP features separately, generating two independent visual dictionaries (codebooks). These codebooks represent the clustering information of color and texture features in TCM material images. Through this clustering approach, superpixel blocks with similar colors or textures can be grouped into the same category, thus achieving initial classification of the images. This process helps reduce confusion between categories in TCM material images, especially when there are certain similarities in appearance, such as when different types of herbs exhibit similar colors or textures, thereby enhancing the efficiency and accuracy of subsequent retrieval and classification processes.
The specific implementation steps for the initial classification of TCM material images using the k-means clustering algorithm are as follows: a) The RGB and LBP features of each image’s superpixel block were extracted. These features represent the color and texture information of the TCM material image. For each feature space, the number of initial cluster centers (codebook size) was set, and the cluster centers were initialized through the k-means algorithm. b) The algorithm was used to calculate the distance from each pixel point to all the cluster centers, assign each pixel to the nearest cluster center, and update the positions of the cluster centers until convergence was reached or a predefined number of iterations was completed. Through this process, each superpixel block in the image was classified into different groups, forming the initial classification result based on RGB and LBP features. c) These cluster centers served as the basis for the initial classification of TCM material images, effectively grouping regions with similar colors and textures into a single category, thus providing a simplified and efficient feature representation for subsequent fine classification and retrieval. Mathematically, assuming a set of V pixel points, A={a1,a2,...,av}, representing the TCM material image to be clustered, the objective is to find J cluster centers, oj={z1,z2,...,zj}. The location of the u-th cluster center, lu, is given by ∑au/vu, where vu represents the number of pixels in the u-th cluster, and the distance from au to lu is denoted as f(au,lu). The target function is expressed as:
$d\left(o_j\right)=\sum_{u=1}^j \sum_{a_u \in z_u} f\left(a_u, l_u\right)$ (1)
The core idea of the FCM clustering algorithm is to assign the feature points of an image to multiple clusters based on fuzzy membership, addressing the limitations of traditional hard clustering algorithms that cannot precisely separate similar features. First, the RGB and LBP features of the superpixel blocks in the TCM material images were extracted. These features represent the color and texture information of the material images. Next, the FCM algorithm was applied to cluster these features. The process began by initializing the membership of each superpixel block for each cluster center. Unlike the hard clustering method used in k-means, FCM allows each superpixel block to have partial membership to multiple cluster centers, providing more flexibility in image classification. This enables a more accurate description of the subtle differences in color and texture within TCM material images. The membership values of each pixel range between 0 and 1, representing the degree of membership to each class. The sum of all membership values for each pixel is equal to 1. The expression for the implementation process is as follows:
$\sum_{u=1}^z i_{u k}=1, \forall k=1, \ldots, v$ (2)
Subsequently, the FCM algorithm was used to calculate the center of each fuzzy group. Based on the membership and the distance of each feature point, the positions of the cluster centers were updated, and the objective function was optimized iteratively until it converged to a minimum value or reached a predefined threshold. This process involves the continuous updating of the membership and the fuzzy cluster centers, ultimately resulting in cluster centers that effectively represent the color and texture features of the TCM material images. Let iuk range from 0 to 1. The center of the u-th cluster is denoted as zu, and the distance between the u-th cluster center and the k-th pixel point is given by fuk=||zu-ak||. The objective function is expressed as:
$K\left(I, z_1, \ldots, z_z\right)=\sum_{u=1}^z K_u=\sum_{u=1}^z \sum_k^v i_{u k}^l f_{u k}^2$ (3)
The necessary condition for the above equation to reach its minimum value is given by:
$\begin{gathered}\bar{K}\left(I, z_1, \ldots, z_z, \eta_1, \ldots, \eta_v\right)=K\left(I, z_1, \ldots, z_z\right) +\sum_{k=1}^v \eta_k\left(\sum_{u=1}^z i_{u k}-1\right)=\sum_{u=1}^z \sum_k^v i_{u k}^J f_{u k}^2+\sum_{k=1}^v \eta_k\left(\sum_{u=1}^z i_{u k}-1\right)\end{gathered}$ (4)
The derivatives of all input variables were computed as follows:
$z_u=\frac{\sum_{k=1}^v i_{u k}^l a_k}{\sum_{k=1}^v i_{u k}^l}$ (5)
$i_{u k}=\frac{1}{\sum_{j=1}^z\left(\frac{f_{u k}}{f_{j k}}\right)^{2 /(l-1)}}$ (6)
Furthermore, image encoding was performed in this study. The fundamental principle behind this process is the feature extraction from TCM material images, followed by clustering and encoding techniques in the RGB and LBP feature spaces, which generate a high-dimensional vector that describes the color and texture features of each TCM material image. Initially, superpixel block features were extracted from the image in the RGB and LBP feature spaces. Then, clustering algorithms such as K-means or FCM were applied to generate two codebooks, which represent the visual dictionaries in the color and texture spaces of the image. For each superpixel block, quantization encoding methods were employed in both feature spaces, mapping it to the corresponding codebook. This results in two independent encodings. The position of each superpixel block in the codebook was determined by calculating the Euclidean distance between the superpixel block and its associated cluster center, followed by updating the corresponding statistical values. This process eventually generated an encoding vector that reflects the overall distribution of the image. The encoding is represented by a statistical histogram, i.e., a V×V matrix, where the elements in the matrix indicate the distribution of the image across different cluster centers. Let the encoding value of a superpixel block at the corresponding position in the V×V codebook be denoted as zuk. The encoding values of the superpixel block in the RGB and LBP codebooks at their respective positions are denoted as ZEu and ZMk, respectively. For each superpixel block, the encoding is represented as:
$z_{u k}=\left\{\begin{array}{llll}1 & z_{E u}=1 & \text { or } & z_{M k}=1 \\ 0 & \end{array}\right.$ (7)
This matrix not only preserves the color and texture feature information of the image but also effectively describes the detailed features of the TCM material image, thereby improving the accuracy of image classification and retrieval. By using this encoding method, fast retrieval and initial sorting of TCM material images in an image library can be achieved, enhancing the efficiency of subsequent fine classification and matching tasks. Figure 2 illustrates a schematic of the TCM material image encoding process.
Figure 2. Schematic of TCM material image encoding
After encoding TCM material images, initial sorting is needed. This is based on similarity calculations between images using the Euclidean distance metric. In RGB and LBP feature spaces, TCM images are converted into feature encodings via quantization. The Euclidean distance between the query image and the encoded images is calculated, assessing color and texture similarity. Images are sorted based on their distance from the query image, with closer images ranked higher. This initial sorting quickly filters the most similar images, providing a foundation for further classification and retrieval tasks.
5.1 Image retrieval reordering based on a weighted Support Vector Machine (SVM) classifier
Although the image ranking obtained through Euclidean distance in the initial sorting can filter out images that are similar to the query image, the retrieval result accuracy needs to be further improved. To enhance the precision of retrieval results, the reordering process was introduced. During the reordering stage, the top Jo images, which are closest to the target image, were selected as positive samples, while the bottom Jv images, which are further from the target image, were selected as negative samples. To ensure the effectiveness of the reordering process, a weighted SVM classifier was introduced to further improve the accuracy of TCM material image retrieval. The key strategy involves generating multiple SVM classifiers through repeated training, and then combining these classifiers to form a strong classifier. This strong classifier was used in the image retrieval process to predict all images in the image library, assessing the similarity between each image and the query target image. By weighting and sorting the prediction results, the image ranking can be more accurately adjusted, thus achieving effective reordering of the initial sorted results.
For the sample set (a,b), u=1,2,...,v, where au represents the sample vector and bu∈{-1,1} denotes the corresponding sample label. The objective of the SVM is to find a classification plane that maximizes the margin between the categories. The objective function expression is given by:
$\begin{array}{ll}\underset{q, y}{\operatorname{MIN}} & \frac{1}{2}\|q\|^2, \\ \text { s.t. } & \left.b_u\left(\left(q \cdot a_u\right)+y\right)+1\right) \geq 1, u=1, \ldots, m\end{array}$ (8)
By introducing slack variables and penalty parameters, the objective function was modified as follows:
$\begin{array}{ll}\underset{q, y}{M I N} & \frac{1}{2}\|q\|^2+Z \sum_{u=1}^l \zeta_u, \\ \text { s.t. } & \left.b_u\left(\left(q \cdot a_u\right)+y\right)+1\right) \geq 1-\zeta_u, u=1, \ldots, m\end{array}$ (9)
The weighted SVM classifier for TCM material image retrieval reordering was primarily implemented through the boosting algorithm, which combines multiple weak classifiers into a strong classifier. Specifically, the boosting algorithm was used to train several base classifiers by randomly sampling training samples of TCM material images. Each base classifier learned a specific subset of image features, thus compensating for the limitations of other classifiers. During the training process, the boosting algorithm assigned weights to each base classifier based on its prediction error, thereby optimizing the performance of the weak classifiers. Ultimately, these weighted classifiers were combined into a strong classifier, which was used to re-rank the TCM material images in the image library to improve the accuracy and relevance of the image retrieval.
5.2 Image retrieval reordering based on the multi-instance algorithm
Due to the complexity and detailed features of TCM material images, traditional image retrieval methods may fail to fully account for the diversity and complexity of these images. In the initial image retrieval ranking phase, although feature encoding and Euclidean distance calculations allow for preliminary ranking, the diversity of features and potential noise within the images may lead to misclassification or improper positioning of certain images. Specifically, when dealing with TCM material images, relying solely on global image features for ranking can be influenced by local image region features, leading to insufficient consideration of certain aspects of the image’s characteristics. Therefore, a multi-instance learning algorithm was chosen to perform a reordering of TCM material images in this study. In the multi-instance learning framework, each image was treated as a bag containing multiple instances, and the overall label of each bag was predicted, allowing for a more comprehensive evaluation of the features of each TCM material image. In this way, although some instances within the image may significantly differ from the query image, the label of the entire bag can effectively guide the reordering process, ensuring that the final returned images are more aligned with the overall characteristics of the query image.
Let the feature of the TCM material in f-dimensional space be represented by a. If a TCM material dataset is represented as {(A1,y1),(A2,y2),...,(Av,bv)}, a bag is denoted by Au, where the TCM material examples within the bag are {au1u,au2u,...,auvu}, and the number of TCM material examples in the bag Au is denoted by vu. A TCM material example is represented by auk, and the class label of Au is denoted by bu, with values of -1 or +1. The following equation gives the function expression for multi-instance learning:
$\begin{gathered}d_{L U M}\left(A_u\right)= \begin{cases}+1, & \exists d\left(A_{u k}\right)=+1 \\ -1, & \forall d\left(A_{u k}\right)=-1\end{cases} \\ \left(1 \leq u \leq v, 1 \leq k \leq v_u\right)\end{gathered}$ (10)
In the context of TCM material image retrieval reordering, the mi-SVM algorithm, primarily used in this study, learned the overall label of each bag and utilized the feature information from multiple instances within the bag to improve retrieval accuracy. For positive bags, at least one positive sample was included, meaning that part of the bag contains features similar to those of the query image. For negative bags, only negative samples were included. By analyzing multiple instances within each bag using mi-SVM, it is possible to accurately distinguish which images are similar to the query image and which are not. This method effectively avoids errors based on a single feature dimension and, through a comprehensive judgment of the instances within the bag, further enhances the accuracy of TCM material image retrieval. The objective function is as follows:
$\begin{aligned} & \underset{\left\{b_u\right\}}{\operatorname{MIN}} \underset{q, y, \zeta}{\operatorname{MIN}} \quad \frac{1}{2}\|q\|^2+Z \sum_u \zeta_u, \\ & \text { s.t. } \forall u: b_u\left(\left\langle q \cdot a_u\right\rangle+y\right) \geq 1-\zeta_u, \quad \zeta_u \geq 0\end{aligned}$ (11)
The implementation steps of the multi-instance algorithm for TCM material image retrieval reordering are outlined as follows: a) In the initial ranking phase, the top Jo images, which are closest to the target image, were selected as positive samples, while the bottom Jv images, which are farther from the target, were selected as negative samples. The features and similarity information of these images form the preliminary foundation for the ranking process. b) Subsequently, the images were divided into positive and negative bags. Each bag contained multiple instances, with the positive bag containing at least one positive sample, and the negative bag consisting entirely of negative samples. The key to constructing positive and negative bags is the random selection of a certain number of images to ensure that the instances within the bag reflect the diversity of the image database. This process provides sufficient training samples for the subsequent multi-instance algorithm, thereby enhancing the accuracy of the image retrieval reordering. c) The positive and negative bags were initialized, and the label for each instance was set. The initial label for instances in the positive bags was positive, indicating that the image is similar to the target image, while the initial label for instances in the negative bags was negative. d) SVM was used to train the instances within each bag, constructing a universal classifier. During the training process, the decision function value for each instance, du=<q,au>+y, was computed, and the label of the instance, du, was updated to adjust the performance of the classifier. For instances in the positive bag, the SVM algorithm iteratively computed the decision values and updated the labels of each instance until the labels no longer changed. This step ensures that the classifier can accurately differentiate between positive and negative samples and progressively optimizes the classification results. e) After each update, the instance with the maximum decision value, which corresponds to the most accurate classification result, was selected as the label and its corresponding label was updated until convergence. f) By repeating the above process, multiple weak classifiers were trained on several positive and negative bags, and their respective classification results were combined to re-rank all TCM material images. g) During the image retrieval phase, a certain number of images were randomly selected from the positive and negative bags, and the multi-instance algorithm was applied to re-label the images. In this process, the combination of multiple positive and negative bags and the training process effectively captured the complex features of TCM material images, especially in aspects such as texture, color, and shape, allowing for more precise retrieval than traditional methods. h) The outputs of all trained classifiers were combined with weighted averaging to generate a strong classifier. This strong classifier, based on the predicted results of each image, can precisely re-rank the image database. The final ranking was based on the classification probability values of the images, and this ranking process significantly enhanced retrieval accuracy. It optimized retrieval performance, especially when addressing the complexity and similarity of TCM material images, and improved the relevance between the images and the query target. A flowchart of the TCM material image retrieval reordering process is shown in Figure 3.
Figure 3. Flowchart of TCM material image retrieval reordering
Figure 4. Image processing effects of TCM materials before and after feature enhancement
In the experimental section, the SLIC superpixel segmentation technique was first employed to process TCM material images, and a comparison was made with traditional full-image feature extraction methods. The experimental results in Figure 4 demonstrate that, after SLIC superpixel segmentation, the features of the local regions in the images were represented more finely. Compared to the traditional methods, the precision of feature representation in the images was significantly improved. The above experimental results indicate that the application of the feature enhancement method in the classification and retrieval of TCM material images significantly improved both the accuracy and efficiency of image processing. By performing fine-grained processing of the local regions in the image, the SLIC superpixel segmentation technique effectively enhanced the representational power of the image features, avoiding the global errors that may occur with full-image feature extraction methods. This approach ensured that the detailed information of the images was better preserved.
According to the data shown in Figure 5, the size of the codebook significantly affects the average precision of TCM material images. For images with a size of 50×50, as the number of returned images increased, the precision gradually decreased from 0.75 to 0.47. For 100×100 images, the precision decreased from 0.79 to 0.48. In the case of 200×200 images, the precision reached its highest value of 0.82 when the number of returned images was 10, and it gradually declined as the number of returned images increased, reaching a minimum value of 0.54. For images of size 250×250, the initial precision was 0.79, but as the number of returned images increased, the precision dropped to 0.51. Overall, as the image size increased, the average precision improved, but in the case of increasing numbers of returned images, the precision generally showed a downward trend. This decline was particularly noticeable for smaller image sizes (e.g., 50×50), where the precision decreased more significantly with an increase in the number of returned images.
Figure 5. Effect of codebook size on average precision of TCM material images
According to the data shown in Table 1, the image retrieval reordering method proposed in this study, based on the weighted SVM classifier, significantly improved the precision of TCM material image retrieval. For different categories of materials, the variation in precision with increasing numbers of returned images showed different trends. For example, in the plant category, the precision for roots decreased from 0.8562 at the top 10 images to 0.7852 at the top 50 images, showing a decline but still maintaining a relatively high level. In contrast, for stems and leaves, the precision exhibited a more significant decline, with the precision for stems decreasing from 0.5784 at the top 10 images to 0.4126 at the top 50 images, and for leaves, from 0.7236 at the top 10 images to 0.4652 at the top 50 images. Flowers, however, showed outstanding retrieval results, with precision remaining relatively high across all returned numbers, decreasing from 0.9326 at the top 10 images to 0.8456 at the top 50 images. For the animal category, precision for mammals remained consistently high, particularly at 1.1125 for the top 10 images, indicating very high retrieval accuracy for this category. Insects and aquatic animals showed fluctuating precision, but aquatic animals performed better overall, with a precision of 0.9785 at the top 10 images and 0.8236 at the top 50 images. For the ore and metal categories, precision remained relatively high, especially for metals, where precision was 0.9752 at the top 10 images and remained at 0.8895 at the top 50 images. Fungi exhibited significant differences in precision, with Ganoderma showing lower precision, particularly at 0.4126 for the top 50 images, whereas mushrooms displayed more stable precision, with 0.8236 at the top 10 images and 0.6785 at the top 50 images.
Table 1. Retrieval precision of TCM material images using the first retrieval method for each category
|
Category |
Precision |
|||||
|
Top 10 |
Top 20 |
Top 30 |
Top 40 |
Top 50 |
||
|
Plant category |
Root |
0.8562 |
0.8562 |
0.8326 |
0.8125 |
0.7852 |
|
Stem |
0.5784 |
0.5236 |
0.4652 |
0.4462 |
0.4126 |
|
|
Leaf |
0.7236 |
0.6124 |
0.5326 |
0.5123 |
0.4652 |
|
|
Flower |
0.9326 |
0.9256 |
0.9158 |
0.8566 |
0.8456 |
|
|
Animal category |
Mammal |
1.1125 |
1.1125 |
0.9852 |
0.9852 |
0.9852 |
|
Insect |
0.7895 |
0.6658 |
0.6125 |
0.5326 |
0.4896 |
|
|
Aquatic |
0.9785 |
0.9546 |
0.9152 |
0.8895 |
0.8236 |
|
|
Mineral category |
Ore |
0.8895 |
0.8465 |
0.7895 |
0.7452 |
0.7215 |
|
Metal |
0.9752 |
0.9785 |
0.9236 |
0.8895 |
0.8895 |
|
|
Fungi category |
Ganoderma |
0.5456 |
0.5126 |
0.4512 |
0.4256 |
0.4126 |
|
Mushroom |
0.8236 |
0.7789 |
0.7456 |
0.7236 |
0.6785 |
|
Table 2. Retrieval precision of TCM material images using the second retrieval method for each category
|
Category |
Precision |
|||||
|
Top 10 |
Top 20 |
Top 30 |
Top 40 |
Top 50 |
||
|
Plant category |
Root |
0.8256 |
0.8326 |
0.8256 |
0.8256 |
0.7785 |
|
Stem |
0.5741 |
0.5124 |
0.4523 |
0.4215 |
0.4213 |
|
|
Leaf |
0.71213 |
0.5263 |
0.5562 |
0.5124 |
0.4895 |
|
|
Flower |
0.9236 |
0.9236 |
0.9125 |
0.8795 |
0.8562 |
|
|
Animal category |
Mammal |
0.9895 |
0.9852 |
0.9825 |
0.9862 |
0.9852 |
|
Insect |
0.7546 |
0.6452 |
0.5785 |
0.5123 |
0.4652 |
|
|
Aquatic |
0.9623 |
0.9523 |
0.9125 |
0.8895 |
0.8235 |
|
|
Mineral category |
Ore |
0.8542 |
0.8326 |
0.8123 |
0.7541 |
0.7152 |
|
Metal |
0.9632 |
0.9452 |
0.9125 |
0.8795 |
0.8456 |
|
|
Fungi category |
Ganoderma |
0.5412 |
0.4785 |
0.4452 |
0.4126 |
0.4236 |
|
Mushroom |
0.8152 |
0.7795 |
0.7326 |
0.7152 |
0.6785 |
|
According to the data shown in Table 2, the image retrieval reordering method based on the multi-instance algorithm significantly improved the retrieval precision in several categories. In the plant category, the precision for roots was 0.8256 at the top 10 images and 0.7785 at the top 50 images, showing relatively stable precision. The precision for stems, however, decreased significantly, from 0.5741 at the top 10 images to 0.4213 at the top 50 images, indicating that the features of images in this category were more ambiguous and difficult to distinguish. For leaves, the precision was 0.7121 at the top 10 images, but fluctuated as the number of returned images increased, ultimately decreasing to 0.4895 at the top 50 images. Flowers exhibited stable and excellent retrieval results, with precision remaining high at 0.9236 at the top 10 images and declining only slightly to 0.8562 at the top 50 images. In the animal category, mammals consistently maintained high precision across all returned numbers, with 0.9895 at the top 10 images and 0.9852 at the top 50 images, indicating high retrieval accuracy for these images. The precision for insects decreased from 0.7546 at the top 10 images to 0.4652 at the top 50 images, reflecting challenges in feature representation for this category. The precision for aquatic animals decreased from 0.9623 at the top 10 images to 0.8235 at the top 50 images, showing a steady decline. For the mineral category, both ores and metals demonstrated good retrieval precision, particularly metals, which achieved 0.9632 at the top 10 images and maintained a high precision of 0.8456 at the top 50 images. In the fungi category, Ganoderma exhibited weak retrieval precision, with 0.5412 at the top 10 images and 0.4236 at the top 50 images. Mushrooms showed moderate performance, with precision decreasing from 0.8152 at the top 10 images to 0.6785 at the top 50 images, indicating less stability in retrieval accuracy for this category.
The data in Figure 6 illustrates the precision trends of three different TCM material image retrieval methods as the number of returned images increases. The precision of the initial ranking-based retrieval method gradually decreased as the number of returned images grew, from 0.82 at the top 10 images to 0.67 at the top 50 images. This indicates that the initial ranking method experiences a significant decrease in precision when handling a large number of returned results. In contrast, the image retrieval reordering method based on the weighted SVM classifier demonstrated more stable retrieval accuracy. The precision at the top 10 images was 0.83, and it remained high at 0.69 at the top 50 images, with a relatively small decline in precision. This suggests that the weighted SVM classifier can effectively enhance the accuracy of retrieval results, especially when the number of returned images is large, maintaining high retrieval precision. The image retrieval reordering method based on the multi-instance algorithm exhibited results that fell between the two other methods. The precision at the top 10 images was 0.823, and it decreased to 0.688 at the top 50 images. This trend was similar to that of the first retrieval method, but with a slightly lower precision, indicating that it is less capable when handling complex image features. The experimental results demonstrate that the performance differences among the three methods reflect their respective strengths in image retrieval reordering. The initial ranking-based method shows a significant decline in precision as the number of returned images increases, suggesting certain limitations in handling the complexity of image features. In comparison, the retrieval method based on the weighted SVM classifier is currently the most effective, providing stable and efficient retrieval results across varying numbers of returned images. This method is suitable for most TCM material image retrieval tasks.
Figure 6. Comparison of average precision for different TCM material image retrieval methods
Table 3. Performance comparison of different TCM material image retrieval methods
|
Category |
Based on Color, Texture and Shape |
SPM-SIFT |
HOG-LBP |
SIFT-LBP |
Proposed Retrieval Method 1 |
Proposed Retrieval Method 2 |
|
|
Plant category |
Root |
0.48 |
0.62 |
0.58 |
0.56 |
0.85 |
0.54 |
|
Stem |
0.32 |
0.47 |
0.48 |
0.57 |
0.51 |
0.52 |
|
|
Leaf |
0.36 |
0.46 |
0.56 |
0.42 |
0.62 |
0.62 |
|
|
Flower |
0.61 |
0.93 |
0.91 |
0.92 |
0.92 |
0.92 |
|
|
Animal category |
Mammal |
0.93 |
0.98 |
0.94 |
0.97 |
1.12 |
0.98 |
|
Insect |
0.48 |
0.58 |
0.52 |
0.58 |
0.67 |
0.65 |
|
|
Aquatic |
0.61 |
0.83 |
0.82 |
0.82 |
0.96 |
0.95 |
|
|
Mineral category |
Ore |
0.72 |
0.62 |
0.52 |
0.67 |
0.54 |
0.84 |
|
Metal |
0.42 |
0.38 |
0.36 |
0.45 |
0.95 |
0.94 |
|
|
Fungi category |
Ganoderma |
0.52 |
0.57 |
0.55 |
0.52 |
0.51 |
0.48 |
|
Mushroom |
0.435 |
0.636 |
0.618 |
0.642 |
0.778 |
0.779 |
|
The data presented in Table 3 demonstrated that the two retrieval methods proposed in this study exhibit significant advantages over other traditional methods for different TCM material image retrieval tasks. For the plant category, the retrieval accuracy for root images achieved by the proposed first method in this study reached 0.85, which is notably higher than methods based on color, texture, shape and Spatial Pyramid Matching with Scale-Invariant Feature Transform (SPM-SIFT), while other methods generally performed poorly for this category. The retrieval accuracy for stem images was more comparable, with the proposed methods showing a slight advantage over other methods. For leaf images, both retrieval methods in this study achieved a stable precision of 0.62, while traditional methods showed greater variation in performance for this category. The retrieval accuracy for flower images was high across all methods, but the proposed methods still maintained a leading position. For the animal category, particularly for mammal images, the image retrieval reordering method based on the weighted SVM classifier demonstrated exceptional performance, achieving a precision of 1.12, significantly higher than the other methods. The retrieval accuracy for insect and aquatic animal images also performed well with the methods proposed in this study. In particular, the retrieval accuracy for aquatic animal images using the proposed methods was 0.96 and 0.95, respectively, showing an improvement over other methods. For the mineral category, the retrieval accuracy for metal images was high with the proposed methods, while for ores, the retrieval accuracy in the weighted SVM classifier-based reordering method was 0.54, still outperforming traditional methods. Among fungi, mushroom images achieved a retrieval accuracy of 0.779 in both methods, demonstrating excellent retrieval performance, while the accuracy for Ganoderma images was relatively lower, especially with the multi-instance algorithm-based image retrieval reordering method, which underperformed compared to other methods.
Table 4. Comparison of results for different TCM material image retrieval methods on synthetic and real image test sets
|
Model |
Based on Color, Texture and Shape |
SPM-SIFT |
HOG-LBP |
SIFT-LBP |
Proposed Retrieval Method 1 |
Proposed Retrieval Method 2 |
|
|
Synthetic images |
PSNR |
12.25 |
14.36 |
14.23 |
14.26 |
16.25 |
24.85 |
|
SSIM |
0.625 |
0.715 |
0.7265 |
0.7236 |
0.5698 |
0.9125 |
|
|
EI |
65.23 |
72.65 |
71.59 |
73.62 |
82.52 |
82.36 |
|
|
PCQI |
0.958 |
0.963 |
0.968 |
0.962 |
1.235 |
1.125 |
|
|
UIQM |
2.235 |
2.458 |
2.652 |
2.623 |
3.215 |
3.125 |
|
|
Entropy |
7.125 |
7.213 |
7.225 |
7.128 |
7.789 |
7.895 |
|
|
Real images |
PSNR |
0.8125 |
0.6652 |
0.8795 |
0.8893 |
0.8741 |
0.9125 |
|
SSIM |
0.8125 |
0.6659 |
0.8892 |
0.8895 |
0.8123 |
0.9152 |
|
|
EI |
73.26 |
67.23 |
72.31 |
81.25 |
82.69 |
82.34 |
|
|
PCQI |
0.978 |
0.956 |
0.971 |
0.982 |
1.214 |
1.023 |
|
|
UIQM |
2.785 |
2.325 |
2.784 |
2.895 |
2.898 |
3.215 |
|
|
Entropy |
7.235 |
7.125 |
7.326 |
7.458 |
7.514 |
7.568 |
|
Based on the experimental results presented in Table 4, the feature enhancement-based image classification and retrieval methods for TCM materials proposed in this study exhibit outstanding performance on both synthetic and real image test sets. In the synthetic image test set, the first method proposed in this study achieved a Peak Signal-to-Noise Ratio (PSNR) of 16.25, a Structural Similarity Index Measure (SSIM) of 0.5698, an Enhancement Index (EI) of 82.52, a Patch-based Contrast Quality Index (PCQI) of 1.235, and an Underwater Image Quality Measure (UIQM) of 3.215, all of which are significantly superior to those of other methods, particularly in the PCQI and UIQM metrics, highlighting its substantial advantage in image quality enhancement. In comparison, the second proposed method achieved a PSNR of 24.85, an SSIM of 0.9125, an EI of 82.36, a PCQI of 1.125, and a UIQM of 3.125. It also demonstrated very strong performance, especially surpassing all other methods in SSIM and PSNR, indicating its remarkable improvement in structural similarity and visual quality. In the real image test set, although the performance of all methods declined, the first proposed method maintained a PSNR of 0.8741, an SSIM of 0.8123, an EI of 82.69, a PCQI of 1.214, and a UIQM of 2.898, still outperforming traditional methods, particularly showing strong robustness in the EI and UIQM metrics. The second proposed method also exhibited excellent performance, with a PSNR of 0.9125, an SSIM of 0.9152, an EI of 82.34, and a UIQM of 3.215, achieving the best results on the real image test set. It notably outperformed other methods in both SSIM and PSNR, reflecting its exceptional performance in complex real image environments.
The kind of feature enhancement-based image classification and retrieval method for TCM materials proposed in this study effectively improves the classification accuracy and retrieval efficiency of TCM material images by integrating techniques such as SLIC superpixel segmentation, feature point classification, clustering encoding, and image reordering. Specifically, the novelty of this study lies in the refinement of image feature extraction through superpixel segmentation, combined with preliminary classification based on feature points and subsequent optimization steps, which collectively enhance the recognition and retrieval capabilities for TCM material images. Experimental results demonstrate that the proposed retrieval methods significantly outperform traditional methods across various performance metrics. Notably, in tests with synthetic and real images, the retrieval method based on the weighted SVM classifier delivers the best image quality and retrieval performance, proving its effectiveness and robustness in practical applications.
However, despite the outstanding performance of the proposed methods in most experiments, certain limitations remain. First, although good results were achieved on synthetic and real images, the methods' performance slightly declined for certain specific image types, such as ores and Ganoderma species, indicating that there is still room for optimization in feature representation and classification accuracy. Second, when handling complex and highly diverse images, the proposed retrieval methods may still be affected by background noise, lighting changes, and image quality. Therefore, further improvements are needed to enhance the robustness of the model, particularly in adapting to high-noise or low-quality images. Future research could focus on the following aspects: first, further optimizing image feature extraction algorithms, especially during the superpixel segmentation phase, and exploring more deep learning-based feature enhancement methods to improve the extraction of detailed image features. Second, combining more image processing techniques with deep convolutional neural networks could be explored to further enhance the accuracy of classification and retrieval, particularly in terms of robustness in complex backgrounds. Additionally, future research could investigate the integration of multimodal data with image data to construct more intelligent TCM material retrieval systems. These improvements could further advance the intelligence of TCM material image classification and retrieval, providing more efficient and comprehensive solutions for the precise identification and utilization of TCM materials.
This paper was supported by Doctoral Research Launch Project of Hubei University of Chinese Medicine (Grant No.: 2023ZXB027) and Collaborative Education Project for Industry-University Cooperation, Ministry of Education (Grant No.: 230806030281526).
[1] Wang, X.H., Zhang, J.Q., Kong, Q., Tu, W.Z., Chen, L.M., Zhao, Y.H. (2023). Application of integrated traditional Chinese and western medicine in treatment of systemic sclerosis: A case report. Explore, 19(3): 463-468. https://doi.org/10.1016/j.explore.2022.06.001
[2] Wang, L.H., Cheng, F.C., Chiang, C.P. (2024). A preliminary study on dental practitioners of traditional Chinese medicine in Taiwan during the early days of Japanese colonial period. Journal of Dental Sciences, 19(1): 688-690. https://doi.org/10.1016/j.jds.2023.09.028
[3] Yang, X. (2023). The inheritance and development of Chinese narrative medicine practice to the philosophical wisdom of traditional Chinese medicine. Chinese Medicine and Culture, 6(2): 164-174. https://doi.org/10.1097/MC9.0000000000000066
[4] Law, S., Lo, C., Han, J., Leung, A.W., Xu, C. (2022). Is the traditional Chinese herb “Bombax Malabaricum” a natural anticancer medicine? Journal of Pharmacopuncture, 25(2): 145-147. https://doi.org/10.3831/KPI.2022.25.2.145
[5] Yu, M., Pei, X. (2024). International standard of Traditional Chinese Medicine techniques: The operating specification for Traditional Chinese Medicine enema (2022). Journal of Traditional Chinese Medicine, 44(5): 1035-1043. https://doi.org/10.19852/j.cnki.jtcm.2024.05.010
[6] Rong, N., Zhao, Y., Rong, P., Zhang, H., Huang, L. (2019). Cherish civilization roots, draw on other civilizations, be forward-thinking: Globalization of Chinese medicine. Chinese Science Bulletin-Chinese, 64(26): 2672-2676.
[7] Li, H., Zhang, B., An, X., Ma, X., Shen, J. (2020). Statistical analysis for user group of opposing traditional Chinese medicine in Weibo. International Journal of Computer Applications in Technology, 64(1): 108-113. https://doi.org/10.1504/IJCAT.2020.111090
[8] Mist, S.D. (2019). Modeling the complexities of traditional Chinese medicine diagnosis. Journal of Alternative and Complementary Medicine, 25(11): 1083-1084. https://doi.org/10.1089/acm.2019.0347
[9] Wong, R.W., Rabie, A.B.M. (2006). Traditional Chinese medicines and bone formation—A review. Journal of Oral and Maxillofacial Surgery, 64(5): 828-837. https://doi.org/10.1016/j.joms.2006.01.017
[10] Wong, K.Y., Phan, C.M., Chan, Y.T., Yuen, A.C.Y., et al. (2024). A review of using Traditional Chinese Medicine in the management of glaucoma and cataract. Clinical and Experimental Optometry, 107(2): 156-170. https://doi.org/10.1080/08164622.2023.2246480
[11] Bartosińska, J.P., Pietrzak, A., Szepietowski, J., Dreiher, J., Maciejewski, R., Chodorowska, G. (2011). Traditional Chinese medicine herbs—Are they safe for psoriatic patients? Folia Histochemica et Cytobiologica, 49(2): 201-205. https://doi.org/10.5603/FHC.2011.0027
[12] Lim, J.W., Chee, S.X., Wong, W.J., He, Q.L., Lau, T.C. (2018). Traditional Chinese medicine: Herb-drug interactions with aspirin. Singapore Medical Journal, 59(5): 230-239. https://doi.org/10.11622/smedj.2018051
[13] Zhang, X.P., Shi, Y., Zhang, L. (2007). Progress in the study of therapeutic effects of traditional Chinese medicine and extracts in treating severe acute pancreatitis. Jop, 8(6): 704-714.
[14] Gan, X., Shu, Z., Wang, X., Yan, D., et al. (2023). Network medicine framework reveals generic herb-symptom effectiveness of traditional Chinese medicine. Science Advances, 9(43): eadh0215. https://doi.org/10.1126/sciadv.adh0215
[15] He, P., Deng, K., Liu, Z., Liu, D., Liu, J.S., Geng, Z. (2012). Discovering herbal functional groups of traditional Chinese medicine. Statistics in Medicine, 31(7): 636-642. https://doi.org/10.1002/sim.4146
[16] Liu, Y.H., Ming, K.J., Sai, C.L., Khang, G.N. (2007). Synergistic effect of traditional Chinese medicine. Asian Journal of Chemistry, 19(2): 867-882.
[17] Lin, W.S., Chan, W.C.L., Hew, C.S. (1995). Superoxide and traditional Chinese medicines. Journal of Ethnopharmacology, 48(3): 165-171. https://doi.org/10.1016/0378-8741(95)01297-Q
[18] Zhou, J. (2009). New understanding of the basic theory of traditional Chinese medicine. Chinese Journal of Integrative Medicine, 15(1): 7-12. https://doi.org/10.1007/s11655-009-0007-y
[19] Chen, T., Zhou, X.Z., Zhang, R.S., Zhang, L.W. (2012). Discovery of regularities in the use of herbs in Chinese medicine prescriptions. Chinese Journal of Integrative Medicine, 18(2): 88-92. https://doi.org/10.1007/s11655-011-0860-6
[20] Chen, K.C., Lu, R., Iqbal, U., Hsu, K.C., et al. (2015). Interactions between traditional Chinese medicine and western drugs in Taiwan: A population-based study. Computer Methods and Programs in Biomedicine, 122(3): 462-470. https://doi.org/10.1016/j.cmpb.2015.09.006
[21] Wang, Y., Wang, Q., Li, C., Lu, L., Zhang, Q., Zhu, R., Wang, W. (2017). A review of Chinese herbal medicine for the treatment of chronic heart failure. Current Pharmaceutical Design, 23(34): 5115-5124. https://doi.org/10.2174/1381612823666170925163427
[22] Li, H., Jiang, H., Liu, J. (2017). Traditional Chinese medical therapy for erectile dysfunction. Translational Andrology and Urology, 6(2): 192-198. https://doi.org/10.21037/tau.2017.03.02
[23] Wang, S., Hu, Y., Tan, W., Wu, X., et al. (2012). Compatibility art of traditional Chinese medicine: From the perspective of herb pairs. Journal of Ethnopharmacology, 143(2): 412-423. https://doi.org/10.1016/j.jep.2012.07.033
[24] Hou, J., Yao, C., Li, Y., Yang, L., et al. (2023). A MS-feature-based medicinal plant database-driven strategy for ingredient identification of Chinese medicine prescriptions. Journal of Pharmaceutical and Biomedical Analysis, 234: 115482. https://doi.org/10.1016/j.jpba.2023.115482
[25] Lv, W.J., Wang, X.F. (2020). Overview of hyperspectral image classification. Journal of Sensors, 2020: 4817234. http://doi.org/10.1155/2020/4817234
[26] Singh Gill, H., Singh Khehra, B. (2020). Efficient image classification technique for weather degraded fruit images. IET Image Processing, 14(14): 3463-3470. https://doi.org/10.1049/iet-ipr.2018.5310
[27] Liu, Y.Z., Zhu, Z.Z., Ma, F. (2021). Review of hyperspectral image classification based on feature fusion method. Laser & Optoelectronics Progress, 58(4): 0400004. http://doi.org/10.3788/LOP202158.0400004
[28] Huu, Q.N., Ha, N.T.T., Quoc, T.N. (2012). An efficient content based image retrieval method for retrieving images. International Journal of Innovative Computing Information and Control, 8(4): 2823-2836.
