Image Description Using the Relation Between Color and Texture in Retrieval Task

Four descriptors from the literature using both color and texture were analyzed: MSD, CMSD, SED and MIFH. Also, two descriptors present in the MPEG-7 standard used in CBIR systems were considered to compare the performance of the descriptors compared to descriptors that only use a low-level feature. A color descriptor “Color Layout Descriptor” (CLD) and a texture descriptor “Edge Histogram Descriptor” (EHD) were chosen. In order to evaluate the performance of the descriptors in Level 2 and Level 3, from the levels proposed by Eakins and Graham [29], four sets of images used in the literature were selected, three of them with conceptual classes, and one with object classes. Likewise, four metrics were selected, including the one proposed by the MPEG-7 standard.

2.1 Image sets and metrics

For this analysis we used three sets obtained from Corel Photo Gallery [30], the Dataset Corel-1k [31], which has 1,000 images divided into 10 classes such as people, beach, Rome, etc. The Dataset Corel-5k, contains 5,000 images divided into 50 classes such as bridge, art, crab, frozen, etc. And the image set “Corel Database for Content based Image Retrieval” [32], called in this work Corel-CBIR, which contains 10,800 images divided into 80 unbalanced concept groups, e.g., autumn, aviation, bonsai, castle, cloud, etc. Likewise, Caltech-101 [33], with a total of 9,146 images distributed in 101 object classes.

Four metrics, taken from the studies [8, 34], were selected to performance an objectively evaluation of the descriptors in image retrieval: Precision P, which is defined in Eq. (1); Recall R, defined in Eq. (2).

$P=|K|^{-1} \sum_{n=1}^{K} r\left(x_{n}\right)$            (1)

$R=|K|^{-1} R_{q}$        (2)

$P_{k}=|k|^{-1} \sum_{n=1}^{k} r\left(x_{n}\right)$                (3)

$A P_{q}=\left|R_{q}\right|^{-1} \sum_{n=1}^{K} P_{n} r\left(x_{n}\right)$           (4)

$r\left(x_{n}\right)= \begin{cases}1, & \text { if } x_{n} \text { is relevant }; \\ 0, & \text { otherwise. }\end{cases}$             (5)

$M A P=|Q|^{-1} \sum_{q=1}^{Q} A P_{q}$          (6)

Mean Average Precision MAP, which is defined with four equations Eqns. (3-6); and the metric proposed by the MPEG-7 standard, Average Normalized Modified Retrieval Rank , which can be defined in six equations Eqns. (6-12).

$\operatorname{Rank}_{k}=\left\{\begin{aligned} \operatorname{Rank}_{k}, & \text { if } \operatorname{Rank}_{k} \leq K_{q} ; \\ 1.25 \times K_{q}, & \text { otherwise } . \end{aligned}\right.$                 (7)

$K_{q}=\min \left\{4 \times R_{q}, 2 \times \max \left[R_{q}, \forall_{q}\right]\right\}$              (8)

$A V R_{q}=\sum_{k=1}^{R_{q}} \frac{R a n k_{k}}{R_{q}}$                  (9)

$M R R_{q}=A V R_{q}-0.5 \times\left[1+R_{q}\right]$          (10)

$N M R R_{q}=\frac{M R R_{q}}{1.25 \times k_{q}-0.5 \times\left[1+R_{q}\right]}$        (11)   

$A N M R R=|Q|^{-1} \sum_{q=1}^{Q} N M R R_{q}$         (12)

where, P_k is the precision to the image at position k, K the total number of retrieved images, x_n the retrieved image at position n, R_q is the total number of images in the class for query q, and Q is the total number of queries performed. In ANMRR, for the images belonging to the same class as the query image, Rank_k is assigned according to their position in the retrieval, punishing the evaluation with respect to K_q, where K_q is adjusted considering the number of images in the query class and the images in the classes of all queries performed. AVR_q can be defined as the average Rank for the query, MRR_q as a modified retrieval Rank, which is normalized to obtain NMRR_q, and finally, the average of all queries results in the ANMRR.

The four metrics result in a value between zero and one, which is considered as a percentage, except for the ANMRR metric that indicates a better retrieval a value close to zero, in the rest of the metrics the percentage closest to 100% is considered a better result.

2.2 Evaluation

For evaluation, we performed queries using only color images, so we eliminated grayscale images, we taken 100 of the 101 classes in Caltech-101. Likewise, we considered the six descriptors. In the case of the CMSD, we used our own implementation, because we detected that the author's code did not faithfully follow the model. In the case of the MIFH descriptor, we considered the best combination, since the authors do not present it in their work. In general, for the evaluations we taken 10 random images, as query images per class, giving a total of 100 queries per descriptor, in the case of Corel-1k; 500 with Corel-5k; 800 with Corel-CBIR; and 1,000 for Caltech-101. For practical purposes and considering that people seek to have a result in the first retrieves, the system was set at K=12, that means, the first 12 most relevant images of each query.

We analyzed the CMSD code provided by the authors, since throughout the analysis, we detected some discrepancies between the algorithm and the proposed model. In general, the discrepancies are centered on the omission or union of values, which caused the values 72 and 88 to be obtained with 0, and the value 78 to be contaminated with equivocally variables. Also, the use of the $\sin$ and $\cos$  function of Matlab in radians and not in degrees; and finally, a discrepancy in the conversion of cylindrical coordinates to cartesian coordinates. An example of the differences between the code provided and the model, is shown in Figure 1, where shows the histograms of image 716 in horse class from Corel-1k, the descriptor provided by the authors is set as “a”, and the corrected one following the model as “b”.

We decided to take the same image that is used as an example in the article where the descriptor is proposed, to find coincidences between the descriptor provided and the data shows in the author's paper. We found that the descriptor provided, which does not faithfully follow the model, delivers the same result shown in the images of their article, likewise, as shown in Figure 1. The feature vector obtained differs with respect to the corrected descriptor, so there is a possibility that the results obtained in their article were made with a version of the descriptor that does not follow the model faithfully. For this reason, we decided to evaluate both versions of the descriptor, to see if the discrepancies affected the performance of the descriptor.

1.png

Figure 1. Histograms of the descriptor CMSD

The results of the evaluation using the four metrics, with the Corel-1k, Corel-5k, and Caltech 101 image sets, are shown in Tables 1-3, where the best result is highlighted with “bold”. The evaluation showed that the descriptor correcting the discrepancies between the model and the algorithm, obtained better results on average, so it was decided to use this version of the algorithm, for the subsequent experiments.

Table 1. CMSD evaluation with Corel-1k

Table 2. CMSD evaluation with Corel-5k

Table 3. CMSD evaluation with Caltech-101

With respect to the MIFH descriptor, the literature did not establish the value of $\Delta_{x}$ and $\Delta_{y}$for obtaining the co-occurrence, as well as the type of sub-sampling performed, so an evaluation was performed with different combinations, using average (AVG); minimum (MIN); and maximum (MAX), for sub-sampling, and three combinations of $\Delta_{x}$ and $\Delta_{y}$, which gave a total of nine combinations. For the evaluation, the metrics and image sets Corel-1k (Table 4) and Corel-5k (Table 5) were used, from the evaluation, the combination that was considered the most stable and with the best results was taken, in this case the combination AVG-01.

Table 4. Evaluation of MIFH combinations with Corel-1k

Table 5. Evaluation of MIFH combinations with Corel-5k

2.png

Figure 2. Images in Caltech-101

Table 6. Results with Corel-1k

Table 7. Results with Corel-5k

Table 8. Results with Corel-CBIR

Table 9. Results with Caltech-101

3.png

Figure 3. Images in Corel-5k

The evaluation was carried out with the rest of the descriptors, with the adjustments recommended by the authors. The results are shown in Table 6-9. Where we observed that the CMSD and MIFH descriptors delivered better results in the Corel-1k, Corel-5k and Corel-CBIR datasets. However, in the Caltech-101 dataset, which has object categories, and some of the classes contain non-real images, i.e., drawings or cartoons, as shown in Figure 2. The descriptors of the MPEG-7 standard stood out, with EHD being the best evaluated. This could be since the descriptors proposed by the standard are oriented to retrievals in terms of shape, color, and textures, which are better defined in Caltech-101.

Apparently, retrieval with concept classes improves using descriptors that use more than one low-level feature, however, when retrieval is performed on images with object classes and with illustrations, descriptors representing one low-level feature performed better. Despite this, it would remain to analyze and test the results, using more sets of images, with object classes. In general, the most complicated classes for most of the descriptors are those referring to a specific place, since they contained a great diversity of contents with no apparent visual relationship, as well as classes with contents like another, as shown in Figure 3 which presents examples of images in the class airplanes and the class flying objects of Corel-5k.

2.3 Weakness detection

In the evaluation, we detected some problematic images and classes, some of the retrievals presented errors apparently unrelated to the query image such as images 1, 6, and 8 shown in Figure 4. Weakness detection focused on the best performing descriptors in the evaluation, in this case the descriptors CMSD and MIFH. To detect areas within the images that could affect the retrieval. We coded a system to select and cut parts of a query image, looking for a relationship between an area or part of the image and the erroneous images, trying to detect the reason for confusion and weaknesses in the descriptors. Figure 5 shows a retrieval performed with the system, where the recovered images are shown with a slice of the query image of Figure 4, placing as name the coordinates of crop.

4.png

Figure 4. Flower retrieval with CMSD

We detected in some retrieves that the confusion could be due to the information introduced by areas in the background. because in the queries performed with a crop of the background area, the unwanted images were retrieved in a better position, even in some situations, the retrieve got others similar unwanted images. However, in some cases we did not find a relationship between a specific area of the image and the unwanted image, so the confusion was not due to a specific area.

5.png

Figure 5. Retrieval using crop of flowers image

We found that the images could have a relationship in the direction or proportion of textures and color, despite not being distributed in the same way. i.e., Figure 6 despite not having an apparent visual relationship, have a similar perspective, and contain similar colors, when quantified, and despite having different textures, they maintain some proportions, in relation to texture-color. Likewise, it was observed that queries on saturated images or with a low variety of colors, i.e., with a poorly distributed color histogram, retrieved saturated images, even though they were not the same colors, as the case shown in Figure 7.

Looking for the possible weaknesses responsible for the unwanted image retrieved, an attempt was made to induce undesired retrievals by selecting images that had characteristics that could be cause an erroneous retrieval. An analysis of the images considered as problematic was performed, obtaining their difference histograms, as shown in Figure 8, for each of the values in the feature vector, with the intention of obtaining values within the vector that caused the confusion, or values that could be used to avoid it.

6.png

Figure 6. Beach retrieval with CMSD

7.png

Figure 7. Horse retrieval with MIFH

8.png

Figure 8. Histogram differences of horse query with MIFH

In the experiment we found that the values that caused confusion were often the values with zero in the color histogram. In the case of the CMSD the saturated images looked similarity in the color histogram since the only thing that differentiated them from the 72 color values were 6 to 10 values having a total of 62 to 66 values the same since there were no colors in that range. In the case of the MIFH descriptor, despite not having more than 54 color values, presented a similarly case, since by generalizing the color more, the colors that apparently was different, the quantification considered the same value.

To find the most or least discriminant values, we chose to calculate the dispersion measures of the feature vector. We used the Range Eq. (13), the standard deviation Eq. (14) and the coefficient of variation Eq. (15). Where X is the population; x each of its elements; N, the amount of data; and $\mu$ the average. With the dispersion measures we looked for values with wide ranges, a small standard deviation between class images and large between classes, and with a coefficient of variation greater than 26%. This was to detect the most discriminating values and conversely to detect values that were not very discriminating.

Range $=\max (X)-\min (X)$          (13)

$\sigma=\sqrt{\frac{\sum(x-\mu)^{2}}{N}}$            (14)

$C V=\frac{\sigma}{\mu}$              (15)

During the evaluation we found that the descriptor values in general did not behave in an equivalent way, varying their discrimination in each class, and apart from the null values, no specific value was detected that did not discriminate or discriminated more than the others. In the case of the MIFH descriptor, it was found that the values where the edges of the image were represented, from value 55 to 102, had values that were apparently not very discriminating and could even be causing confusion. In general, the last five values of each of the edge feature integration, i.e., the values [66, 67,…,70, 81, 82,…, 86, 98, 99,…, 102]. Since the majority images obtained a zero, and in some other cases only the values [1.3863, 1.0986, 0.69315], and with no apparent relationship between classes.

2.4 Discussion

Some of the errors are due to problems in the image sets, since there are classes that are very similar, or that have very varied contents, as is the case of the classes that belong to a specific region, for example, the Roma class. Likewise, there are images that could well belong to more than one class, or that have more information of another class than the one in which they are labeled. It appears that using more than one low-level feature of the image decreases its effectiveness in lower-level retrieval queries. In general, MIFH and CMSD can be considered as the best performing descriptors, however, they seem to be sensitive to the background of the images, as it introduces information that affects retrieval, more so in cases where the background is present in a higher percentage than the object of interest. As well as in cases where the background is present in a higher percentage than the object of interest. The descriptors also give great importance to color, in the CMSD descriptor 82% of the vector belongs to color, and in the case of MIFH, despite being less than 53%, the color histogram is directly obtained from the HSV color space, it means, depend entirely on the color of the image, not on a relation, unlike CMSD. On the other hand, the generalization of color by quantization causes that, apparently different colors are cataloged as the same, since from $2^{24}$ they become 72 and 54 colors, respectively. In addition, some values in the feature vectors of the descriptors seem to be unrepresentative, which could be causing erroneous retrievals.

The combined use of texture and color to describe images seems to improve the representation of images retrievals in conceptual classes, however, it does not seem to improve in object classes. In average, the descriptors CMSD and MIFH show better results than the rest. In spite of this, we can mentioned three possible weaknesses, related to unwanted images in the retrieval, or without an apparent relation to the query image: one of them is color, both its representation and its importance; sensitivity to background information; and finally, some weaknesses in the representation, since there are values within their vectors that are not very representative and despite the fact that the descriptors analyzed consider the relationship between two image features, unwanted images are still obtained very similar to those of a low-level descriptor, since they tend to give greater importance to a feature.

In the experiments, some variants perform better than the original descriptors, especially in level 2 retrievals. However, it would be necessary to study in which type of images they work better than the current ones, since they are still not stable enough to maintain the best result in all the sets, so their use would be recommended in specific cases. In addition, it is necessary to study and evaluate the variants with other combinations and adjustments, such as changing the quantification method. The modifying descriptors to address a weakness may affect their performance by generating new weaknesses, so it is not an easy task. Therefore, the area required research and information, since they could not only benefit image retrieval systems, but also any other system that requires image analysis. Since most systems currently use descriptors that represent a low-level feature and do not consider the relationships between them, losing relevant information that is not found directly in the image, information that could facilitate its analysis or relate more directly to high-level features.