Assessing Semantic Similarity Measures and Proposing a WuP-Resnik Hybrid Metric for Enhanced Arabic Language Processing

The purpose of this section is to conduct a comparative evaluation of different measures of semantic similarity and assess their suitability for use in both WN and AWN. In this comparison, we will consider the differences between WN and AWN. The chosen measures will then be applied to an Arabic dataset and their results will be analyzed. The aim is to identify the measure that exhibits the best performance for automated tasks.

3.1 Arabic dataset benchmark

The dataset has been used in this study is the Arabic dataset benchmark created by Faza et al. [40]. The dataset was created following the same methods as the English dataset benchmarks for semantic similarity. The dataset was created in two stages. In the first stage, the Arabic word pairs set was selected and translated into Arabic using an English-Arabic dictionary. Two sets of Arabic noun pairs ranging from high similarity of meaning (HSM) to medium similarity of meaning (MSM) and low similarity were generated. In the second stage, the human similarity rate for word pairs was specified. 60 participants from various Arabic countries were asked to rank the set of 70 Arabic word pairs gathered in the first stage in order of importance. They ranked each word pair on a five-point scale ranging from 0.0 (unrelated) to 4.0 (the same). The dataset is important for evaluating semantic similarity between Arabic words.

Table 2 presents a sample of the data benchmark used in our study.

Table 2. A snippet of the benchmark dataset

The column labeled "N" is represented by the following abbreviation: The numbering of the word.

3.2 Applicable measures on AWN

As mentioned earlier, most semantic similarity measures, including feature-based measures and corpus-dependent information content measures cannot be used on AWN due to its limitations. Table 3 illustrates the reasons why certain common semantic similarity measures are not applicable to AWN.

Table 3. Assessing the suitability of traditional semantic measures on AWN

"Measure" is represented by the following abbreviation: SP: The Shortest Path, PM: PATH measure, WP: Wu & Palmer’s, F: Faaza, LC: Leakcock &Chodorow, Li: Li, R: Resnik, J: Jiang, L: Lin, M: Meng, MF: Most of feature-based measures, Zo: Zouaghi, Zh: Zhou and GA: Ghandi Aldiery.

3.3 Choosing synsets in AWN

We discussed the difficulty of selecting synsets in AWN, which led us to use two methods for selecting synsets: automatic selection by the program and manual selection. To ensure accurate results, we primarily chose synsets automatically, but some were manually corrected. In all comparisons, the same synset was chosen for each term to eliminate any variations that may arise from using different synsets. We specifically selected the synset of a concept instead of its synonyms. It is possible that our results may differ from those of other studies because we prioritized the selection of specific synsets over the general ones.

In AWN's file, each synset has a unique name, but some synsets may have multiple values, making it difficult to locate the correct synset for a given concept. For example, in the Figure 3, if we search for the word "سفر", we may find multiple synsets, and while one may be correct, the other may provide a better similarity score. In our study, we present two results: one represents the best possible outcome, while the other represents the most realistic outcome for automated work, given the lack of a reliable method for determining the best synset.

Table 4. The selected synset for the first experiments (using the best possible synsets)

image005.png

Figure 3. A snippet of AWN's xml file

3.4 Using conventional measures with AWN and WN

The aim of these experiments is to compare several widely used measures that can be applied to both AWN and WN. The purpose is to identify the WN and measure combination that produces superior outcomes.

3.4.1 Experiment 1: Comparison of measures on AWN with synset selection for optimal results

Selecting synset. In this comparison, we only considered synsets that yield superior outcomes, regardless of whether they are the correct synsets or not.

Table 4 reveals that numerous synsets are missing in AWN, such as "مشعوذ", "مشفى", and "موقد". Moreover, several pairs yield a similarity score of 1, indicating that both terms belong to the same synset, and some terms have no synset at all. However, as mentioned earlier, selecting synsets in this manner may lead to many errors. For instance, in pair number 4, the synset "xaAdim_n1AR" was selected for the concept "عبد", but in pair 24, the synset "Eabod_n1AR" was selected for the same concept. Although this approach may improve results, it lacks a synset selection methodology. In this experiment, we chose the synset that results in a smaller error based on the provided data.

Result. We evaluated six measures, as shown Table 5.

Table 5. Result of the first experiment (applying similarity measures on AWN using the best possible synsets)

The first row is represented by the following abbreviations: N: the numbering of the word, HN: Human rating, SP: Shortest path, W: WuP measure, Li: Li measure, F: faza, A: Aldieryand LC: Leakcock & Chodorow.

The Table 6 shows the correlation and mean squared error for each measure in this comparison.

qqtu_pian_20231108160156.png

Figure 4. The correlation between human ratings and similarity measures scores in AWN experiment using the best possible synsets

Among the six measures utilized, the li measure had the highest correlation value, as shown in Figure 4, and the lowest mean squared error, indicating that it generated better similarity scores. It is unsurprising that the shortest path had the lowest correlation because it only takes into account the distance between concepts.

Table 6. The correlation and MSE of the measures in the first experiment

3.4.2 Experiment 2: Comparison of measures on AWN with synset selection of correct synsets

Selecting synset. For this experiment, we only used the right synsets for each concept, as shown in Table 7, even if they do not produce the best results, so the correlation value decreased in this experiment.

Table 7. The selected synset for the second experiments (Right synsets)

The column labeled "N" is the abbreviation of the numbering of the word, and "HR" is the abbreviation of Human Rating

Result. Table 8 indicates that numerous synsets, such as the term "قدح" which shares the same synset as "كأس", are still missing in AWN. Additionally, a noteworthy observation is the significant contrast in the degree of similarity between "قرية" and "ريف"; in the previous experiment, their similarity score was 1, but in this experiment, it dropped to zero since their correct synsets do not belong to the same hierarchy in AWN.

Table 8. Result of the second experiment (applying similarity measures on AWN using the correct synsets)

The first row is represented by the following abbreviations: N: the numbering of the word, HN: Human rating, SP: Shortest path, W: WuP measure, Li: Li measure, F: faza, A: Aldieryand LC: Leakcock & Chodorow.

We observe a more substantial discrepancy between the human ratings and the measurement outcomes in this experiment compared to the previous one, indicating an elevated MSE and a reduction in correlation. Table 9 demonstrates that the ranking of similarity measures is not significantly different from the previous experiment, as the li measure continues to yield the most favorable results in terms of both correlation and MSE.

We can observe that all similarity measures exhibit a decrease in correlation and an increase in MSE, highlighting the impact of synset selection on achieving accurate results. The Table 8 presents a more realistic outcome for automated similarity measures, providing a clearer picture of the extent of variation between each measure (Figure 5).

Table 9. The correlation and MSE of the measures in the second experiment

qqtu_pian_20231108160204.png

Figure 5. The correlation between human ratings and similarity measures scores in AWN experiment using the correct synsets

3.4.3 Experiment 3: Comparison of traditional measures on WN

Selecting synset. We utilized the NLP library that incorporates the English WN to pick the synset in the WN. It is noteworthy that in the English WN, finding the synset of a concept was more straightforward since it has a greater number of synsets than the AWN, and it provides a more precise definition of synsets because the term is accessible in both Arabic and English, thereby mitigating the issue of homonyms (Table 10).

Result. We compared five different similarity measures from various categories on the English WN, including Path, Wu & Palmer's, Leacock & Chodorow, Resnik, and Lin measures. Table 11 presents a comparison of the most widely used similarity measures in WN, with WuP, Path, and Leacock & Chodorow belonging to the path category and Resnik and Lin to the information content - corpus dependent category. This category was not applicable in AWN due to the unavailability of the required corpus in Arabic. Upon initial observation, we noticed that the English WN includes all the necessary concepts for this experiment, and each word has its synset, unlike AWN, where the issue of synset deficiency arose.

Table 10. The selected synset for the third comparison (synsets in English WN)

Table 11. Result of the third comparison (applying similarity measures on English WN)

The first row is represented by the following abbreviations: N: the numbering of the word, HN: Human rating, W: WuP measure, Li: P: Path measure, LC: Leakcock & Chodorow, R: Resnik measure and L: Lin.

image008.png

Figure 6. The correlation between human ratings and similarity measures scores in in AWN experiment using the correct synsets

According to Table 12, the WuP measure in English WN yielded the best similarity scores among all measures in experiments 2 and 3, as indicated by its highest correlation value (Figure 6) and lowest mean squared error.

Table 12. The correlation and MSE of the measures in the third comparison

3.5 Experimental observations and the introduction of a novel hybrid Similarity measure combining WuP and Resnik measures

3.5.1 A comparison of AWN and WN

This section explores the contrasts between AWN and WN identified in previous experiments, along with the benefits and drawbacks of utilizing each of them.

Advantages of leveraging AWN in the domain of natural language processing for Arabic include:

• Every word in AWN is marked with Arabic vowels, making it better equipped for avoiding homonyms when seeking a synset for an Arabic word.

1. It performs exceptionally well when processing Arabic books that contain Arabic vowels (diacritics).
2. Outperforms other ontologies in tasks related to natural language processing, such as determining word roots.

Regarding the constraints associated with the utilization of AWN in the context of natural language processing for Arabic, these encompass:

• AWN's limited synset coverage requires WN to locate missing synsets, which are not available in AWN.
• Decreased efficiency in tasks involving words without Arabic vowels.
• AWN has not received an update since 2010.
• When used with modern books and online publications lacking Arabic vowels, it has poor efficiency.
• The lack of resources for the Arabic language limits the use of many similarity measures.

While the merits of employing WN in the realm of natural language processing for Arabic encompass:

• WN contains a larger number of synsets than other language specific WNs.
• WN has been updated more frequently than AWN.
• Provides better accuracy in calculating the degree of similarity between concepts.
• Allows for the use of all similarity measures.
• Performs better with modern documents because they lack Arabic vowels, only requiring word translation.
• English WN has a wealth of resources in terms of corpus and tools, making it easier to implement than other WNs.

Drawbacks associated with the utilization of WN in the context of natural language processing for Arabic comprise:

• Requires correct translation of words to obtain the desired result.
• Cannot be used in many Arabic language processing tasks.

3.5.2 Comparing results of similarity measurements and introduction of a hybrid measure

We analyzed the results of similarity calculation using various measures and compared the measures that yielded the best outcomes in experiments where we used the correct synset.

As previously mentioned, the WuP measure exhibited a high correlation coefficient (0.82) with human ratings, indicating a good linear relationship with human ratings. Figure 7 illustrates the squared error between human ratings and scores obtained from three different measures.

Through this experiment, we note that measure WuP has a good ability to measure the similarity between words, where we note that the results of measure WuP are excellent for words that have a large percentage of similarity, while we note relatively irregular values for words that have little similarity, unlike measure Resnik , which was able to get very close to the exact result of these words from the conclusion of the observation that we made in many experiments, it can be concluded that measure WuP can be used as a good separator between words that are close in meaning and words that are far apart in meaning.

From the Figure 7, it is evident that WuP outperformed the other measures on pairs with similarity scores above 0.55. On the other hand, for pairs with similarity scores below 0.55, the squared error was high, whereas the Resnik measure exhibited better performance on pairs with similarity scores below 0.45. Finally, the LCH measure performed well on pairs with moderate similarity scores (between 0.45 and 0.55).

According to the results of the similarity measures, it was found that the WuP measure is more effective in determining the similarity of word pairs with a similarity score greater than a certain threshold, while providing a slightly higher score for pairs with a similarity value between them below this threshold. Conversely, the Resnik measure is more suitable for calculating the similarity score between words with similarity values below this threshold (1).

image009.png

Figure 7. Comparison of similarity measures squared error values in experiment 3 (the result of applying measure on WN)

Our earlier experiments indicate that employing a sole measure might not adequately capture the accurate degree of similarity between two words. We advocate for a more intricate method in gauging the similarity between word pairs. Rather than depending on a single metric, we propose categorizing word pairs into 'semantically close' and 'not semantically close'. Each category requires a distinct measure, as our results demonstrate that a universal measure lacks efficacy. This approach is expected to produce.

Based on these observations, a new similarity measure is proposed that incorporates the WuP and Resnik measures. The proposed measure takes into account several factors, including the depth of concept, the length of the shortest path between the synsets of two words, and the information content of their Least Common Subsumer (LCS).

$\begin{aligned} & \operatorname{sim}_{\text {Resnik } \& \text { WuP }}\left(c_1, c_2\right)=\left\{\begin{array}{l}\operatorname{sim}_{\text {resnik }}\left(c_1, c_2\right), \operatorname{sim}_{\text {wup }}\left(c_1, c_2\right)<\text { threshold } \\ \operatorname{sim}_{\text {resnik }}\left(c_1, c_2\right), \operatorname{sim}_{\text {wup }}\left(c_1, c_2\right) \geq \text { threshold }_{\text Sim_{{Resnik } \& \text { WuP }}}\left(c_1, c_2\right)\end{array}\right. \\ & =\left\{\begin{array}{c}\operatorname{IC}\left(\operatorname{lso}\left(c_1, c_2\right)\right), \frac{2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)}{\operatorname{len}\left(c_1, c_2\right)+2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)}<\text { threshold } \\ \frac{2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)}{\operatorname{len}\left(c_1, c_2\right)+2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)}, \frac{2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)}{\operatorname{len}\left(c_1, c_2\right)+2 \times \operatorname{depth}\left(\operatorname{lso}\left(c_1, c_2\right)\right)} \geq \text { threshold }\end{array} \text { and threshold }=0.45\right.\end{aligned}$                (1)