Emoji-Integrated Polyseme Probabilistic Analysis Model: Sentiment Analysis of Short Review Texts on Library Service Quality

2.1 PLTS in service quality reviews

There may be various sentiment polarities in service quality reviews. But it is not very accurate to simply divide the reviews into positive or negative class [35], for a single review may simultaneously contain multiple sentiments [36]. For example, an online review of library service quality may go as: “Although the books are boring, we are impressed by the service quality of the service staff; Besides, the air-conditioning is poor.” If quantified by the degree of polarity, the review is 50% neutral (the books are boring), 30% positive (service quality is good), and 20% negative (air-conditioning is poor). Then, the subjective review of the library can be expressed as feel(library) = {terrible(0.2), ordinary(0.5), favorable(0.3)}. To quantify the information and proportions, it is important to consider the subjective feelings of users, and other knowledge-based information, such as probabilistic distribution [37, 38], importance [39], and confidence [40]. Ignoring the information may lead to inaccurate analysis and erroneous subsequent decisions.

An effective way to solve uncertainty evaluation is to apply PLTS to decision-making. The sentiment information of each word can be transformed into a PLTS: $\left\{S_{-\tau}(p), S_{-\tau+1}(p), \cdots, S_{-\tau}(p)\right\}$, where τ is the number of levels, and $S_{i}$ is the level of sentiment polarity. If i>0, the greater the i, the higher the ranking of positive sentiment polarity; if i<0, the smaller the i, the lower the ranking of negative sentiment polarity. $p$ stands for the probability of $S_{i}$. Then, negative, neutral, and positive meanings can be expressed as subsets $\left\{S_{-\tau}(p), S_{-\tau+1}(p), \cdots\right\}$ $\left\{\cdots, S_{-1}(p), S_{0}(p), S_{1}(p), \cdots\right\}, \quad$ and $\left\{\cdots, S_{\tau-1}(p), S_{\tau}(p)\right\}$ respectively. Take $\tau=4$ for example. The sentiment information of funny can be described by a PLTS: $L(p)=\left\{S_{-3}(0.1), S_{-1}(0.2), S_{3}(0.7)\right\}$. That is, the probability for the word to convey level 3 negative meaning is 10%, that for the word to convey level 1 negative meaning is 20%, and that for the word to convey positive meaning is 70%. Obviously, a large $\tau$ value allows thorough description of sentiment information. In this way, PLTS illustrates the sentiment information of words to the maximum possible degree.

2.2 Flow of FECLSQ

To eliminate the effects of uncertain factors (polysemes) on short text sentiment analysis, the FECLSQ framework was divided into three processes: preprocessing, Word to PLTS (Emoji to words), and classification. The flow of this framework is shown in Figure 4.

2.3 Preprocessing

To improve data quality and improve analysis, preprocessing was carried out, including spelling correction, negative word check, and stop word removal.

2.3.1 Negative word check

The sentiment polarity of a text may be changed or reversed by negative words. If negative words like “no” and “not” appear in a text (e.g., not happy, and no success), the sentiment polarity of the positive words (e.g., happy, and success) in the text will be reversed. The reversal time depend on the number of negative words in the text. The negative coefficient $\lambda$ can be expressed as:

4.png

Figure 4. Flow of FECLSQ

Table 1. POS label conversion

Note: Stanford POS Tagger http://nlp.stanford.edu/software/tagger.shtmachinelearning

Language Technology Platform (LTP) Tagger http://www.ltp-cloud.com/

$\lambda=(-1) n$                   (1)

where, n is the number of negative words. If λ=-1, the sentence is negative, and the sentiment polarity is reversed; if λ=1, the sentiment polarity remains unchanged. Note that if the negative words in the short text appear with a certain interval, the change of sentiment polarity may cease to be effective, and such a short text should be filtered out.

2.3.2 Part-of-speech (POS) labeling

POS labeling aims to mark the words as nouns, verbs, adjectives, adverbs, etc. This paper utilizes the Stanford POS Tagger to process English short texts, and the Language Technology Platform (LTP) Tagger of Harbin Institute of Technology to process Chinese short texts. The taggers adopt Penn Treebank POS labels, and convert them into the labels of nouns, verbs, adjectives, and adverbs (Table 1). This paper only tags nouns, verbs, adjectives, and adverbs, without considering prepositions and numbers.

2.3.3 Stop word removal

In a language, the stop words refer to the words that do not have special meanings, such as “it”, “is”, and “the”. These words are useless in text sentiment analysis. To reduce their effects on computing and storage space, these words are filtered out against the stop word list.

2.4 Transforming word/emoji to PLTS

Each word/emoji can be transformed into a PLTS in three steps: conversion, update, and calculation.

2.4.1 Conversion

Every emoji is a binary vector $\left\langle i m g_{e m o j i}\right.$, emoji label $\rangle$ (Figure 5). If the emoji in the original sample is an image $i m g_{e m o j i}$, it should be converted into lemoji label; otherwise, the emoji label can be directly used. Through this mapping, all emojis can be converted into words, and then integrated to the subsequent PLTS calculation process.

5.png

Figure 5. Emoji vector

Then, the difference SynsetScore between positive review score PosScore and negative review scoreNegScore is calculated, and used to evaluate the sentiment intensity difference between the words in the SWN dataset. The SynsetScore can be expressed as:

SynsetScore $=$ PosScore $-$ NegScore, SynsetScore $\in[-1,1]$                (2)

Next, a one-dimensional array ArrSS is defined to describe the scores of the synonym set for the word:

$\operatorname{ArrSS}\left(\right.$ word $\left.\right)=\left[\right.$ synset $_{1}$, synset $_{2}, \cdots$, synset $\left._{n}\right]$             (3)

where, n is the size of the synonym set for the word.

To ensure the correspondence to linguistic terms $L^{(k)}$, SynsetScore is mapped from [-1, 1] to [-8, -7, ⋯, 7, 8]:

$f_{\text {trans }}($ SynsetScore $)=([8 *$ SynsetScore $])=\alpha$                 (4)

In this way, SynsetScore is converted into $\alpha(k)$, $\alpha \in$[-8, -7,⋯, 7, 8]. Thus, ArrSS can be converted to a 1D array of the linguistic term ArrLT:

$\operatorname{ArrL} T_{(\text {word })}=\left[S_{\alpha_{1}}, S_{\alpha}, \cdots, S_{\alpha_{n}}\right]$              (5)

After that, it is possible to calculate the frequency of occurrence S of each linguistic term, and estimate the probability p of each term. The sentiment polarity of a word can be converted into a PLTS:

$L_{(\text {word })}(p)=\left\{S_{\alpha}(p) \mid S_{\alpha} \in S, p \geq 0, \sum p=1\right\}$           (6)

2.4.2 Updated SWN

The preceding subsection extracts sentiment information from SWM, and converts it into PLTS. But the extracted PosScore and NegScore are not fully accurate. Deviation may arise for several reasons:

(1) The scores calculated by the semi-supervised method may be biased. For example, Table 2 lists the sentiment polarity scores of the word “ridiculous” $\left(L_{\text {ridiculous }}(p)=\left\{S_{0}(0.333), S_{3}(0.333), S_{5}(0.333)\right\}\right)$. The data show that “ridiculous” conveys a positive meaning, but this word is actually a negative word.

Table 2. Sentiment polarity scores of the word “ridiculous”

(2) The same word appears in different texts at different probabilities. For instance, “good” in SWM has up to ten meanings. Some meanings are common, and some are rare. The imbalance and sparsity of words and word polarities hinder the polarity judgement. More importantly, SWN does not provide any information about word probability.

(3) The sentiment polarity of words is affected by context. In different contexts, sentiment polarity will transfer or strengthen. The same word may have opposite polarities in different areas.

The above factors may lead to under-fitting. To solve the problem, this paper updates the SWN dictionary by the statistics on the word frequency distribution. On this basis, the context sentiment weight (CSW), which falls between -1 and 1, is introduced to the context:

$C S W_{\text {word }}=\frac{\text { Freq }(\text { word } \mid p)-\text { Freq }(\text { word } \mid n)}{\text { Freq }(\text { word })}$             (7)

where, Freq $($ word $\mid p)$ and Freq $($ word $\mid n)$ are the frequencies of a word appearing in texts labeled as positive and negative, respectively. Then, the total word frequency can be expressed as Freq $($ word $)=\operatorname{Freq}($ word $\mid p)+$ Freq $($ word $\mid n)$. For example, $C S W_{\text {best }}=1$ indicates that the word "best" merely appears in texts marked as positive.

Similarly, CSW is mapped from $[-1,1]$ to $\alpha \in[-8,-7, \cdots, 7,8]$ using $f_{\text {trans }}$, and a word frequency distribution information dictionary is established. The dictionary is named context sentiment weight dictionary (CSWD). Table 3 gives an example of CSWD.

Finally, CSWD is employed to modify the PLTSs extracted from SWN:

$L_{\text {word }}(p)=\left\{\begin{aligned} S_{-8}\left(\frac{p}{1+m}\right), S_{-7}\left(\frac{p}{1+m}\right), \cdots ,\\ \times S_{\alpha}^{\text {word }}\left(\frac{p+m}{1+m}\right), \cdots, S_{8}\left(\frac{p}{1+m}\right) \end{aligned}\right\}$             (8)

where, $L_{\text {word }}(p)$ and $S_{\alpha}^{\text {word }}$ are the PLTS and linguistic term of the word, respectively $($ Freq $($ word $\mid p) \alpha$ is looked up in CSWD); $m \in 0,+\infty)$ is an empirical adjustment factor that manually regulates model accuracy. If m=0, no adjustment is necessary. The greater the m, the more significant the adjustment. But a large m may result in over-fitting. Here, the PLTS of SWN is modified at m=1.

2.4.3 PLTS calculation

Each sentence is a combination of a series of words, which contain (hide) sentiment information. By synthetizing the sentiments of all the words in the series, it is possible to obtain the reduced theme sentiment of the sentence.

Let n be the number of words segmented from a sentence. Then, these words will be converted into n PLTSs. The heavy presence of neutral words adds to the computing load. Thus, the neutral words can be filtered out by the following rule:

word $\rightarrow L(p)=\left\{S_{-8}(p), S_{-7}(p), \cdots, S_{8}(p)\right\}, p_{S_{-1}}+p_{S_{0}}+p_{S_{1}} \geq 0.8$           (9)

where, $P_{S_{\alpha}}$ is the probability of $S_{\alpha}$.

After the filtering of neutral words, there are m remaining PLTSs. The sentiment polarity of the sentence can be characterized by the mean $P L_{a v g}$ of probabilistic language:

Table 3. An example of CSWD

Table 4. Input format of SVM

$\begin{aligned} L_{\text {sentence }}(p) &=\lambda * P L_{\text {avg }}(L(p))=\lambda * \sum_{i=1}^{m} L_{i}(p)=\left\{S_{\alpha}(p)\right\} \end{aligned}$           (10)

where, $L_{\text {sentence }}(p)$ is the PLTS of the sentence. If the sentence is negative, $\lambda=-1$; otherwise, $\lambda=1$.

Normally, $L_{\text {sentence }}(p)$ contains lots of $S_{\alpha}(p)$, and the adjacent $S_{\alpha}$ have a very small difference. To further reduce computing load, $S_{\alpha}(\alpha \in[-8,8])$ is mapped to $S_{\alpha}(\alpha \in$ $[0,1, \cdots, 100])$, and the $S_{\alpha}(p)$ with the same suffix $a$ is merged.

During the transformation of words into PLTSs, each sentence of a text is expressed as one $L_{\text {sentence }}(p)$, which represents the possibility of each sentiment polarity. This approach fully utilizes the sentiment information and word frequency information in the sentiment dictionary.

2.5 SVM-based sentiment classification

According to the PLTS calculation in $2.4$, $L_{\text {sentence }}(p)$ contains lots of $S_{\alpha}(p)$. Even if repetitive terms are eliminated by mapping, the resulting $S_{\alpha}(p)$ will affect the sentiment polarity of the sentence. To improve classification performance, this paper constructs a classifier based on SVM, a supervised learning strategy using radial basis function (RBF) kernel. Each linguistic term is expressed as a feature, and the probability P corresponding to that term is regarded as the eigenvalue. Table 4 shows the input format of SVM.

Then, $L_{\text {sentence }}(p)$ is converted into a format that can be understood by SVM classifier:

$\langle$ Label $\rangle\left\{\begin{array}{l}\text { feature }[<\text { index }>,<\text { value }>], \\ \text { feature } e_{2}[<\text { index }>,<\text { value }>], \\ \cdots, \\ \text { feature }_{n}[\cdots]\end{array}\right\}$             (11)

where, ⟨Label⟩ corresponds to positive class and negative class; feature corresponds to n linguistic terms; value is the probability p of each term.

4.1 Datasets

The performance of PLTS-FECLSQ was tested on three public datasets and two self-built datasets:

(1) Comment Dataset Library Service Quality (CDLSQ)

This dataset of library service quality reviews was established by the authors. There are 8,176 reviews in the datasets. Among them, 4,782 reviews contain emojis.

(2) Movie reviews (MR) [41]

This dataset contains 10,662 movie reviews from Rotten Tomatoes. Half of them is labeled positive, and half are labeled negative. In the original reviews, positive and negative reviews are described by “fresh” and “rotten”, respectively.

(3) Stanford Twitter Sentiment (STS) [42]

The STS contains 1.6 million tweets, which are automatically labeled as positive or negative. This paper randomly selects 20,000 tweets from the dataset.

(4) TripAdvisor reviews (TR) [43]

This paper crawls over 15,000 text reviews from tripadvisor.com, a tourism networking and community network, and obtains 1,100 positive reviews and 950 negative reviews through automatic filtering and artificial evaluation.

(5) DeepMoji [44]

This dataset contains more than 120 million filtered tweets. Referring to the polyseme list of SWN, this paper filters the dataset, and chooses the samples containing only one emoji. After filtering, a self-built dataset DMx was established. Then, DMx was divided randomly into a training set and a test set at the split ratio of 4:1. Table 5 provides the details of the datasets.

4.2 Metrics and results

The classification models were evaluated comprehensively with four metrics: accuracy (Acc), precision (Prec), recall (Rec), and F-measure (F-M) [45]:

Accuracy $=\frac{T P+T N}{T P+F P+T N+F N}$               (12)

Precision $=\frac{T P}{T P+F P}$               (13)

Recall $=\frac{T P}{T P+F N}$               (14)

$F-$ Messure $=2 * \frac{\text { Precision } * \text { Recall }}{\text { Recall }+\text { Precision }}$               (15)

where, TP, TN, FP, and FNare true positives, true negatives, false positives, and false negatives, respectively; F-measure is the harmonic mean between precision and recall, and an index of comprehensive evaluation.

Firstly, the performance of five mature sentiment dictionaries, namely, GI, TB, VADER, SCN, and SWN, was compared on the datasets. Table 6 shows the test results of these dictionaries, and Figure 8 displays the mean accuracy of each dictionary.

Among all dictionaries, VADER achieved the best performance, followed by TB. SCN differed very slightly from SWM, while GI realized the highest stability. For the five sentiment analysis tools/techniques, the mean accuracy, recall, and F-measure on MR were 58.50%, 69.39%, and 63.36%, respectively; those on STS were 62.36%, 57.92%, and 59.77%, respectively; those on TR were 63.99%, 78.38%, and 70.41%, respectively. The recall was higher than the precision of dictionaries, suggesting that dictionary-based methods are more sensitive to FP than to FN. On the three datasets, the F-M averaged at only 62.5%, and peaked at 77.67%. Thus, dictionary-based methods are far from satisfactory. In addition, the five dictionaries performed poorly on CDLSQ, which is related to their focus on Chinese language.

Furthermore, PLTS-CLSQF was compared with three machine learning methods: NB, SVM, andLR. The classifiers were trained by the BOW [46]. As shown in Table 6 and Table 7, the highest values on the five datasets were 78%, 76%, 79%, 74%, and 88%, respectively. Therefore, machine learning-based methods far outshine dictionary-based methods.

PLTS-FECLSQ, which combines supervised learning with unsupervised learning, improves the best performance on CDLSQ and DMx. The results indicate that the accuracy and F-measure increased with the scope of the training set. PLTS-FECLSQ can effectively solve various problems, using dictionary information, and word frequency distribution. Firstly, the sentiment information of each word is extracted from the current dictionary and word frequency distribution. Each word/emoji is expressed as a PLTS. Next, the PLTSs of all words are synthetized into the PLTS of the sentence L_sentence(p). Finally, SVM is called to enhance the classification performance. Compared with unsupervised classifiers, PLTS-CLSQF introduces word frequency distribution and SVM to overcome the problems of neighborhood dependence and contextual information. Compared with supervised classifiers, PLTS-CLSQF can extract the sentiment information of words from the current dictionaries, which is difficult to acquire through data training alone. Our approach can effectively solve the unavailability and sparsity of data. More importantly, the PLTS and related theories are adopted to treat the concurrence of polysemy and emojis.

8.png

Figure 8. Mean accuracies of the five dictionaries on the datasets

Table 5. Dataset overview

Table 6. Test results of five dictionaries on the datasets

Table 7. Comparison between machine learning and PLTS-FECLSQ