Impact of Using Bidirectional Encoder Representations from Transformers (BERT) Models for Arabic Dialogue Acts Identification

In Human-Machine Interaction dialogue systems, the Dialogue Acts recognition is considered an important component. The research in this area has made great progress during the last few years. Kumar et al. [6] was based on a hierarchical recurrent neural network that learns representations at word, utterance, and conversation levels. The conversation level representations consider all previous utterances and their dialogue acts. They validated their approach on two datasets: Switchboard (SwDA) and Meeting Recorder Dialogue Act (MRDA). The performance outperformed the state-of-the-art methods by 2.2% and 4.1% absolute points, respectively. a simple RNN has been used to model the previous utterances context. They evaluated their model on the Switchboard dataset and achieved an accuracy of 77.34% [7]. Lee and Dernoncourt [8] have also presented a model based on RNN and CNN networks that incorporates the preceding short texts. Their model was validated on three different datasets (DSTC 4, MRDA, and SwDA), and achieved state-of-the-art results. Chen et al. [9] proposed a 2-steps approach based on the CRF-Attentive Structured Network. The approach outperformed several state-of-the-art solutions on SwDA and MRDA datasets. An improved dynamic memory network with hierarchical pyramidal utterance encoder has been proposed by Wan et al. [10]. Experiments showed that the model was robust and achieved better performance compared with some state-of-the-art baselines. Dai et al. [11] proposed a hierarchical model to capture intra-sentence and inter-sentence information based on self-attention. Their model was tested on two datasets: SwDA and DailyDialog, and achieved promising performance with an accuracy of 80.34% and 85.81% respectively.

With the introduction of Bidirectional Encoder Representations from Transformers (BERT), lots of NLP tasks started outperforming other models. Saha et al. [12] presented a novel model for the identification of speech acts in Twitter on top of BERT. They introduced a BERT-extended classifier for the task, with a model based on calculating the attention weights over the token representations of a sequence obtained from the pre-trained BERT model. The proposed model was evaluated on an open-access data set released by Saha et al. [13] and outperformed the state-of-the-art approach, with an accuracy of 75.97%. Then Saha et al. introduced [14] the BERT-Caps model, which is built on top of BERT. The new model takes into consideration both feature optimizations from BERT and capsule layer to better learn the traits and attributes. Their results outperformed several strong baselines and state-of-the-art approaches with an accuracy of 77.52%. Saha et al. [15] studied the role of sentiment and emotion in speech act classification in Twitter. They proposed a Dyadic Attention Mechanism based multi-modal (emojis and text), adversarial multi-task framework for joint optimization of Tweet Acts, sentiment and emotions. Their experiments on EmoTA tweets acts dataset, showed that the proposed framework boosts the classification performance by benefitting from the Sentiment and Emotion Analysis.

Concerning non-English languages, different researches focused on multilingual domain. Cerisara et al. [16] explored in recurrent models to capture the sequence of words within sentences, and studied the impact of using pre-trained word embedding representations on the recognition task. The model was tested on English, French and Czech, with a consistent performance across the three languages, and a comparable performance to the state-of-the-art results in English. For Persian language, a dictionary-based statistical technique was proposed for speech acts recognition [17]. Authors used lexical, syntactic, semantic, and surface features to detect seven classes of speech acts. They evaluated their proposed technique using four classification methods: Random Forest RF, Support Vector Machine SVM, Naive Bayes NB, and K-Nearest Neighbors KNN. The best classification accuracy was obtained using RF and SVM.

Recently, Arabic speech acts classification task started to show preliminary initiatives. Early works were dedicated to Modern Standard Arabic (MSA). Shala et al. [18] applied different machine learning classifiers (SVM, NB and Decision Trees) to classify Arabic discourse speech acts using a dataset of about 400 MSA utterances collected from newspapers. Sherkawi et al. [19] presented a bootstrapped rule-based model to detect MSA Arabic Speech Act types. The studied speech acts types were: Affirmation, Hope, Condition, Praise, Dispraise, Negation, Confirmation, Interrogation, Imperative, Forbidding, Wishing, Vocative, Prompting, Rebuke, Exclamation, Swear. The model was tested on a corpus of about 1500 MSA sentences. Sherkawi at al. [20] proposed a machine learning approach based on surface features, cue words and contextual information to recognize MSA Arabic speech acts. Authors compared the results of Decision Trees, Naïve Bayes, Neural Networks and SVM algorithms on their corpus of 1500 MSA sentences. The Decision Tree algorithm had the best results. Other studies focused on the dialectal language. For Tunisian Dialect Graja et al. [21] used the TuDiCoI corpus (12182 utterances) to develop a discriminative algorithm based on CRF to semantically label spoken turns which are not segmented into utterances. For Egyptian Dialect, Elmadany et al. [22] proposed a machine learning approach based on multi-classes hierarchical structure to classify dialogue acts using the JANA corpus (4725 utterances). The model attained an average F-measure scores of 91.2%. Algotiml et al. [23] proposed a speech act classification model for Twitter asynchronous conversations, where they studied several machine learning methods including SVM and deep neural networks. They applied the proposed methods on the ArSAS tweets dataset [24]. The results showed that deep learning methods had better performance compared to SVMs, where Bi-LSTM achieved an accuracy of 87.5% and a macro-averaged F1 score 61.5%. Joukhadar et al. [25] compared various machine learning algorithms with different features being used to detect the correct speech act categories. They compared the results of the proposed models on a hand-crafted corpus of 873 sentences in the restaurant's orders and airline ticketing domain in Levantine Arabic Dialect. The best result was given by SVM algorithm with 2-gram word features.

Our proposed system is based on AraBERT, a pre-trained BERT transformer model (a stacked Bidirectional Transformer Encoder( for the Arabic language [26]. The basic idea of BERT is to pre-train deep bidirectional representations from unlabeled text including context from both directions then fine-tune all the output parameters on the studied task.

Following the original BERT pre-training objective, AraBERT employs the Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP) tasks.

In this work, we explore the different pre-trained versions of the Arabic-specific BERT provided by Antoun et al. [26], and fine-tune them for Arabic Dialogue Acts Recognition task. The six models are: AraBERT Original (v0.1, v1), AraBERT Base (v0.2, and v2) and AraBERT Large (v0.2, and v2).

The AraBERT Original models were trained on about 23GB of Arabic text, collected from Arabic Wikipedia and news articles from different media in different Arab regions, and therefore can be representative of a wide range of topics discussed in the Arab world. This corpus contains about 77M sentences with 2.7B words. Both AraBERT Base and Large models were trained on a larger corpus of about 77GB of Arabic text, which approximately corresponds to 200M sentences with 8.6B words.

Due to the complex concatenative system of Arabic language, it has a lexical sparsity issue, where words can have different forms and share the same meaning. For example, the definite article "الـ", is always concatenated as a prefix to the next word, and never appear independently. Thus, when using an ordinary tokenizer, tokens will appear twice (with and without “الـ”) which implies a huge unnecessary redundancy. To avoid this issue, both v1 and v2 of AraBERT uses pre-segmented text where words are segmented into stems, prefixes and suffixes using the Farasa Segmenter (https://farasa.qcri.org/segmentation/). For instance, “اللغة - Alloga” becomes " ال + لغ + ة - Al+ log +a”. Farasa Segmenter is an SVM^rank-based segmenter that uses a variety of features and lexicons to rank different possible segmentations of a word. The features used account for: likelihood of stems, prefixes, suffixes, and their combination; presence in lexicons containing valid stems and named entities; and underlying stem templates. Table 4 shows the different parameters for each model version.

Table 4. The different parameters for each BERT model version

The first step of the process consists in tokenizing and representing the input sentence using the studied AraBERT model. Diacritics and extensions were removed, but English characters have been retained, as it is common to mention named entities, scientific or technical terms in its original language. Wordpeice was used for tokenization step. If the studied AraBERT model uses morphologically segmented words, we used Farasa for pre-segmentation step. Then, a fully connected neural network is applied on the output of the AraBERT model to classify the sentences according to their Dialogue acts. Figure 1 presents the architecture of the model.

1.png

Figure 1. The architecture of the model

Different experiments were performed to evaluate the different Arabert model. The experiments on the ArSAS dataset and the corresponding results are presented in the first subsection, followed by the experiments on the LevInt dataset.

The performance was evaluated through various measures including: Accuracy, Macro and Weighted average of Precision, Recall and F1-measure. The classification accuracy is the total number of correct predictions divided by the total number of predictions made for a dataset. Let TP the true positive samples, FP the False positive samples, and FN the false negative samples. The precision, recall and F1 measure are given by the following formula:

Precision $P=T P /(T P+F P)$

Recall $R=T P /(T P+F N)$

F1 Measure $=2 * P R /(P+R)$

A Macro-average compute the metric independently for each class and then take the average, hence treating all classes equally, whereas a Weighted average calculates metrics for each class, and finds their average, weighted by support (the number of true instances for each class).

6.1 Model effectiveness on ArSAS dataset

As the ArSAS dataset was highly imbalanced, a resampling step was conducted before the classification. The dataset was divided into 80 % for training data and 20% for test data. Table 6 shows the results achieved by the different models on the dataset.

We remark that all models have achieved good classification results. The AraBERTv0.2-base and AraBERTv0.2-large models outperform the rest with an accuracy of 89%. The best classification performance of all the models is obtained using the AraBERTv0.2-base pre-trained model, based on Precision, Recall and F1-scores calculated using Macro and Weighted average.

To understand the results, we looked into the detailed classification report for each class. Table 7 shows the detailed classification report using the weighted average.

Table 6. Comparison between different BERT-Based models on ArSAS, in terms of Accuracy, Macro and Weighted average of Precision, Recall, and F-measure

Table 7. The detailed classification report on ArSAS for each class (arabert-v02 model)

The Expression Dialogue act has the best results with a precision, recall and F1 measure of 89%, 93%, and 91% respectively. It is followed by the Assertion with a precision, recall and F1 measure of 92%, 88%, and 90% respectively.

Although the overall performance is relatively good with an accuracy of 89% and a weighted F1-Measure of 89%, some classes have very poor performances. It could be explained by the high imbalance of the dataset classes. The Miscellaneous Dialogue act, for example, which is the smallest Dialogue act set, has a precision, recall and F1 measure of 20%, 17%, and 18% respectively, followed by the Recommendation Dialogue act, which has comparable results.

On the other hand, the Question and Request Dialogue acts have better, yet not good, results with an F1 measure of 64%, and 44% respectively (as they are still extremely small compared to Expression and Assertion sets).

Moreover, we further analysed the confusion matrix to determine the correct/incorrect labels corresponding to the sentences. Table 8 shows the confusion matrix of our best model (arabert-v02).

Table 8. The confusion matrix of our best model (Arabert-v02)

We notice that most errors were made in sentences that belong to the class “Assertion” that were predicted as “Expression” (with 160 cases out of 1560 cases), and the class “Expression” which were predicted as “Assertion” (with 110 cases out of 2212 cases).

Intents that were misclassified to a high extent include: Miscellaneous, Question, Recommendation, and Request. They also tend to be classified as Expressions, with 7 out of 12 for Miscellaneous, 53 out of 141 for Question, 10 out of 20 for Recommendation, and 16 out of 34 for Request. We noticed that these four intents have a very low number of examples in total, which is the reason for the models having problems in classifying them correctly. Another plausible reason behind the faults in the Dialogue Acts prediction - beside the skewed dataset- is that sentences in the dataset are composite in nature, and thus encompassing diversified intentions in a single sentence. For example, in the sentence “التطرف والفساد وجهان لعملة واحدة، يتحركان في نفس الاتجاه وسلاح محاربتهما واحد إرساء دولة القانون“, we can identify two Dialogue acts at the same time, Expression at the beginning and Recommendation in the final phrase. In Table 9, we present some examples, from ArSAS, where the model fails to predict the correct Dialogue acts.

Table 9. Examples of Labelling Errors on ArSAS

We also compare our best model, AraBERTv0.2-Base, to the state-of-the-art on the same dataset performed by Algotiml et al. [23], who compared deep learning methods against machine learning methods. In their work, since the smallest two classes in the dataset “miscellaneous” and “recommendation” have only 60 and 109 tweets respectively, authors decided to merge these two classes into one and called it miscellaneous. Therefore, in order to compare with their results, we merged the two classes, and fine-tuned our model again against the new dataset.

Table 10 shows a comparison between the best results of Algotiml et al. [23] in both machine learning and deep learning methods and our results using AraBERTv0.2-Base with/without doing resampling.

Table 10. Performance comparison with the state of the art on ArSAS

Our accuracy results outperform Algotiml et al. [23] results by 1.5%, and we notice that using resampling process, substantially raised the value of Macro-F1.

6.2 Model effectiveness on LevInt dataset

We applied the previous Arabert models on the Levantine Arabic dataset LevNet, which is balanced, and contains 873 sentences tagged with 8 speech acts. Table 11 presents the performance of the different models.

Table 11. Comparison between different BERT-Based models for LevInt speech act classification

We remark that the best classification performance of all the models is obtained using the AraBERTv0.2-base pre-trained model, based on all metrics. This result confirms our previous conclusions produced based on the ArSAS dataset.

We also compare the result of our best model, AraBERTv0.2-Base, to the state-of-the-art model performed by Joukhadar et al. [25] on the same dataset. Authors in [25] compared different machine learning methods, and the only metric used for evaluating the different multi-labelling classifiers was the accuracy. Their results show that the SVM classifier outperforms the rest of the classifiers with an accuracy of 86%. When compared to our best accuracy (93.7%) stated in Table 11, we can conclude that our model outperforms their model by 7.7%.