Hate Speech in the Arab Electronic Press and Social Networks

2.1 The proposed approach

2.1.1 The BERT method

In our study, we use BERT, which stands for Bidirectional Encoder Representations from Transformers [27]. Compared to the word embedding model, BERT can also be described as a text representation method, which is a combination of various advanced deep learning algorithms, such as Long Short Term Memory, and also transformers.

Work on learning linguistic representations by pre-trained models on large unlabeled data sets of text documents began from word embeddings such as Word2Vec. This technique changed the way NLP tasks were performed. Thanks to the pre-trained models, we could obtain incorporations capable of creating contextual links between words.

The main limitation of these techniques was the use of superficial language models.

Thus, the new approach to solving NLP tasks has become a two-step process: The first performed by training a language model on a large body of text data set (unlabeled, and unsupervised or semi-supervised), and in our case, we use ArabicBERT [26]. The second is to refine the large model obtained to a specific NLP task, so that we can use the important repository of knowledge acquired by the (supervised) model, for our task we refine ArabicBERT using a hate speech labeled dataset to create a text classification model.

The BERT architecture is based on Transformer. We currently have two variants of this model:

(1) BERT Base: composed of 12 layers (transformer blocks), 12 attention heads, and 110 million parameters.

(2) BERT Large (used in our study): 24 layers (transformer blocks), 16 attention heads and, 340 million parameters.

As state of the art of natural language processing, we can cite the OpenAI GPT model [28], and Embeddings from Language Models (ELMo) [29]. OpenAI GPT is a large-scale, language model that produces convenient paragraphs in a given language, it achieves peak performance on many language modeling benchmarks, and performs reading comprehension, machine translation, question-answering and summary, all without the need of a fine tuning training, ELMo has been proven to be an important model in the context of natural language processing, it can be trained on a massive datasets, and then we can use it as a component in other models that require language support.

This model got its understanding of language by training it in masked language modeling to predict the next sentence.

1.png

Figure 1. Architecture showing its bidirectional character compared to other language processing models

In Figure 1 we see a visualization of the neural network of BERT architecture in comparison with other state-of-the-art contextual linguistic pre-training techniques. The arrows show the information flow from a layer to another. The green squares indicate the end of contextualized representation of every input.

Usually we have modules which are trained to predict the next word in a sentence or trained in left to right context, which makes them prone to many errors due to loss of significant information etc. This is where BERT improves a lot. Compared to other NLP methods because it is bidirectional, it also combines two techniques, masked language model and next sentence prediction. BERT was introduced by Google researchers in the year 2018 and has proven its power in being at the cutting edge of technology for various natural language processing applications.

2.1.2 BERT in application

In our work we approach a natural language processing task which can be described as a text classification, in which we use the Bidirectional Encoder Representations from Transformers transfer learning technique, where we train a deep learning model on a large labeled dataset of different Arabic dialects, the trained model is able to capture the sequential information present in the text, and therefore is used to perform the classification task on different sources.

The BERT model we work on as a basis for our fine-tuned Arabic hate speech detection model is the arabic-bert-large model, that was pre-trained on ~8.2 billion words. For Masked Language Modeling and Next Sentence Prediction tasks, in our approach we train a new model on a labeled dataset, to perform the text classification task.

The architecture we are using is a Transformer model, as it goes beyond traditional RNNs in its ability to take an entire input sequence at a time, which is a big improvement that allows the model to be accelerated by GPUs.

2.png

Figure 2. Transformer architecture [30]

As we can see in Figure 2, transformers are ENCODER-DECODER models with ATTENTION, there are a multitude of models based on transformers, the most popular are BERT and GPT, and previously we have explained the strengths of BERT on GPT regarding processing tasks natural language, which is why we use BERT for text classification.

The BERT encoder is made up of N=6 similar layers. Each one composed of two sublayers. The first is a multi-head self-attention network, and the second is a simple feed-forward mechanism, and a linking is used for each of the two Sub-layers, then normalization is applied. So, the output of every sublayer can be written as:

$\text { LayerNorm }(x+\text { Sublayer }(x))$        (1)

where, Sublayer(x) the function is implemented by the sublayer  itself.

The product of every sublayers in the model, and also of the embedding layers, is outputs of kind and measure:

$d_{\text {model }}=512$        (2)

The decoder is made of a stack of N=6 equal layers, the decoder inserts, after the two sub layers, of any encoder layer, a third sub-layer, which performs multi-head attention over the output of the encoder stack. The self-attention layer in the decoder is changed to stop current positions from affecting the next ones. This masking added to the fact that the embeddings in the output are different by one position, guaranties that the predictions for a given position i depend only on the already known outputs at positions less than i.

Attention is a form of simple arithmetic, it is a function which takes X as input constituted by query and a set of key-value pairs, and gives back another Y identical sequence in length, composed of vectors of the same length as those by X.

The output is computed as a weighted sum of the values, where the weight assigned to each value is calculated by a query compatibility function with the corresponding key.

In Bert attention is called "Scaled Dot-Product Attention". Formally we have a query Q, a key K and a value V and compute the matrix of outputs as shown in (1) [30]:

$\operatorname{Attention}(Q, K, V)=\operatorname{softmax}\left(\frac{Q K T}{\sqrt{d k}}\right)$        (3)

One of the strong characteristics of the bidirectional transformer is the multi-head attention mechanism; this unity works by the attention mechanism several times simultaneously. Then, the results of independent attention are serialized and linearly transformed into the expected dimension. Automatically, multiple attention heads allow you to attend parts of the sequence differently.

2.2 Materials

2.2.1 Programming language and platform

This article introduces a linguistic model for automatic detection of hate speech in Arabic, all the programming work, is done using the programming language Python, version 3.4.9, on a Windows 10 platform, note that the training of the model is performed using Google hardware acceleration, with the runtime type defined as Python 3, and the GPU as hardware accelerator with the following characteristics: NVIDIA-SMI 455.32.00, Driver Version: 418.67, CUDA Version: 10.1, the python versions operated on the cloud are: CPython 3.6.9 IPython 5.5.0.

2.2.2 Modules

Carrying out the different stages of training the model requires the use of various modules that are not included in the python library, the Table 2, summarizes the used modules, their versions, elements and parameters.

Table 2. Summary of modules and parameters

2.3 Process

3.png

Figure 3. Process flowchart

The graph in Figure 3 is a description of the different stages of the work process, which can be broken down into two main parts: the first being the preparation of the refined detection model from the collected data, and the second being the use of the model in order to detect hate speech in user text.

In the first phase, we collect and process the data to form a database used for model training which is the last part of this step. The second phase begins with processing the user's text, followed by the application of the model from the first phase, for the detection of hate language.

2.4 Dataset

In this study we use as shown in Figure 4, a dataset that is a combination of three different hate speech datasets: The first is L-HSAB Dataset [30], which is the first Levantine Arabic hate speech and abusive language dataset constructed from Levantine tweets retrieved via the Twitter API (Tweepy). The second dataset is a Multi-Platform Arabic News Comment Dataset for Offensive Language Detection [22], built using comments from different platforms. The third one is a dataset of YouTube comments collected in July 2017 [31]. The combination resulted in a data set of 38,654 entries made up of texts in Classical Arabic, Levantine, and North African Dialect.

The partial choice of the data to be incorporated into our database is made manually, after careful verification, of the entries and their label, the choice relates to the elements which present the unanimity of the judges, and which do not show any character of confusion, however the check of the completeness of the dataset elements is carried out automatically, using the Pandas module.

After the verification, we carry out a count of the different classes of our dataset with the help of Pandas module, from which we use the Dataframe.info function to print a concise summary of the DataFrame, including the index dtype and columns, non-null values and memory usage. The function shows the following results: 28898 entry of the "abusive" class and 9756 entry of the "neutral" class, to manage the problem of unbalanced data, we carry out an oversampling by replacement using the SMOTE module.

4.png

Figure 4. Dataset verification using pandas, the content is equivalent to scores, so there are no missing entries

2.4.1 Labeling

After combining the three datasets, the information was processed to eliminate non-Arabic and special characters, and the various comments were rated as neutral scored 0, or abusive scored 1, the result as appears in Figure 5 is an unbalanced labeled dataset.

5.png

Figure 5. Labeled dataset before balancing has 28898 hate speech texts and 9756 neutral

2.4.2 Balancing the dataset

Trying to learn from an imbalanced dataset always favors the bigger class, which can lead to misleading results after the training. This is a specific problem because we want to perform a correct classification between two classes, not multi-class. The BERT classification algorithm have a bias toward majority classes, since it works in a way to discover the rules with high values of accuracy and coverage, whereas specific rules that predict minority instances are ignored or treated as noise. Consequently, minority instances are misclassified, because the classifier is designed to minimize the error rate. If we do the classification training on an imbalanced dataset, while it will be performing the task, the model, will produce general rules and will have a bias toward majority instances, and it will ignore the minority ones, which means that in this case, the model will consider even texts that doesn't include hate speech as if they do.

Several methods can be used to deal with such problem and since we are working on a text (string) database, two best solutions are available:

Down sample: which means reducing the existing items in the majority class to match the other class, several techniques can be used to apply this method, such as, the random undersampling, in which entries will be removed randomly from the dataset, without any concern about their importance or significance. Another technique is the discerning undersampling, in this technique, the elements of the majority class are classified according to their importance and the ones that are removed correspond to the less important class.

Oversampling is done by applying a data augmentation. There are several techniques to do it, such as Increasing significant categories of samples, by applying bidirectional translations, or replacing expressions by their synonyms, which increases accuracy.

The second solution is the one we used, since the down sample method will cost us a considerable loss of training data which could have a negative impact on the results. The oversampling is applied by increasing the minority class to balance the dataset. The post-processing dataset, as we see in Figure 6, consists of a mixture of 57,796 text entries from classical Arabic and other different Arabic dialects, and we can see examples of it in Figure 7.

6.png

Figure 6. Oversampled dataset details

7.png

Figure 7. Example extracted from the final result of the dataset including classical Arabic, Levantine, and North African dialects

The combined Dataset can be downloaded using this Google drive file ID: id 1MCXY5eyI7myKyQQ2ZPpI1RHPZR7Emd2e, or this URL: https://drive.google.com/file/d/1MCXY5eyI7myKyQQ2ZPpI1RHPZR7Emd2e/view?usp=sharing.

2.4.3 Tokenization

In order to use the embedded BERT text as input to train our hate detection model, we need to tokenize the text of our comments. Tokenization refers to breaking down a sentence into individual words.

The Tokenizer applied by BERT code utilizes word-piece embedding processes [32] that use a shared learning framework from sequence to sequence. It consists of three components: layers of encoders, layers of decoders, and an attention layer. The task carried by the encoder is converting the original sentence into an index of vectors, one for each and every symbolized input. The work of the decoder is to generate a specific symbol for each input, until it reaches the special symbol for the last word in sentence. The link between the encoder and the decoder is the attention layer which enables the decoder to work on various parts of the input sentence while applying the decoder.

Let (X, Y) be a pair of source and target sentences. Let X=x₁, x₂, x₃, ..., x_M be the sequence of M symbols in the original expression and let Y=y₁, y₂, y₃, ..., y_N be the sequence of N symbols in the output expression. The encoder is in general a function of the form shown in (4):

$x_{1}, x_{2}, \ldots, x_{M}=\text { EncoderRNN }\left(x_{1}, x_{2}, x_{3}, \ldots, x_{M}\right)$        (4)

In this equation x₁, x₂, ..., x_M constitutes a list of vectors of determined size. The count of members in the list is similar to the number of symbols in the original expression (M in this example). Using the Conditional Chain Rule the probability of the sequence P(Y|X) can be detailed as in (5):

$\begin{gathered}

P(Y \mid X)=P\left(Y \mid x_{1}, x_{2}, x_{3}, \ldots, x_{M}\right) \\

=\prod_{i=1}^{N} P\left(y_{i} \mid y_{0}, y_{1}, y_{2}, \ldots, y_{i-1} ; x_{1}, x_{2}, x_{3}, \ldots, x_{M}\right)

\end{gathered}$        (5)

where, y₀ is a special "first part of the expression" symbol that is preceded by each output expression. During inference, the probability of the coming symbol is calculated given the encoding of the original phrase and the output phrase decoded so far as in (4):

$P\left(y_{i} \mid y_{0}, y_{1}, y_{2}, y_{3} \ldots, y_{i-1} ; x_{1}, x_{2}, x_{3}, \ldots,\right)$        (6)

To tokenize our Data, we use the AutoTokenizer module from the Transformers library, and we set “asafaya/bert-large-arabic” as Tokenizer [26], this step enables us to convert the text into Tokens and the Tokens into Ids that can be used in the training process, the Tokenization is done using the “encode_plus()” method, with truncation, and maximum length set to 512. After storing the tokens, we choose the maximum sequence length for the training process, in order to reduce the training time, because our dataset does not include very long texts. This choice is made after having displayed in the form of a bar graph the number of tokens stored for each entry, this technique shows us, that the maximum length is always less than 100 Tokens in this case. The information that we have is used to create a Pytorch Dataset, and we split our data into, test data frame 5%, validation data frame 5%, and training data frame 90%.

In Figure 8 we see an example of a tokenized sentence.

8.png

Figure 8. Sentence tokenized with ArabicBERT tokenizer

9.png

Figure 9. Representation of tokens contained in different comments

As BERT works with a fixed length of tokens, having for maximum 512, we have stored the tokens of each comment and we have represented them in graph, to be able to detect the maximum length on which we have set our training.

As we can see in Figure 9, most of the comments appeared to contain less than 80 tokens, but to be on the safe side, we chose a maximum length of 100.

2.4.4 Training

In this step, we worked with PyTorch deep learning library [33], which is an open source machine learning framework that accelerates the path from research prototyping to results deployment. We used the hyper parameters of the BERT paper and we used the AdamW optimizer provided by Hugging Face [34] which is a stochastic optimization method that modifies the typical implementation of the weight decay in Adam, decoupling the weight decay from the gradient update. L₂ Regularization in Adam is usually implemented with the modification below where w_t is the rate of weight decay at time t as it is in (7):

$g_{t}=\nabla f\left(\theta_{t}\right)+\omega_{t} \theta_{t}$        (7)

While AdamW adjusts the weight decay term to appear in the gradient update, as in (8):

$\theta_{t+1, i=} \theta_{t, i}-n\left(\frac{1}{\sqrt{\hat{v}_{t}+\epsilon}} \cdot \hat{m}_{t}+\omega_{t, i} \theta_{t, i}\right)$        (8)

In our case, AdamW corrects the weights decay, so it is similar to the original paper. We also used a linear planner with no warm-up steps.

For fine-tuning, we applied the recommendations of the authors of the BERT paper [27], namely: Batch size: 16, 32 Learning rate (Adam): 5th-5, 3rd-5, 2nd-5.

For the number of epochs, we proceeded by trial and error, having tried several values and our choice ended up being 10 epochs, since it gave the best results.

During the training process we create for every fraction of the data, a loader, with a batch size equal to 16, immediately afterwards we import the BERT model for Arabic, "asafaya/bert-large-arabic" [26] from the Transformers library, the classifier function using the BERT model is defined, with a Dropout layer set at 30%, and a fully-connected output layer, the defined function, returns the raw output of the Last layer, and we move the classifier to the GPU. We apply a Softmax function to the outputs, in aim to have probabilities for the trained model.

We use AdamW optimizer, from the Transformers library, to correct the weight decay, with a learning rate set to 2e-5; a training loop is defined, by iteration, for 2 Epochs, 4 Epochs, 8 Epochs, 10 Epochs, 12 Epochs, 14 Epochs, and the best results are those of a number of Epochs equal to 10, we store the training history, and when the training process finishes, we get the trained model, which can be downloaded using this Google drive file ID: 1InzP8KVk8C-qLINXa-tSsGM6PftOsW7j, or this URL: https://drive.google.com/file/d/1InzP8KVk8C-qLINXa-tSsGM6PftOsW7j/view?usp=sharing.

2.4.5 Application

To test the result, we create an automatic system for hate speech detection, based on the trained model, in different steps, the araby element is imported from the Pyarabic module, it’s used to tokenize the input text, and split it into small paragraphs and sentences, the input text is classified afterwards using a classifying function with the help of the model, which returns the name of the class as result for the input text.

Text that will be automatically detected for hate language requires specific processing for better performance, this process is essential for two reasons, first, the diacritics and lengthening of the text should be removed so that it is similar to what the training was done on. Second, the text submitted for detection must correspond to the maximum length of the content used in the training, which has been set at 100 tokens, because beyond this length, the detector loses its performance. This step requires the use of PyArabic [35], an Arabic language processing library that allows us both to make the necessary modifications to the input text, as well as its tokenization for conversion into sentences with length that is less than the maximum. We try this system on Arabic and different dialects to test the convenience of the results, found to be accurate in a relation with the metrics, different examples are shared in the Results Chapter.

Thanks to the BERT method, we were able to have a general rate of accuracy in distinguishing hate speech from neutral one of about 83%, with regard to the detection of hate speech, the metrics of the model show a superior performance with a Precision of 0.87, Recall of 0.91, and an F1-Score of 0.89 (Figure 10). In Figure 15 we can see the confusion matrix showing the superiority of the model in detecting hate speech than the neutral language, resulting in an overall precision of 83%.

15.png

Figure 15. Model performance confusion matrix

Compared to the previous work mentioned in Table 1, the model we have trained performs a more specific task of detecting hate speech, and is not interested in abusive language or offensive, which are more general than the concept of hate speech and which requires taking into consideration the context of words and not just their current use, which is why we have chosen the two-way Transformer method.

The preceding works have taken into account only the classical Arabic language, or one determined dialect at a time, but in our case we proceeded through the training of the model to process the classical Arabic language in addition to several other dialects, based on the Arabic language.