Fine-Tuning BERT Based Approach for Multi-Class Sentiment Analysis on Twitter Emotion Data

Social networks have grown, and several social services have expanded exponentially and have attracted millions of users in a small space of time. Thus, these social networks play an essential role in the lives of individuals as they chat and post their day-to-day events, providing an unparalleled pool of raw data that can be used for commercial and non-commercial purposes. Any of the profitable fields in which this data is collected is to assess the users' emotions or feelings in the areas of sentiment analysis (SA) and emotional analysis (EA) [1, 2]. As the names SA and EA mean, they determine the feelings and emotions in a piece of document, respectively. Both are commonly treated as training and evaluation data sets for binary or multi-class supervised learning problems. For SA, the feelings are typically considered (positive vs negative) or (positive vs neutral vs negative). About EA, most of the existing publications in the literature either adopt the six essential emotions of Ekman (Anger, Disgust, Fear, Joy, Sadness, Surprise, etc.) or the eight fundamental emotions of Plutchik (which are, besides anticipation and confidence, the six emotions of Ekman) [3]. The EA area differs considerably from that of SA. Any emotions can be feelings about a finer granularity. People may easily see them as negative sentiments, but they’re distinct [4]. EA should then be achieved over the (relatively) easier SA as a supplement layer. It increased the difficulty of the problem when dealing with a more significant number of groups. In addition, it may not be obvious the boundary between two emotions. Indeed, the same text extract may simultaneously convey different emotions, making the classification problem a multi-label text [5, 6]. The problem of the transmission of different feelings (conflicting or mixed sentiments), on the contrary, is neglected in SA. Sentiment analysis employs computer software through which the emotional tone behind words is determined. The study of feelings is not a novel phenomenon.

There are thousands of labelled datasets, from basic positive and negative schemes to more complicated systems that decide whether a text is positive or negative. The usage of a pre-labelled dataset of Twitter tweets that are positively or negatively marked for this article. Using this data, the generated model that classifies every tweet with Scikit-learn as either positive or negative. In tweet sentiment analysis, most scientific literature has obtained state-of-the-art results through taking a training language model approach directly from scratch, starting with corporate tweets, to enable models to better handle tweet jargon, characterized by a particular syntax and grammar without a point, with contract jargon being a common approach to Twitter determination based on whether a tweet is positive or negative, also regarded as an interpretation of binary (or polar) emotions [7, 8]. In this area of research, machine learning algorithms including Naive Bayes, Random Forest, Regression of logistic systems and classifiers of linear support vectors have been proven to be efficient [9, 10].

As a multi-class sentiment classification problem, the goal is to classify texts into five emotion categories: joy, sadness, anger, fear, and neutral (or a combination of these). This article presents dataset preparation, traditional machine learning (e.g., Naive Bayes, Random Forest, Logistic Regression, and SVM) with Scikit-learn, and transfer learning to use the suggested BERT algorithm (TensorFlow Keras). Learn how to preprocess and tokenize data for BERT classification, build TensorFlow input pipelines for text data, and train and evaluate a fine tuned BERT model for text classification in this paper.

In this section, we outlined the proposed solution for tweet classification and sentiment interpretation. It is essentially a two-stage pipeline, as illustrated in Figure 1. The first stage comprises a series of pre-processing processes that convert Twitter jargon into plain text, such as emoji’s and emoticons. The second stage is a classification scheme based on an M-BERT language model pre-trained on a single text corpus. This step entails the processing of the processed data. To classify five emotional states, we employ the TF-IDF (term frequency inverted text frequency) method. Vectorised terms are used to prepare a data set and regular expressions. Experts classify sentences into five types of sentiments: surprise, sadness, anger, fear, and happiness.

1.png

Figure 1. Process flow of two-stage pipeline of the system

3.1 Dataset description

After determining the emotional model, acquiring data specific to the course is the next important phase in identifying emotions from the text. There are several freely accessible, structured, annotated ED databases for testing. The data collection we will use is publicly accessible in this article. The Emotion-Text Dataset was 70% divided into training, and 30% tested. The dataset was combined from regular dialogue [23], isear [24], and emotional stimulus [25] to include a balanced dataset containing five different labels: joy, sadness, anger, fear, and surprise, and even the Smile dataset [26] includes other annotations of disgust, anger, sadness, happiness, and surprise. We composed mostly the texts of short messages and speeches.

3.2 Data pre-processing

Before applying the final sentiment analysis techniques, we execute the pre-processing phase. We used the following phases in the typical production process: The raw Twitter data usually includes unique emojis (e.g. #,?...).

1. Stop word Removal: stop words are sometimes used in the real meaning of a document that has little contribution (e.g. a, and, the, etc.). Then the removal of the stop word is performed.

2. Stemming: In addition, variations of a base term, such as stems, are used in most sentences. Therefore, the variants of words are often simplified to their root shape as stems (for example, "joyful" or "joyful" are reduced to "joy" stems). This is used for the function of Porter’s stemming algorithm.

3. Tokenization: The word is further separated into actual token words. These are both measures for cleaning the data and preparing it for further study.

3.3 Feature extraction or data (text) representation

TF-IDF: Statistical algorithms use mathematics to understand model machine learning. Although numbers function in mathematics, we first have to translate the document into numbers to allow mathematical algorithms to perform for text. To achieve this, there are three major approaches: Word Bag, TF-IDF, and Word2Vec. We used the TF-IDF system in this current study.

3.4 Classifier model

Because the analysis has a multi-class dilemma, we discovered that the multi-class text classification of the NB is superior to multinomial Naive Bayes (MNB).

Naive Bayes: The Naive Bayes (NB) classification has proven efficient for the study of sentiments. The principle behind the Bayes classification is the Bayes theorem to evaluate the probability of a feature vector of a class S.

$P(s \mid \vec{f})=\frac{P(\vec{f} \mid s) P(s)}{P(\vec{f})}$          (1)

The simplistic (name-listed) hypothesis is that all characteristics are independent of each other. Because of this supposition, and since P(f~) for all groups is the same, equation 2 can find the most likely class ~ s for a variable f~.

$\hat{s}=\arg \max P(s) \prod_{j=1}^{n} P\left(f_{j} \mid s\right)$

$s \in S$         (2)

Random Forest: It is a controlled classifier using a combination of learning technology in various training stages of decision-making bodies. The number of trees calculated the exactness of the classifier and increased the number of trees in the method to improve the system's precision. The Decision Tree is a structure with a monitoring testing dataset, and these trees generate rules for each class during training. They then used these principles as the evaluation data parameters that may vary from the training data collection. The random forest selects some features from all the features provided for each class at the beginning of the algorithm. The test dataset progresses through every tree specified during the learning stage. We then assessed the target votes by comparing each prediction target. The Random Forest algorithm has a benefit that was never generated compared to the results of the classification. Since the problem model is often a minor error in the Random Forest, we may also use this technology for regression, classification, and extraction tasks.

Logistic regression: It is a regression class used to forecast the dependent variable of an independent variable. The Dual logistic regression is where the dependent variable has two types. If the dependent variable has over two types, the logistic regression is multinomial. If the vector type is classified, then the ordinal logistic regression (OLS). Turn the dependent variable into the logic function to achieve the highest probability estimate. Logit is primarily a natural log of the variable which shows whether the occurrence will take place. Ordinal logistic regression does not assume a linear association between the contingent and the independent variable.

Multi-Support Vector Machine (MSVM): We monitor it using a machine learning algorithm used in classification and regression models. Compared with artificial neural networks, the SVM method provides better classification outcomes since it has a plethora of generalizations that prevent the algorithm from overfitting. The SVM is a versatile machine that learns to handle nonlinear data for regression and classification problems with an appropriate kernel feature. The two key hyper-parameters are C and γ in the SVM algorithms. We should adjust the hyper-parameters before training the model. We used the overall parameters to train the SVM according to the training information. The model on the validation collection is then checked. Finally, on the reference dataset, we tested the model. We used linear, polynomial and RBF as an SVM kernel and as a decision function one-on-one (ovo). After a comparative analysis, when attempting various parameters, such as C, γ and kernel. In training, validation and testing, we have observed that classifiers of almost all kinds of combinations end up the same. It’s because SVM was first designed for binary classification, which is an issue for multi-class classification. We use ovo, which means they are binary versus one. We have, however, noticed that SVM could not be so appropriate for this task. The outcome shows that data precision is not so satisfactory. Following specific exploration, we found three multi-class problem-solving systems. They are ovr (one vs residue), M (M-1)/2 and expand the SVM formula.

The adapted classifier has the following form.

$f_{t a r}(x)=\sum_{k=1}^{M} \tau^{k} f_{\operatorname{src}}^{k}(x)+\Delta f(x)$        (3)

where, $\tau^{k} \in(0,1)$  is the weight of each base classifier $f_{\operatorname{src}}^{k}(x)$, $\Delta f(x)$ is the perturbation function that is learnt from a small set of labelled target-domain data in $D_{\text {tar }}^{l}$. As shown in the study [27] it has the form:

$\Delta f(x)=w^{T} \phi(x)=\sum_{i=1}^{N} \alpha_{i} y_{i} K\left(x_{i}, x\right)$          (4)

where, $w=\sum_{i=1}^{N} y_{i} y_{i} \phi\left(x_{i}\right)$  is the model parameters under which the labelled example $D_{\text {tar }}^{l}$ and $a_{i}$  is to be measured and where is the target-domain case ith-labeled function coefficient. Non-linear mapping of features often caused the kernel function as $K(., .) \equiv \phi(.)^{T} \phi(.)$. They learnt it $\Delta f(x)$ in regularized scientific risk minimization [28]. The adapted classifier $f_{\text {tar }}^{(x)}$  studied in this context seeks to minimize the classification error in the target-domain examples mentioned and the distance from the basic classifiers $f_{S r c}^{k}(x)$ to achieve a more bias-variance. We use an expanded multi-classifier adjust mechanism suggested by Yang & Hauptmann [29] to enable the weight controls $\left\{\tau^{k}\right\}_{k=1}^{M}$  based on the weight output of the limited range of target-domain examples in the base classifiers $f_{S r c}^{k}(x)$  to be automatically learned. To do this, the adaptive classifier is now written as:

  $\min _{w, \tau, \xi} \frac{1}{2} w_{w}+\frac{1}{2} B(\tau)^{T} \tau+C \sum_{i=1}^{N} \xi_{i}$

 s.t. $y_{i} \sum_{k=1}^{M} \tau^{k} f_{s r c}^{k}(x)+y_{i} w^{T} \phi\left(x_{i}\right) \geq 1-\xi_{i}$

$\xi_{i}^{m} \geq 0, \forall\left(x_{i}, y_{i}\right) \in D_{s r c}$       (5)

where, $\frac{1}{2}(\tau)^{T} \tau$  the total input of the base classifiers is steps, is added to the regularized loss minimization process. This objective function aims to prevent dependency and over-complexity $\Delta f(.)$  on the basic category. Parameter B balances the two targets. By rewriting the objective function as a Lagrange (primal) function minimization problem and setting its derivative w, and ξ to zero, we have:

$w=\sum_{i=1}^{N} \alpha_{i} y_{i} \phi\left(x_{i}\right) \tau^{k}=\frac{1}{B} \sum_{i=1}^{N} \alpha_{i} y_{i} f_{s r c}^{k}\left(x_{i}\right)$        (6)

where, $\tau^{k}$  a weighted sum $y_{i} f_{s r c}^{k}(x i)$ and the classification performance of the target domain. Consequently, if we classify the labelled destination domain info well, we have allocated more massive base classifiers. With the new decision function provided (3), (4) and (6) now:

$f_{t a r}(x)=\frac{1}{B} \sum_{k=1}^{M} \sum_{i=1}^{N} \alpha_{i} y_{i} f_{\operatorname{src}}^{k}\left(x_{i}\right) f_{\operatorname{src}}^{k}(x)+\Delta f(x)$

$=\sum_{i=1}^{N} \alpha_{i} y_{i}\left(K\left(x_{i}, x\right)+\frac{1}{B} \sum_{k=1}^{M} f_{\operatorname{src}}^{k}\left(x_{i}\right) f_{\operatorname{src}}^{k}(x)\right)$        (7)

Compared with (7) a regular SVM model $f(x)=\sum_{i=1} a_{i} y_{i} K\left(x_{i}, x\right)$, this multi-classification adaptation model may be interpreted as applying additional features to the projected labels of the basic classifiers in the target domain. The scalar B combines the influence of the initial characteristics and more features according to this interpretation.

3.5 Fine-tuning BERT for sentiment analysis

BERT is a pre-training technique created by Google for NLP (Natural Language Processing) [30]. We designate BERT to pre-train deep bidirectional representations from an unlabeled document by shaping both left and right instances in both layers. As a result, only one additional output layer will complete the pre-trained BERT model to construct state-of-the-art models for various tasks, including answering questions and language inference, without significant task-related changes to the design, as seen in Figure 2.

We used the library of transformers and designed a model of BERT. But it wasn’t appropriately prepared because of some unknown issue. We moved and developed the model into another library named Ktrain. "Ktrain is a lightweight wrapper for the TensorFlow Keras (and other libraries) deep learning library to support the development, training, and deployment of neuronal network and other learning systems models. We used BERT from the Ktrain library.

2.png

Figure 2. Architecture of BERT model

Proposed fine tuning of modified BERT: When we evaluated the efficiency of the MBERT to that of a baseline model that uses a TF-IDF vectorizer and an MSVM classifier. The transformers library allows us to fine-tune the cutting-edge BERT model fast and inexpensively, yielding an accuracy rate higher than the baseline method.

Step 1: Simply install the Hugging Face Library

This includes PyTorch variants of BERT (from Google), GPT, and pre-trained network weights.

Step 2: Tokenization and Input Formatting

Prior to actually tokenizing our content, we will perform some basic text processing, such as extracting entity mentions (e.g., @united) and even some special characters. Since BERT was trained with whole sentences, the level of processing here is substantially lower than in previous approaches. We must utilize the tokenizer given by the library in order to apply the pre-trained BERT.

This is due to the fact that

• The model has a set vocabulary and
• The BERT tokenizer has a specific way of dealing with out-of-vocabulary terms.

Furthermore, we must add special tokens to the beginning and end of each sentence, pad and truncate all sentences to a single consistent length, and specifically define which tokens are padding tokens using the "attention mask."

Step 3: Train Our Model

BERT-base is made up of 12 transformer layers, each of which takes in a list of token embeddings and outputs the same number of word embedding with the same hidden size (or dimensions). The output of the [CLS] token's final transformer layer is being used as the sequence's features to feed a classifier.

The Bert for Sequence Classification class in the transformers library is intended for classification tasks. Therefore, we will develop a new class in order to specify our own classifiers. In the following section, we will build a Bert Classifier class that uses a BERT model to extract the last hidden layer of the [CLS] token and a single-hidden-layer feed-forward neural network as the classifier.

Step 4: Optimizer and Learning Rate Scheduler

We'll need an optimizer to fine-tune our Bert Classifier. We set batch size to be 32, learning rate to 5e-5, and number epochs be 4.

Step 5: Build a Training Loop

For 4 epochs, we will train the Bert Classifier. We will train the model and evaluate its performance on the validation set at the end of each epoch. Compute the average loss across all training data. After each training epoch, evaluate the performance of the model on our validation set and computes the accuracy rate.