Leakage-Aware Arabic Emotion Classification for E-Learning Using Domain-Adaptive Transformer Models and Prototype Feedback Support

Emotional states influence student engagement, motivation, satisfaction, performance, and learning outcomes. Affect is therefore an important component of the educational experience [1]. Self-regulated learning also contributes substantially to academic performance in online and blended learning environments [2]. In these settings, emotional states may reveal difficulties that are not immediately apparent from academic performance alone. For example, anxiety may reflect assessment pressure or technical uncertainty, whereas frustration may result from usability problems or unclear instructional requirements. Identifying such states can support the design of more appropriate educational interventions.

Emotion classification is particularly relevant to e-learning because many behavioral and facial cues available in face-to-face classrooms are limited or absent in online environments. Previous studies have examined emotion recognition as a basis for adaptive educational interventions [3, 4]. Student-written responses, reflections, course reviews, and feedback therefore provide accessible and relatively non-intrusive sources of information about learners’ experiences.

Arabic affective natural language processing (NLP) remains challenging because of morphological complexity, orthographic variation, and dialectal diversity [5]. These challenges are particularly pronounced in informal Iraqi Arabic, which includes colloquial vocabulary, non-standard spelling, and expressions that differ from Modern Standard Arabic. Although several Arabic sentiment and emotion resources are available, resources specifically targeting Iraqi Arabic remain limited. Existing resources are also derived mainly from social-media discourse rather than learner-generated educational text [6, 7].

Pretrained transformer models provide contextual representations that can address some forms of linguistic variability in Arabic text. Bidirectional Encoder Representations from Transformers (BERT) and Arabic-specific models, including ARBERT and MARBERT, have demonstrated effective transfer learning across Arabic varieties [8, 9]. However, transferring models from general-domain or social-media text to educational text may introduce domain mismatch. Domain-adaptive pretraining (DAPT) can reduce this mismatch by further adapting a pretrained model to relevant unlabeled text before supervised fine-tuning [10, 11].

Reliable evaluation is also important because self-reported emotion datasets may contain explicit emotion names or lexical cues that directly indicate the target label. Classifiers may exploit these patterns and produce overly optimistic performance estimates without learning the intended contextual relationships. This behavior represents a form of shortcut learning [12]. To examine this risk, the present study compares RAW and CLEAN preprocessing settings. The CLEAN setting removes predefined label-indicative lexical cues and exact duplicate responses to provide a more conservative evaluation.

Existing Iraqi Arabic resources illustrate both the linguistic and domain gaps addressed in this study. Askar and Sjarif [6] developed a Mesopotamian-Iraqi social-media corpus for sentiment and emotion classification. Almahdawi and Teahan [7] introduced an Iraqi Facebook dataset annotated according to Ekman’s six basic emotions. In the present study, the text from the latter dataset was used without its original labels for Iraqi-dialect DAPT. Although both resources are valuable for Iraqi Arabic affective NLP, they consist of general social-media content rather than educational responses produced by students.

Arabic educational NLP has also received increasing attention. Recent studies have analyzed student feedback and course evaluations using sentiment-analysis and transformer-based methods [13]. However, this work has focused primarily on sentiment polarity or general course evaluation. Fine-grained emotion classification of Iraqi Arabic educational text under leakage-aware evaluation therefore remains insufficiently investigated.

The present study addresses these limitations through a six-class emotion-classification framework based on responses written by Iraqi university students about their e-learning experiences. The study compares RAW and CLEAN preprocessing settings and evaluates classical term frequency-inverse document frequency (TF-IDF)-based classifiers together with four transformer models. The transformer models are assessed before and after Iraqi-dialect DAPT. Their predicted emotions are then integrated into a prototype rule-and-retrieval feedback module. The novelty of the study therefore does not lie in introducing transformer models or DAPT as new techniques. Rather, it lies in evaluating and integrating these methods within a leakage-aware framework designed specifically for Iraqi Arabic educational text.

The main contributions of this study are as follows:

(a) A domain-specific Iraqi Arabic Student-Emotion dataset was constructed from university students’ descriptions of their e-learning experiences.

(b) A leakage-aware evaluation strategy was developed to examine the effects of predefined lexical-cue removal and exact duplicate removal on emotion-classification performance.

(c) Classical TF-IDF-based classifiers and four transformer models were systematically evaluated, with the transformer models assessed before and after DAPT.

(d) The predicted emotions were integrated into a prototype hybrid rule-and-retrieval feedback module for emotion-aware learner support.

(e) The model confidence score was incorporated as an auxiliary signal for downstream feedback selection.

Figure 1 illustrates the architecture of the proposed emotion-aware Arabic e-learning framework. The framework was designed to provide reproducible and interpretable emotion classification together with a prototype feedback module. In this study, emotion classification refers to the supervised prediction of a single dominant emotion from student-written text. Emotion recognition refers to the broader research field, whereas emotion-aware support refers to the use of predicted emotions to guide learner feedback.

Student-written Iraqi Arabic text was processed using two configurations: RAW and CLEAN. As described in Section 3.2, RAW retained validity-filtered text after whitespace removal, whereas CLEAN additionally removed exact duplicate responses and predefined label-indicative lexical cues to reduce shortcut learning and label leakage.

The processed text was evaluated using classical and transformer-based classifiers. The transformer models were also evaluated after DAPT on a separate Iraqi Arabic corpus. The predicted emotion and its confidence score were then passed to the feedback module. The primary pathway used a hybrid rule-and-retrieval strategy, whereas an optional Gemini-based pathway generated short emotion-conditioned feedback. Emotion classification was evaluated using a fixed development/test split and stratified cross-validation (CV). The feedback module was demonstrated qualitatively as a proof-of-concept prototype.

1.png

Figure 1. Overall architecture of the proposed emotion-aware Arabic e-learning framework

3.1 Description and collection of the dataset

The Student-Emotion dataset was collected through a structured Google Forms questionnaire distributed across public and private Iraqi universities. Questionnaire distribution was supported by an official facilitation letter issued by the University of Information Technology and Communications (UOITC). Data collection lasted approximately five months, from September 2025 to February 2026. Participation was voluntary and based on informed consent, and no personally identifying information was retained.

The questionnaire contained eight items covering academic status, gender, age group, e-learning experience, platform-use frequency, and the main difficulty encountered during e-learning. Participants also selected one dominant emotion from six predefined categories: Joy, Anger, Anticipation, Trust, Neutral, and Anxiety. An open-ended item asked participants to describe their emotional experience in Iraqi Arabic. Each labeled instance consisted of the selected emotion and its corresponding free-text response.

The six-category scheme was adopted as a compact, task-specific taxonomy rather than as a comprehensive representation of academic emotions. Joy and Trust represented broadly positive responses, whereas Anger and Anxiety represented broadly negative responses. Neutral represented a non-polar state, while Anticipation represented a future-oriented state. Trust reflected confidence in technology-mediated learning, whereas Anticipation reflected expectations regarding future learning experiences or improvements. The categories were selected to capture affective states that could be directly interpreted from students’ self-reported e-learning experiences and subsequently mapped to different forms of learner support.

The selected taxonomy does not imply that Trust and Anticipation are more important than other learning-related emotions. Frustration, confusion, and boredom are also discussed in educational and affective-learning research [1, 4]. However, these states were not included in the predefined questionnaire. The six-class scheme should therefore be interpreted as a task-specific representation of students’ e-learning emotions rather than as a comprehensive taxonomy of academic affect. Its limited coverage is acknowledged as a dataset limitation.

After initial validation, empty and invalid responses were removed, resulting in 1,176 valid instances. Because no persistent personal identifier was retained, the number of unique participants could not be verified. The records are therefore reported as valid responses rather than as unique participants. Further leakage-control procedures, including duplicate removal, are described in Section 3.2. The six classes were relatively balanced, with proportions ranging from 15.5% to 18.7%.

Although background information was collected, aggregate demographic statistics were not retained in the anonymized modeling corpus. Institution-specific distributions were also unavailable because the questionnaire did not require an institution identifier. These limitations restrict the assessment of dataset representativeness and generalizability. Table 1 presents representative responses from the six emotion categories.

The final modeling corpus contained three fields: a numerical identifier, an Iraqi Arabic free-text response (Post), and its corresponding emotion label (Label).

Table 1. Representative Iraqi Arabic examples from the six emotion categories

3.2 Preprocessing and leakage control (RAW vs. CLEAN)

All student responses underwent validity filtering before the two preprocessing configurations were generated. Records with missing or empty Post or Label values were excluded. Leading and trailing whitespace was also removed from all textual fields.

Two configurations were evaluated: RAW and CLEAN. RAW retained the validity-filtered text without additional leakage-control procedures. CLEAN removed predefined lexical cues that explicitly indicated the six target emotions.

The cue list included فرح، سعيد، مبسوط for Joy; غضب، غاضب for Anger; ثقة، واثق for Trust; ترقب، توقع، انتظار for Anticipation; قلق، خوف، خايف، متوتر for Anxiety; and حياد، محايد for Neutral. The corresponding English class names were also included.

After cue removal, empty responses were discarded. Exact duplicate records were then removed using the processed Post and Label fields. These operations directly defined the CLEAN configuration used in the experiments. Table 2 summarizes the differences between the RAW and CLEAN configurations.

Table 2. Comparison of RAW and CLEAN preprocessing settings

The cue list was manually defined rather than derived from the evaluation data. The first author selected explicit emotion names and common Arabic and Iraqi Arabic variants associated with the six categories. The second author subsequently reviewed the list. This procedure was intended to reduce direct lexical shortcuts and encourage the classifiers to rely more strongly on contextual information.

Because the procedure was lexicon-based, expressions not included in the predefined list could remain in the processed text. These expressions included unseen Iraqi Arabic terms, morphological variants, synonyms, and indirect paraphrases. Additional terms could be incorporated into future versions of the lexicon. Therefore, CLEAN reduced explicit lexical label leakage and shortcut learning but did not eliminate all possible semantic cues.

3.3 Modeling setup: Classical baselines and transformer fine-tuning

Classical machine-learning baselines and transformer models were evaluated using the same CLEAN preprocessing configuration, fixed development/test split, random seed, and evaluation protocol. This setup enabled a controlled comparison between non-contextual lexical representations and contextual pretrained representations.

For the classical baselines, Iraqi Arabic text was represented using TF-IDF word unigrams and bigrams. Classification was performed using a Calibrated Linear Support Vector Classifier (SVC) and Complement Naive Bayes. These models served as lightweight, non-contextual reference baselines. However, their reliance on lexical-frequency features limited their ability to capture contextual information, negation, and subtle distinctions among related emotions.

Four pretrained transformer models were evaluated. MARBERTv2 was selected because it was pretrained on Arabic social-media and dialectal text, which made it suitable for informal Iraqi Arabic. AraBERTv2 was included as an Arabic-specific baseline. Multilingual BERT (mBERT) and XLM-RoBERTa (XLM-R) were included as multilingual baselines with different pretraining regimes. Each transformer model was fine-tuned for six-class, single-label emotion classification by adding a classification head and updating the model parameters using the labeled Student-Emotion dataset.

The selected models represented complementary modeling approaches, including conventional lexical, Arabic-specific, dialect-oriented, and multilingual representations. This model set was considered appropriate for the comparative objectives of the study because all approaches could be evaluated under the same preprocessing and evaluation protocol. The comparison was designed to address two experimental questions: whether contextual pretrained representations outperformed TF-IDF-based baselines and whether Iraqi-dialect DAPT improved transformer performance. The objective was not to provide an exhaustive benchmark of recent Arabic emotion-classification architectures. Instead, the study aimed to conduct a controlled comparison aligned with its specific experimental objectives.

3.4 Domain-adaptive pretraining

A two-stage training strategy was adopted to address Iraqi-dialect variation and the limited amount of labeled educational data. In the first stage, DAPT was performed using the Iraqi Arabic Emotion Corpus introduced by Almahdawi and Teahan [7]. The original corpus was manually collected from Facebook between December 2016 and August 2018 using emotion-related seed words and Facebook search filters, including a Baghdad-Iraq location constraint. Manual collection was used to reduce noise from non-textual content, such as links and images.

The original study reported 1,365 posts and evaluated the corpus using four external annotators, with an average inter-annotator agreement of 0.751. The version used in the present experiments contained 1,361 Iraqi Arabic Facebook posts covering the same six Ekman emotion categories: Anger, Disgust, Fear, Happiness, Sadness, and Surprise.

Only the text field was used during DAPT. The original emotion labels were discarded because adaptation was performed through masked language modeling (MLM) rather than supervised emotion classification. Missing and empty entries were removed, and the remaining posts were treated as unlabeled Iraqi Arabic text. This stage adapted the transformer representations to Iraqi-dialect vocabulary, colloquial expressions, informal writing styles, and non-standard linguistic patterns.

The Facebook corpus was selected because it shared important linguistic characteristics with the Student-Emotion dataset. However, the two corpora represented different domains. The Facebook posts covered broad social and personal topics, whereas the Student-Emotion responses focused on academic, technical, and organizational aspects of e-learning. Therefore, the Facebook corpus was used only for dialect-oriented adaptation and not as a substitute for educational training data.

In the second stage, the adapted models were fine-tuned on the labeled Student-Emotion dataset for six-class emotion classification. Task-specific learning was based exclusively on the educational dataset, while the Facebook corpus provided only additional exposure to Iraqi Arabic. The held-out Student-Emotion test data were excluded from both DAPT and fine-tuning and were used only for final evaluation. Figure 2 illustrates the two-stage training protocol and the separation of the adaptation, fine-tuning, and test data.

2.png

Figure 2. Leakage-aware data separation protocol

3.5 Labeling strategy, annotation reliability, and confidence-based feedback control

Each Student-Emotion instance consisted of a short text paired with one self-selected emotion category from the predefined questionnaire. To assess annotation reliability, 200 responses were randomly selected and independently labeled by a second annotator who was blinded to the original labels. Inter-rater agreement was measured using Cohen’s kappa, as shown in Eq. (1):

$\kappa=\left(P_o-P_e\right) /\left(1-P_e\right)$              (1)

where, P_o denotes the observed agreement between the two raters, and P_e denotes the agreement expected by chance.

Annotation reliability and model confidence served different purposes within the proposed framework. Cohen’s kappa was used during dataset validation to assess agreement between human label assignments. At inference time, the classifier produced a predicted emotion label ê and an uncalibrated model-confidence score c. The score was defined as the maximum Softmax probability across the emotion classes, as shown in Eq. (2):

$c=max ({Softmax}(z))$                (2)

where, z denotes the model logits.

The confidence score was used as an auxiliary control signal rather than as a calibrated measure of predictive reliability. When c < 0.40, the rule-and-retrieval pathway produced a general clarification-oriented response instead of making a strong emotion-specific assumption. Predefined greeting, gratitude, and congratulatory intents were excluded from this rule. When c < 0.60 and the probability assigned to a secondary emotion exceeded 0.25, the prototype could also include the secondary predicted emotion in the response.

These thresholds were heuristic prototype settings and were not derived through formal probability calibration. Accordingly, the confidence score should be interpreted as an operational control signal rather than as a calibrated estimate of predictive reliability. Formal probability calibration should be evaluated before confidence scores are used in higher-stakes educational decisions.

3.6 Hybrid emotion-aware feedback delivery via rule and retrieval, with an optional large language model-based alternative

The feedback module operated after emotion classification, as illustrated in Figure 3. Its primary pathway combined rule-based decision logic with the retrieval of previously stored responses. A large language model (LLM) pathway based on Gemini was included only as an optional exploratory alternative.

3.png

Figure 3. Hybrid emotion-aware feedback architecture with primary rule-and-retrieval and optional Gemini-based pathways

Given a student message x, the best-performing saved classifier predicted the emotion ê and its associated confidence score c. In parallel, a lightweight context-detection module used keyword-based heuristics to infer the message intent or topic and to identify the likely issue category, such as internet connectivity, platform access, time constraints, or deadlines. A short key phrase was also extracted using TF-IDF over sentence segments. The system then represented the case as the tuple (ê, intent, topic, issue).

The retrieval module searched the stored feedback memory, feedback_database.pkl, using multilingual sentence embeddings. A stored response was reused when its cosine similarity reached the predefined threshold of 0.78. Otherwise, the rule-based pathway constructed a response using curated templates, tone configurations, and issue-specific playbooks. Newly constructed rule-based responses could subsequently be stored for reuse in similar cases.

For example, a message expressing anxiety about weak internet connectivity could be assigned the emotion Anxiety, the intent venting, and the issue category internet/platform. If no stored response reached the similarity threshold, the system applied the corresponding rule-based blueprint, such as Anxiety + venting.

Gemini was used only to generate or paraphrase short Arabic responses based on the student message, predicted emotion, and confidence score. This optional pathway was not formally evaluated and was not required for prototype operation. The primary rule-and-retrieval pathway therefore operated independently of Gemini.

3.7 Evaluation metrics

Model performance was evaluated using accuracy, Macro-F1, and confusion matrices. Accuracy represents the proportion of correctly classified instances. Macro-F1 assigns equal weight to all emotion classes by averaging their class-specific F1 scores. Confusion matrices were used to examine class-level errors and recurring misclassification patterns.

For the six-class, single-label classification task, accuracy was calculated as shown in Eq. (3):

$\text {Accuracy}=\frac{\sum_{i=1}^K C_{i i}}{N}$               (3)

where, C_ii denotes the number of correctly classified instances for class i, and N denotes the total number of evaluated instances.

Precision and recall were calculated using Eqs. (4) and (5), respectively:

$\text { Precision }=T P /(T P+F P)$              (4)

$\text { Recall }=T P /(T P+F N)$               (5)

where, TP, FP, and FN denote true positives, false positives, and false negatives, respectively.

The F1-score was calculated using Eq. (6):

$\mathrm{F} 1=\frac{2 \times \text {Precision} \text {×} \text {Recall}}{\text {Precision}+ \text {Recall}}$               (6)

Finally, Macro-F1 was calculated using Eq. (7):

$\text { Macro }-\mathrm{F} 1=\frac{1}{K} \sum_{i=1}^K \mathrm{~F} 1_i$              (7)

where, K denotes the number of emotion classes and F1_i denotes the F1 score for class i.

This study presented an emotion-aware Arabic e-learning framework that combined leakage-aware emotion classification with a prototype feedback module. The combined removal of explicit lexical cues and duplicate records resulted in a more conservative estimate of model performance. MARBERTv2 achieved the strongest overall performance, whereas the effects of DAPT varied across transformer models. These findings indicate that Iraqi-dialect adaptation may improve performance for some model architectures under the present experimental setting, but its effect is model-dependent.

Several limitations should be considered when interpreting these findings. The Student-Emotion dataset was based on self-reported responses from Iraqi university students and used a fixed six-category emotion taxonomy. Its relatively limited size, together with the absence of detailed demographic and institution-level information, restricted the assessment of representativeness and generalizability. The dataset also reflects the cultural and linguistic characteristics of Iraqi students, which may limit the transferability of the findings to other Arabic-speaking contexts and dialects. Possible self-report bias should also be considered.

Additional limitations relate to domain adaptation and prototype evaluation. The DAPT corpus originated from social media rather than an educational domain, and the confidence thresholds were heuristic rather than formally calibrated. Moreover, the feedback module was neither evaluated with real students nor deployed within an operational learning platform. Accordingly, the proposed framework should be regarded as a research prototype rather than as a validated educational tool.

Future research should examine larger and more diverse Arabic educational datasets and evaluate generalization across Arabic dialects and cultural contexts. The emotion taxonomy could also be extended to include learning-related states such as frustration, confusion, and boredom. Further work should include structured expert assessment, probability calibration, comparisons with baseline feedback methods, and evaluations involving students and instructors in real educational settings.