Multi-Modal Affective Computing: An Application in Teaching Evaluation Based on Combined Processing of Texts and Images

MAC could exert an important role in teaching evaluation since it integrates the processing of texts and images, detects and judges students’ affection state based on the content of submitted text feedback and images of their facial expressions, provides personalized learning resources and suggestions according to their affection state and learning requirements, and figures out students’ understanding and acceptance of course content by analyzing text answers and images uploaded by students in online tests or surveys, in addition, it could also evaluate teachers' performance and assist mental health specialists to analyze students’ affection state. These applications are conductive to improving teaching quality, education equity, and the all-round development of students.

To apply MAC in teaching evaluation requires to collect these text and image data: 1) Student text data: including assignments and exam answers, text data from classroom discussions, forums, chat tools and other scenarios, and their feedback and comments about teachers, courses, and teaching methods, so as to understand their satisfaction and needs; 2) Student image data, including students’ facial expressions in class, their  postures and movements, and images they used in their paintings, designs, crafts, and other course works; 3) Teacher text data, including teaching plans and lecture notes, and text data drawn from interactions and communications between the teacher and students; 4) Teacher image data, including the teacher's facial expressions in the class, as well as the postures and movements.

In this study, the texts entered were divided into two parts: main body and hash tag. The text data of teaching evaluation usually has a high dimension, and the word fragment vector expression could convert the high-dimensional text information into the vector form with a lower dimension, thereby reducing computation complexity and improving the efficiency of model training and prediction. When turning texts into a vector form using word fragment-based algorithms, the semantic information of original texts could be well preserved, which is good for increasing the accuracy of MAC models in analyzing text features. The idea of this method is to convert texts into mathematical vectors, so as to support comparison and similarity calculation between texts, and this facilitates the recognition of texts with similar affection features in teaching evaluation scenarios and helps increasing the accuracy of affection calculation. Figure 1 shows the principle of word segmentation algorithm.

1.png

Figure 1. Principle of word segmentation algorithm

Assuming: u represents the u-th sample, Y^q_u represents the input main body sequence of teaching evaluation texts, Y^y_u represents the hash tag sequence, then the two pars of texts (main body and hash tag) can be written as Y_u={Y^q_u,Y^y_u}; also, assuming q_k represents the k-th word in the main body sequence of teaching evaluation texts, l represents the length of main body sequence, then the input main body sequence of teaching evaluation texts can be written as Y^q_u={q₁,q₂,...,q_k,...,q_l}; similarly, assuming y_k represents the k-th hash tag in the hash tag sequence, j represents the length of hash tag sequence, then the hash tag sequence can be written as Y^y_u={y₁,y₂,...,y_k,...,y_j}.

The process of transforming teaching evaluation texts into vectors is described by the following formula:

$G_{0}^{y}={{d}_{TE}}\left( Y_{u}^{q};{{\varphi }_{TE}} \right),G_{0}^{y}\in {{E}^{l\times f}}$    (1)

After the input teaching evaluation texts were preprocessed, they were converted into word vectors in two steps. Assuming: q_k represents the input word, C represents the size of word list generated correspondingly, the value of C equals to the size of one-hot vector, {0,0,0,...,0,1,0,...,0}∈E^C represents each word vector in one-hot encoding; if the dimension of the word vector is represented by f, then the weight initialization matrix between the input layer and the output layer can be characterized by a matrix Q_r with a size of C×f.

$Q_r=\left(\begin{array}{cccc}q_{11} & q_{12} & \cdots & q_{1 n} \\ q_{21} & q_{22} & \cdots & q_{2 n} \\ \cdots & \cdots & \cdots & \cdots \\ q_{C 1} & q_{C 2} & \cdots & q_{C n}\end{array}\right)$        (2)

Hash tags (such as: #keyword) may contain important affection information in the teaching evaluation texts, namely keywords or phrases that are closely related to the content of the texts, which are helpful in more accurately analyzing the affection features of the texts. The encoder based on gate control mechanism can effectively extract these key information and provide more valuable input for subsequent affection calculations. Moreover, keywords in hash tags may have similar or same semantics with words in the main body of the texts. By encoding and distinguishing the hash tags, the weight of these keywords in text vectors could be increased, so as to better capture the semantic information of the texts.

Assuming: y_k represents the k-th hash tag in a sample, j represents the number of hash tags, then the operation of removing the “#”symbol from each hash tag can be written as Y^g_u={y₁...,y_k,...,y_j}; in the meantime, assuming G^g₀ represents the hidden state of the generated hash tag, d_TE represents the network for processing the texts, ϕ_BM represents the parameter to be learnt by the network, G^g_u represents the eigenvector of tag, d_BM represents the network for encoding hash tags, then the process of hash tag encoding can be expressed by the following formula:

$G_{0}^{g}={{d}_{BM}}\left( Y_{u}^{g};{{\varphi }_{BM}} \right),g_{0}^{g}\in {{E}^{l\times f}}$     (3)

Similar to the processing of main body of the texts, hash tags should be expressed in the vector form as well, assuming C_g represents the number of hash tags, Q_g∈E^Cg×f represents a a learnable matrix, then there is:

$r_k^g=Q_g \cdot y_k$     (4)

The initialization parameters of the neural network could be attained by uniformly sampling in the interval, let b_SR and b_SC respectively represent the number of neurons in the input layer and output layer, there is:

$q \sim I\left(-\sqrt{\frac{6}{b_{S R}+b_{S C}}}, \sqrt{\frac{6}{b_{S R}+b_{S C}}}\right)$   　　(5)

2.png

Figure 2. Principle of hash tag extraction algorithm

To apply MAC in teaching evaluation, a necessary work is to ensure which hash tags automatically learnt by the model during training are related to affections or not. In this way, the model can focus on hash tags containing affection information when dealing with teaching evaluation texts, then the accuracy of affection calculation could be improved. To implement the above algorithm, the soft gate control method was adopted to optimize the encoding method based on gate control network. Instead of simply classifying the tags into two categories (pass or not pass), the soft gate control method allows multiple hash tags to pass through with a certain probability, in this way, the relative weight between tags could be kept so that the model can better capture the differences between hash tags, and the effect of affection calculations could be enhanced. Figure 2 shows how the hash tag extraction algorithm works.

By using a vector i_v ∈E^f to represent the global context representation of the teaching evaluation texts, the calculation of similarity between the current y_k and the corresponding i_v can be completed through the gate control mechanism, based on which the probability of y_k passing the gate control mechanism can be judged. Specifically, assuming r^g represents the vector form of y_k, which was subjected to non-linear transformation and its hidden expression was learnt; assuming i^g_k represents the hidden expression of the k-th hash tag, then the transformation matrix and bias term can be represented by Q_gg∈E^f×f and y^y_g∈E^f respectively, and this process can be described by the following formula:

$i_{k}^{g}=\operatorname{Tanh}\left( {{Q}_{gg}}\cdot r_{k}^{g}+n_{g}^{y} \right)$　　　　　(6)

The sigmoid function was selected to calculate the probability β_k of each hash tag being selected by the gate control mechanism:

${{\beta }_{k}}=\frac{1}{2+{{e}^{-i_{k}^{Y}\cdot {{i}_{v}}}}}$　　　　　(7)

Then, get the weighted sum of the hidden expressions of all hash tags:

${{r}^{g}}=\sum\limits_{k=1}^{j}{{{\beta }_{k}}\cdot r_{k}^{g}}$　　　　(8)

At last, the expressions of hash tags were subjected to a linear transformation to merge the information of all modes in the same semantic space. Assuming: G^g₀∈E^f represents the final output hash tag, Q_gg∈E^f×f represents the transformation matrix, then there is:

$G_{0}^{g}={{Q}_{gg}}\cdot {{r}^{g}}$　　　　　(9)

After feature extraction of the collected text and image data was completed, to further realize the application of MAC in teaching evaluation based on combined processing and texts and images, the MAC model was divided into modal sharing tasks and modal private tasks to attain a better adaptability in case of new teaching evaluation scenarios. The modal sharing tasks can integrate information between different modes, and extract cross-modal sharing features, which is conductive for the model to better capture the relationship between texts and images, and improve the accuracy of affection calculation. Modal private tasks focus on the internal features of each mode, and could retain the unique information inside the modes, in this way, when the model merges the multi-modal information, it won’t lose the features of each mode, and it could better reflect the affection information in teaching evaluation scenarios. Via such task division operation, the generalization ability of the model could be enhanced. This is because modal sharing tasks can learn common features across modes, while modal private tasks focus on the features of each mode. This design enables the model to have a higher adaptability when coping with new teaching evaluation scenarios.

4.png

Figure 4. Principle of modal private task

The key to realizing modal private tasks (Figure 4) lies in the prediction and analysis of each mode, in this way, the unique information insides each mode could be dug out, namely the Dirichlet distribution parameters and the subjective opinions. This helps to prevent the loss of modal features when merging multi-modal information, thereby better displaying the affection information in teaching evaluation scenarios. After that, prediction results of each mode were merged according to the D-S Evidential Theory. Since Dirichlet distribution parameters and subjective opinions can provide an uncertainty matric for the prediction results of each mode, so we need to apply the D-S Evidential Theory to effectively processing those incomplete, uncertain, and conflicting information, so that the model can merge the information flexibly between modes, thereby improving the adaptability and accuracy of the model in teaching evaluation scenarios.

Assuming: L^y and L^c represent the output subjective opinions of each mode, β^y and β^c represent the Dirichlet distribution parameters, B represents the sample number, J represents the sample type number, $\hat{t}_v=\left[\hat{o}_1, \hat{o}_2, \ldots, \hat{o}_J\right]$ represents the final output prediction results of each mode merged based on the D-S Evidential Theory, i represents the uncertainty estimate, then the loss function can be expressed as:

$Los{{s}_{UN}}=-\sum\limits_{u=1}^{B}{\sum\limits_{k=1}^{J}{{{t}_{uk}}\log \left( {{{\hat{o}}}_{uk}} \right)}}$     (14)

The key to realizing modal sharing tasks (Figure 5) is to construct a feature fusion network based on tensor decomposition and use it to effectively fuse the features of different modes, and integrate the multi-modal information of texts and images into a unified feature space. This conduces to enhancing the ability of the model in capturing affection information in teaching evaluation scenarios. With the help of tensor decomposition, the computation complexity can be reduced and the running efficiency of the model can be improved during the process of feature fusion. By performing low-rank approximation on multi-modal features, the dimension of fused features could be reduced greatly, so the computation load of the model could be reduced as well. At last, the multi-layer perception (MLP) was adopted for prediction classification based on the feature fusion results. MLP can effectively fuse the multi-modal features, decrease computation complexity, capture high-level interactive information, realize end-to-end training, and has good flexibility and extensibility, by adopting MLP, more valuable model designs could be provided for the application of MAC in teaching evaluation scenarios.

5.png

Figure 5. Principle of modal sharing task

Assuming: g_y and g_c represent eigenvectors of texts and images after going through modal representations; after fusion, the feature tensor can be represented by X=x_y⊗x_c, the product of tensors can be represented by ⊗, dimension of the eigenvectors can be represented by f_y and f_c, and there is X∈E^fy×fc, then the eigenvectors of each model can be expanded by one dimension using the above formulas:

$\begin{align}  & {{x}_{y}}=\Gamma \left( {{G}_{y}},1 \right),{{x}_{y}}\in {{E}^{{{f}_{y}}}} \\ & {{x}_{c}}=\Gamma \left( {{G}_{C}},1 \right),{{x}_{c}}\in {{E}^{{{f}_{c}}}} \\\end{align}$     (15)

Assuming: Q represents the weight of linear layer, n represents the bias, X represents the third-order tensor, Q∈E^fy×fc×fG represents the fourth-order tensor, the extra dimension is the dimension of output vector f_G, g is output through linear layer h("), then there is:

$G=h\left( X;Q,n \right)=Q\cdot X+n,g,n\in {{E}^{{{f}_{G}}}}$     （16）

Q can be regarded as a superposition of f_G second-order tensor Q^~_j ∈E^fy×fc. For any Q^~_j ∈E^fy×fs×fc, it can be decomposed with rank E as follows:

${{\tilde{Q}}_{j}}=\sum\limits_{u=1}^{E}{q_{y,j}^{\left( u \right)}\otimes q_{c,j}^{\left( u \right)}}$      （17）

Assuming: q^(u)_y,j ∈E^fy and q^(u)_c,j ∈E^fc represent the decomposition factors of tensor Q^~_j with a modal rank of E corresponding to texts and images, here E was set as a constant, the E-rank decomposition factor {q^(u)_y,j,q^(u)_c,j}^Eu₌₁, j=1,...f_G can be used to re-construct tensor Q^~_j. Let q^(u)_y=[q^(u)_y,1,q^(u)_y,2,...q^(u)_y,fG], q^(u)_s=[q^(u)_s,1,q^(u)_s,2,...q^(u)_s,fG], q^(u)_c=[q^(u)_c,1,q^(u)_c,2,...q^(u)_c,fG], then the weight Q in Formula 16 can be represented by the following formula:

$Q=\sum\limits_{u=1}^{E}{q_{y}^{\left( u \right)}\otimes q_{s}^{\left( u \right)}\otimes q_{c}^{\left( u \right)}}$     (18)

Further, Formula 16 can be written as:

$\begin{align}  & g=\left( \sum\limits_{u=1}^{E}{q_{y,j}^{\left( u \right)}\otimes q_{s,j}^{\left( u \right)}\otimes q_{c,j}^{\left( u \right)}} \right)\cdot X=\sum\limits_{u=1}^{E}{\left( q_{y,j}^{\left( u \right)}\otimes q_{s,j}^{\left( u \right)}\otimes q_{c,j}^{\left( u \right)}\cdot X \right)} \\ & =\sum\limits_{u=1}^{E}{\left( q_{y,j}^{\left( u \right)}\otimes q_{s,j}^{\left( u \right)}\otimes q_{c,j}^{\left( u \right)}\cdot {{x}_{y}}\otimes {{x}_{s}}\otimes {{x}_{c}} \right)} \\ & =\left( \sum\limits_{u=1}^{E}{q_{y}^{\left( u \right)}\cdot {{x}_{y}}} \right)\otimes \left( \sum\limits_{u=1}^{E}{q_{s}^{\left( u \right)}\cdot {{x}_{s}}} \right)\otimes \left( \sum\limits_{u=1}^{E}{q_{c}^{\left( u \right)}\cdot {{x}_{c}}} \right) \\\end{align}$       (19)

Let Q_v ∈E^fG×1, the bias is represented by n_v, the affection prediction result is represented by $\hat{t}_v$, at last, the fully-connected layer was used to perform multi-modal affection prediction, and the output result can be expressed as:

${{\hat{t}}_{v}}=Q_{v}^{Y}G+{{n}_{v}}$     (20)

The loss function of modal sharing tasks can be expressed as:

$Los{{s}_{CO}}=\frac{1}{B}\sum\limits_{u=1}^{B}{{{\left( {{{\hat{t}}}_{u}}-{{t}_{u}} \right)}^{2}}}$     (21)

Assuming: β represents a hyperparameter, then the overall loss function of the MAC model based on combined processing of texts and images can be expressed as:

$Los{{s}_{MA}}=Los{{s}_{CO}}+\beta Los{{s}_{UN}}$      (22)

This study explored the application of MAC in teaching evaluation combining text and image processing. At first, the input texts were divided into two parts: main body and the hash tag, which were subjected to feature extraction respectively. The image features were extracted from two angles: object and scene, since the two angles can give image information of different levels. The MAC model was divided into modal sharing tasks and modal private tasks to attain better adaptability in case of new teaching evaluation scenarios. In the experimental results, the changes in affection classification accuracy corresponding to different numbers of key areas were given, the performance of four models was analyzed based on four indicators, Precision, Recall, F1 and Accuracy, and the results verified the validity of the proposed method in teaching evaluation. Through comprehensive analysis of text and image information, the model can more accurately classify affections. The experimental results under the conditions of 50%, 75% and 100% data volume were given, which again verified the advantages of the proposed MAC method combining text and image process in teaching evaluation. At last, the experimental results of different models under different modes on MAC datasets were given, and the reasons why the proposed model has exhibited high validity on the datasets were summarized.