Personalized Learning Pathway Generation for Online Education Through Image Recognition

Within the context of online education, the implementation of group behavior recognition, predicated on image processing, is instrumental in the construction of mechanisms for generating personalized learning paths. This recognition mechanism provides an intuitive and dynamic way to understand learners' learning attitudes, engagement, and responses by capturing and analyzing their visual behavior data, enabling the teaching system to more accurately adjust teaching content and pace to meet the specific needs of different learners. For example, by monitoring the attention distribution of learners, the system can determine whether they understand the current teaching content or feel confused and uneasy.

A method is proposed, amalgamating transfer learning with dual-stream convolutional networks, for the recognition of group behavior among learners in vocational college online education. Figure 1 depicts the structure of a conventional dual-stream convolutional network. Transfer learning can utilize existing rich data resources to quickly adjust model parameters to adapt to the behavioral characteristics of specific groups of learners. In addition, dual-stream convolutional networks can more comprehensively capture and analyze the dynamic behavioral changes of learners during the learning process by simultaneously processing spatial features and time series information. Integrating the fast adaptation characteristics of transfer learning with the deep information processing capabilities of dual stream convolutional networks, this image recognition driven behavior analysis not only improves the generation efficiency of personalized learning paths, but also provides a more delicate and personalized teaching strategy for online education in vocational colleges, helping to achieve a truly learner centered teaching model.

Figure 1 illustrates the structure of a traditional dual-stream convolutional network. In the task of identifying group behavior of learners in online education, the spatial and temporal flow networks of the dual stream convolutional network are configured to jointly handle the static features and dynamic changes of video frames. The spatial network is mainly responsible for parsing the image content of each individual video frame, capturing the facial expressions, postures, and other spatial features related to learning behavior of learners. Usually, this stream adopts a standard convolutional neural network (CNN) architecture, which has been proven to have strong feature extraction capabilities in various image recognition tasks. Time networks focus on sequence information, processing the flow of actions and behaviors between consecutive frames, and mining the behavior patterns of learners in the temporal dimension, such as gesture actions, changes in gaze points of the eyes, etc. This flow usually uses optical flow method to pre calculate and input motion information between frames, and encodes these dynamic features through CNN architecture. Figure 2 presents the framework of the recognition model, grounded in transfer learning and dual-stream convolutional networks, as conceptualized in this study.

The spatial network's output is manifested as a high-dimensional feature vector, encapsulating the spatial information of the current frame. This serves as the foundational basis for subsequent behavior recognition and analysis. Inputs to the spatial stream network predominantly comprise raw video frame imagery, capturing static visual characteristics of learners at discrete moments, such as posture, facial expressions, and interactive actions. In contrast, the temporal network yields an alternative set of feature vectors, delineating the dynamic alterations in learners' behaviors over time, exemplified by the fluidity of gestures and the velocity of head rotations. The temporal stream network inputs a succession of consecutive video frames comprising optical flow images, specifically engineered to calculate and depict the motion information transpiring between frames, thereby reflecting changes in object and surface movements.

The optical flow images for the temporal stream network are customarily derived from processing red, green and blue (RGB) images of learners' group behavior in online education settings, utilizing optical flow methodologies. Optical flow, a method for estimating pixel motion across consecutive video frames, operates on the premise that pixel intensity remains unaltered over brief periods, with only minor spatial shifts occurring in corresponding pixel locations. Within the scope of analyzing group behavior in online education, optical flow initially acquires continuous RGB frame sequences and subsequently computes a motion vector for each pixel from one frame to the next. This vector is composed of both horizontal and vertical displacement components. The foundational assumption of brightness constancy in optical flow facilitates the tracking of learners’ movements, such as head rotations and gestural alterations. The direction and velocity of these movements are estimated by examining the variations across the image sequence. Assuming the velocity vector of a pixel is denoted by n= (i, n), the brightness constancy assumption is articulated as follows:

$U(a, b, s)=U(a+f a, b+f b, s+f s)$     (1)

Expanding this using Taylor's series results in:

$U(a, b, s)=U(a, b, s)+\frac{\partial U}{\partial a} f a+\frac{\partial U}{\partial b} f b+\frac{\partial U}{\partial s} f s+\gamma$    (2)

Defining i=fa/fs, n=fn/fs, U_a=∂U/∂a, U_b=∂U/∂b, and U_c=∂U/∂c, the combination of the preceding expressions yields the following relationship:

$U_a i+U_b i+U_s=0$     (3)

In the context of a single optical flow image, the information pertaining to pixel motion is confined to that between two consecutive frames. This limitation might not suffice to capture complex or gradual motion patterns, a scenario often encountered in group behavior recognition tasks that necessitate analysis over extended time sequences. To address this challenge, a trajectory tracking method is employed, processing the input from a singular optical flow image. This technique is instrumental in accumulating and amalgamating motion information across a series of consecutive frames. The process involves tracking the movement paths of designated feature points or pixel blocks over multiple frames, thereby constructing trajectories that encapsulate motion changes over prolonged durations. The input vector is denoted by U_s, and the following procedure is applied to each frame s:

$\begin{aligned} U_n(i, n, 2 j-1) & =f_{n+j-1}^a\left(O_j\right) ; U_n(i, n, 2 j)=f_{\pi+j-1}^b\left(O_j\right) \\ i & =[1, q], n=[1, g], j=[1, M]\end{aligned}$  (4)

O_j denotes the j-th layer along the trajectory. In the frames capturing learner group behaviors, it is posited that a recursive relationship exists, commencing from (i, n):

$O_1=(i, n) ; O_j=O_{j-1}+f_{n+j-2}\left(O_{j-1}\right), j>1$     (5)

To harness the knowledge embedded in models pre-trained on extensive datasets, thus expediting and refining the learning process for learner behavior recognition in specific online education scenarios, this study integrates transfer learning into the dual-stream convolutional network designed for vocational college learner group behavior recognition. Transfer learning facilitates the transference of feature representations, honed in the domain of image recognition, to the realm of learner behavior recognition within online education. This approach serves a dual purpose: for behavior recognition tasks in online education, where data is often scarce, pre-trained models offer a substantial foundation of knowledge, diminishing the dependency on extensive annotated data needed for training models from the ground up. Additionally, it endows the model with robust feature extraction capabilities from the onset of learning, enabling quicker adaptation and refinement in recognizing learner behavior features during subsequent training phases. Simultaneously, the study incorporates a formalized approach to maximizing average differences using multiple kernels. This strategy aims to augment the efficacy of model testing, essentially enhancing the model's capacity to discern true effects, while concurrently minimizing Type II errors, which entail the risk of erroneously accepting a spurious hypothesis or failing to reject an invalid null hypothesis.

1.png

Figure 1. Structure of the traditional dual-stream convolutional network

2.png

Figure 2. Framework of the proposed recognition model based on transfer learning and dual-stream convolutional networks

In unsupervised domain adaptation, the source domain is defined as F_t={(a^t_u,b^t_u)}^vt_u=1, and v_t is marked as an example. The target domain is defined as F_u={a^s_k}^vs_k=1, where v_s is an unmarked example. The probability distributions of source and target domains are represented by o and w, respectively. By maximizing the testing power, the multi-kernel maximization of average differences aids in accurately capturing the real behavioral pattern changes in online education scenarios. By minimizing Type II, it ensures that the model does not easily miss important behavioral signals when facing unseen behavioral samples. The goal is to build a deep neural network that can learn transferable features across domain differences, and to construct a classifier b=ϕ(a) to minimize the target risk R(ϕ)= Oe_(a,b)-w[ϕ(a≠b)]. In the context of semi-supervised adaptation, where the target domain contains a limited number of labeled samples, annotated examples v_x for both the source and target domains are represented as F_x={(a^x_i, b^x_u)}.

It is hypothesized that the reproducing kernel Hilbert space, equipped with feature kernel j, is represented by G_j. The average embedding of distribution o in G_j, identified as the unique element ω_j(o), ensures that for all d∈G_j, the relation R_a~Ed(a)=<(d(a),ω_j(o)>G_j is upheld. The distance between the average embeddings of distributions o and w in the reproducing kernel Hilbert space, termed as the multi-kernel maximization of average differences LJ-LLF f_j(o,w), is defined. The squared formula for this multi-kernel maximization of average differences is articulated as:

$f_j^2(o, w) \stackrel{\Delta}{=}\left\|R_o\left[\theta\left(a^s\right)\right]-R_o\left[\theta\left(a^s\right)\right] \mid\right\|_{G_j}^2$     (6)

The model developed in this research is predicated on transfer learning and dual-stream convolutional networks, drawing upon the VGGNet architecture. Its network depth is adept at extracting intricate feature representations, essential for comprehending learners' behaviors and emotional states. The network comprises 13 convolutional layers and 5 maximum pooling layers, designed to capture features across varying scales and to abstract higher-level semantic information progressively. Three fully connected layers integrate these features, and a preceding softmax layer processes them for probability output before classification. In the realm of transfer learning, adaptation layers (14 to 16) are appended to the VGGNet foundation, facilitating the fine-tuning of the pre-trained model's weights for enhanced performance on novel datasets. Additionally, by merging features from both the spatial and temporal streams before the softmax layer, the model harnesses static information (e.g., posture, expressions) and dynamic data (e.g., gestures, motion trajectories) from learners, enabling more precise behavior predictions.

To harmonize the distributions of source and target domains under the latent representation of the fully connected layers (14^th to 16^th), the CNN risk incorporates a multi-layer adaptation regularizer grounded in the multi-kernel maximization of average differences. It is posited that the network's final three layers are denoted as m₀, m₁, and m₂. The aggregation of all labeled data from both source and target domains is represented by a^x_u and v_x. Consequently, the network's optimization objective is formulated as follows:

$\underset{\Phi}{M N} \frac{1}{v} \sum_{n=1}^{v_p} K\left(\phi\left(a_n^x\right), b_n^x\right)+\eta \sum_{m=U_0}^{U_2} f_j^2\left(F_t^s, F_s^s\right)$     (7)

The essence of the multi-kernel maximization of average differences is founded on employing a suite of kernel functions to translocate data into a high-dimensional feature space. Within this space, the disparities in distributions between source and target domains are quantified. The process of computing the inner products of these kernels fundamentally entails evaluating the similarities between data from source and target domains across multiple dimensions. Nonetheless, this approach may be susceptible to biases arising from unbalanced sample sizes, potentially leading to skewed estimations. An unbiased estimation technique is utilized to mitigate this concern. By making up for any potential bias in sample selection, this technique aims to prevent systematic biases and align the predicted expectations with the actual distribution differences. Using c_u=∆(a^t_2u-1, a^t_2u,a^s_2u-1, a^s_2u)f²_j(o.w)=2/v_t∑^vt/2_u=1h_j(c_u), one may express the unbiased estimation:

$\begin{aligned} & h_j\left(c_u\right) \stackrel{\Delta}{=} j\left(a_{2 u-1}^t, a_{2 u}^t\right)+j\left(a_{2 u-1}^s, a_{2 u}^s\right) -j\left(a_{2 u-1}^t, a_{2 u}^s\right)-j\left(a_{2 u-1}^t, a_{2 u}^s\right)\end{aligned}$             (8)

In the experimental evaluation, the method incorporating transfer learning and dual-stream convolutional networks, specifically designed for the recognition of group behaviors in vocational college learners, demonstrated superior performance. The comparative analysis of recognition rates, presented in Table 1, reveals that this method achieved remarkable success across various group behavior recognition tasks. Notably, in the task of synchronous interaction, the proposed method significantly excelled, outstripping competitors such as Residual Network (ResNet) and EfficientNet, underscoring its proficiency in complex behavior analysis. While the EfficientNet model registered a marginally higher recognition rate (98.9%) in the discussion and cooperation category, the proposed method's performance (95.6%) remained commendably close. Regarding collective emotional response, the method's rate (97.8%) was on par with ResNet, though the implausibly high rate (101%) of EfficientNet suggests potential methodological inaccuracies. For categories such as participation changes and discipline and order, data for EfficientNet and GoogLeNet were not available, indicating either non-assessment in these areas or subpar results. On average, the proposed method led with a recognition rate of 96.1%, closely tailed by GoogLeNet's 94.5%. The efficacy of the method proposed in this study, integrating transfer learning with dual-stream convolutional networks, was demonstrated to surpass that of other prevalent CNN models. This superiority is ascribed to the method's capacity to harness pre-trained knowledge comprehensively, thereby augmenting feature extraction capabilities and accurately capturing dynamic behavioral aspects of learners. These results so confirm that the suggested method can be used to identify group behaviors among vocational college learners. This methodology is demonstrated to be a highly effective tool for creating customized learning paths, suggesting that it has broad relevance in the vocational education domain.

Based on True Positive Rates (TPR) and False Positive Rates (FPR) at various thresholds, the created group behavior recognition model's Receiver Operating Characteristic (ROC) curve was plotted to assess the classifier's performance. A ROC curve that is ideal approaches the upper left corner, signifying a high TPR accompanied by a minimal FPR. As can be seen in Figure 5, the curve shows a growth in TPR in tandem with growing FPR, a common feature of ROC curves where classifiers produce more false alarms while stepping up their efforts to identify true positives. The steady rise in TPR in regions with low FPR (from 0 to roughly 0.2) indicates the model's limited ability to correctly detect true positives while preserving a low rate of false positives. With further escalation in FPR, a marked acceleration in TPR growth is observed, particularly notable in the range from 0.2 to 1, where TPR elevates from 0.04 to 1. This trend indicates the model's proficiency in identifying true positives, albeit accompanied by an increase in false alarms. Reaching a TPR of 1 at an FPR of 1 implies that the classifier ultimately detects all true positives, but also misidentifies all negatives as positives. The ROC curve, thus, underscores the utility of the group behavior recognition model, especially in scenarios where a higher tolerance for false alarms exists.

In the context of model training, Figure 6 elucidates the accuracy trajectory of the group behavior recognition model across varying epochs. The initial training phase (0-20 epochs) witnessed a modest rise in accuracy from 0.315 to 0.375, indicative of the model's initial phase of feature learning for group behavior recognition. Accuracy increased steadily during the course of training (20-60 epochs), peaking at 0.645 by the 60^th epoch, indicating the model's continuous adaptability and learning improvement. Specifically, accuracy increased significantly between 60 and 100 epochs, reaching a maximum of 0.915, indicating a notable improvement in the model's capacity to identify group behaviors. This steady increase in accuracy during the training period demonstrates how well the model learns the characteristics of group behavior. The model achieves excellent accuracy, as seen by the highest accuracy of 0.915, and the overall trend shows an upward pattern in accuracy. Despite fluctuations in later training stages, the model's performance distinctly surpassed random guessing, emphasizing the robustness and effectiveness of the proposed method in recognizing group behaviors.

Table 1. Comparative recognition rates of different group behavior recognition models

Table 2. Impact of threshold values on recognition accuracy for 5-way-1-shot learning strategy

Table 3. Comparison of F1-score and recognition accuracy with several micro-expression recognition models

5.png

Figure 5. ROC curve of the constructed group behavior recognition model

6.png

Figure 6. Accuracy curve of the constructed group behavior recognition model

The investigation of learner micro-expression recognition utilizing GCNs was conducted to more precisely discern the nuances of learner emotional changes. Table 2 delineates the model's recognition accuracies on both training and validation sets under varying threshold settings, employing a 5-way-1-shot learning strategy. Recognition accuracies on the training set were observed at 0.562, 0.578, and 0.559 for threshold values of 0.05, 0.14, and 0.21, respectively, while accuracies on the validation set were 0.415, 0.421, and 0.418. These relatively modest accuracies, particularly on the validation set, indicate constrained generalization capabilities. Enhancement in accuracies was noted when threshold values were increased to 1.1, 1.3, and 2.1, achieving 0.748, 0.789, and 0.745 on the training set, and 0.725, 0.745, and 0.736 on the validation set, respectively. This improvement suggests an upward trend in model performance with elevated threshold values. The threshold setting of 1.3 emerged as the most effective, yielding the highest accuracy on both the training (0.789) and validation (0.745) sets, thereby indicating its optimality for training and generalization. The findings imply that threshold values significantly influence the performance of micro-expression recognition models, with higher settings enhancing accuracy in both training and validation contexts. This underscores the efficacy of the proposed methodology and the critical role of appropriate threshold settings in bolstering model generalization, particularly emphasizing the GCNs' capability in micro-expression capture.

Figure 7 showcases the learner micro-expression recognition model's loss curves on both training and validation sets across various training epochs. Initially, an identical loss of 0.518 was exhibited on both sets, reflecting the model's commencement of learning from a state of random weight initialization. In the initial 20 epochs, a marginal decline in loss was observed on the training set, contrasted with a rise followed by a decrease on the validation set, indicating the model's adaptation to the dataset's characteristics. The loss on the training set displayed a gradual yet fluctuating decrease from epochs 20 to 100, with the validation set showing greater fluctuation. Beyond 100 epochs, a stabilization in loss trends on both sets suggested the attainment of optimal learning capacity. Overall, a progressive reduction in loss was noted throughout the training, manifesting effective generalization on the validation set.

The accuracy evolution of the micro-expression recognition model across training epochs, tracked in Figure 8, reflects the model's learning effectiveness over time. Initially, in the first 20 epochs, a rise in accuracy on both training and validation sets was observed, indicating the model's commencement in assimilating useful features from the data. Continued training up to 100 epochs led to a steady increase in accuracy, a trend that persisted without substantial overfitting, as demonstrated by the concurrent rise in accuracy on the validation set. Post 100 epochs, accuracies approached 0.796 on the training set and attained 0.797 on the validation set, highlighting the model's robust generalization ability. The closeness of accuracies on both sets throughout the training process is indicative of sound model generalization, suggesting the model learned generalizable features applicable to the validation set. A sign of a successful model generalization is when the accuracy on the validation set continuously matched that of the training set during the training process. This observation suggests that instead of overfitting on the training data, the model absorbed general features that worked just as well on the validation set. Consequently, the study's learner micro-expression recognition model achieved excellent levels of accuracy on both the training and validation sets and demonstrated a steady rise in accuracy during the training stages. These outcomes support the model's excellent generalization capabilities and demonstrate its ability to distinguish between different micro-expressions.

7.png

Figure 7. Loss change curves of the learner micro-expression recognition model

8.png

Figure 8. Accuracy change curves of the learner micro-expression recognition model

Table 3 contains several micro-expression recognition models as well as two essential metrics: recognition accuracy and F1-score, which are used to assess how well classification models work. While the F1-score, a harmonic mean of precision and recall, shows the comprehensiveness and accuracy of positive class recognition, accuracy represents the percentage of correctly classified samples. With an accuracy of 0.789 on the training set, the suggested methodology performed better in terms of categorization than the other methods. Its F1-score of 0.751 was likewise among the top on the training set. The suggested method had the highest accuracy (0.765) and F1-score (0.744) on the validation set, indicating its strong ability to differentiate between real positive samples while retaining high accuracy and completeness. The suggested technique's exceptional capacity for generalization is demonstrated by its excellent accuracy and F1-score on training and validation sets. This capacity is crucial for micro-expression recognition algorithms to function well on unseen data. To sum up, the approach described in this research is remarkably effective at identifying micro-expressions. It performs exceptionally well on the training set and retains high accuracy and F1-score on the validation set. The approach's great generalization capabilities are indicated by the consistency seen between the outputs obtained during training and validation. Working with complex data types such as micro-expressions emphasizes how crucial it is for the approach to precisely identify and interpret minuscule changes in facial expressions. The results demonstrate the efficacy of the suggested methodology, particularly with regard to managing the multitude of subtleties included in micro-expression data.