Multimodal Image Processing and Fine-Grained Learning Behavior Recognition in Higher Education Smart Classrooms

The digital transformation of higher education has driven the evolution of smart classrooms toward greater precision and personalization. Fine-grained learning behavior recognition [1, 2], as a core technology for perceiving classroom dynamics and optimizing instructional processes, has been regarded as a key determinant of the intelligence level of smart classrooms [3]. Fine-grained learning behaviors encompass a wide range of specific in-class actions, which serve as critical indicators for analyzing learners’ attention and engagement. Consequently, substantial significance has been attributed to such behaviors for enabling personalized instructional guidance and improving overall teaching quality. Multimodal imaging has been increasingly recognized as an effective approach for fine-grained learning behavior recognition [4, 5]. Red, Green, and Blue (RGB) images [6] are utilized to capture appearance and texture information, depth images [7] are employed to characterize spatial structural features, and thermal images [8] are leveraged to reflect physiological states and attention distribution. Through the complementary integration of these modalities, the inherent limitations of single-modality data can be effectively overcome, thereby enhancing the comprehensiveness and robustness of behavior recognition.

Image processing techniques [9, 10], serving as the foundation of multimodal behavior recognition, play a pivotal role in smart classroom environments, as their performance directly influences the accuracy of subsequent feature fusion and behavior recognition. However, the complexity of smart classroom scenarios introduces several unique challenges to multimodal image processing and fine-grained learning behavior recognition. Variations in classroom lighting conditions, particularly under low-light environments, tend to result in increased image noise and blurred details. In addition, diverse sitting postures, frequent occlusions among learners, and sensor-induced noise further degrade image quality. Due to the heterogeneous nature of multimodal data acquired from different sensors, spatial misalignment across modalities is commonly observed. Meanwhile, the subtle differences among fine-grained learning behaviors make accurate discrimination difficult for conventional methods, while limited model interpretability hinders the identification of critical factors underlying behavior recognition. These challenges collectively constrain the advancement of intelligent smart classrooms and limit the in-depth application of multimodal image processing technologies in the educational domain. Therefore, the investigation of multimodal image processing and fine-grained learning behavior recognition methods tailored for smart classrooms in higher education is of substantial theoretical importance and practical relevance.

Multimodal image enhancement serves as a fundamental step for improving the performance of behavior recognition. Existing approaches can generally be categorized into traditional methods and deep learning-based methods. Among traditional approaches, the Retinex algorithm [11, 12] enhances images by decomposing illumination and reflectance components; however, under low-light classroom conditions, a balance between noise suppression and edge preservation is difficult to achieve. The Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm [13, 14] is capable of improving image contrast, yet it is prone to local overexposure and lacks adaptability to the heterogeneous characteristics of multimodal images, thereby limiting its ability to simultaneously optimize RGB, depth, and thermal images. Cross-modal feature fusion [15-17] constitutes a critical stage for integrating multimodal information, with existing strategies broadly classified into early fusion, intermediate fusion, and late fusion. Early fusion methods, such as simple concatenation, fail to account for the heterogeneity of multimodal features, often resulting in feature redundancy and mutual interference. Intermediate fusion approaches driven by conventional attention mechanisms are capable of emphasizing salient features; however, spatial misalignment across modalities is not effectively addressed, and the complementary relationships among multimodal features are insufficiently exploited, thereby limiting the realization of synergistic advantages.

In the domain of fine-grained learning behavior recognition, most existing methods rely on single-modality images or skeletal keypoint representations [18, 19], while neglecting the integration of complementary multimodal information such as texture, depth, and thermal characteristics. Although certain studies have incorporated multimodal data, the temporal dynamics of behaviors have not been comprehensively captured, making it difficult to distinguish highly similar fine-grained actions. Furthermore, the model decision-making process lacks interpretability [20], thereby failing to satisfy the dual requirements of recognition accuracy and interpretability in smart classrooms. Overall, substantial limitations remain in current research, and a comprehensive solution tailored to the complex scenarios of smart classrooms has yet to be established, which constitutes the primary motivation for this study.

A multimodal image processing and fine-grained learning behavior recognition framework tailored for smart classrooms in higher education is developed to address the critical technical bottlenecks in multimodal image enhancement, cross-modal alignment and fusion, and fine-grained behavior recognition. Through this framework, multimodal image quality is optimized, heterogeneous features are efficiently integrated, and fine-grained learning behaviors are accurately recognized, while the requirements for real-time performance and model interpretability in classroom environments are simultaneously satisfied. This framework is expected to provide robust technical support for the intelligent advancement of smart classrooms. The main contributions are summarized below. A unified multimodal image preprocessing and enhancement module is designed to accommodate the complexity of classroom environments. By addressing the inherent limitations of RGB, depth, and thermal image modalities, synchronized optimization and spatial alignment of multimodal images are achieved, thereby establishing a reliable foundation for subsequent cross-modal feature fusion and behavior recognition. A deformable convolution and cross-attention alignment fusion network is constructed. Precise spatial alignment of multimodal features is accomplished through deformable convolution, while complementary relationships among features are effectively exploited via a cross-attention mechanism. A gated adaptive fusion strategy is further introduced to achieve unified feature representation, thereby resolving challenges associated with cross-modal heterogeneous feature alignment and fusion. A multi-scale spatiotemporal graph convolutional network is developed to integrate multimodal features with dynamic spatiotemporal graph information. Behavioral patterns at different temporal scales are captured through multi-scale spatiotemporal convolutions, while critical spatial regions and temporal segments are emphasized using a spatiotemporal attention mechanism. As a result, precise fine-grained learning behavior recognition is achieved, and model interpretability is enhanced. An end-to-end multi-task learning strategy combined with a multi-objective loss function is proposed to jointly optimize the three sub-tasks of image enhancement, feature fusion, and behavior recognition. The issue of class imbalance is alleviated through tailored loss functions, leading to improved overall model performance.

The remainder of this study is organized below. In Section 2, the tasks of multimodal image processing and fine-grained learning behavior recognition in smart classrooms are formally defined, and the core technical challenges are mathematically formulated along with their associated constraints. In Section 3, the proposed framework is described in detail, with particular emphasis on the design principles and technical implementation of each core module, constituting the central part of the study. In Section 4, the effectiveness of the proposed method is quantitatively validated through comparative experiments, ablation studies, visualization analyses, and robustness evaluations. In Section 5, the principal advantages and limitations of the proposed approach are critically discussed, and future research directions are outlined in light of emerging trends in the field. Finally, the conclusions are presented in Section 6, where the key findings and academic contributions are summarized.

3.1 Overall framework overview

A multimodal image processing and fine-grained learning behavior recognition framework for smart classrooms in higher education is developed based on a hierarchically progressive three-layer architecture. These layers are designed to operate in a coordinated and interdependent manner, forming a closed logical loop that closely aligns with the overall pipeline, thereby enabling end-to-end processing from multimodal image input to fine-grained behavior recognition output. The multimodal image preprocessing and enhancement layer serves as the foundational input stage of the framework. RGB, depth, and thermal image sequences are received as inputs, and modality-specific deficiencies are systematically addressed through targeted optimization and spatial alignment. As a result, high-quality and spatially synchronized multimodal image sequences are generated, providing reliable data support for subsequent feature processing. The cross-modal feature alignment and fusion layer operates on the enhanced multimodal images. Through a dedicated network architecture, heterogeneous features are accurately aligned in the spatial domain and efficiently fused. The complementary information across modalities is effectively exploited, enabling the generation of unified cross-modal feature representations that simultaneously encode appearance texture, spatial structure, and physiological state information. The fine-grained learning behavior recognition layer functions as the core output stage of the framework. Spatiotemporal features are extracted from the fused representations, and critical information is selectively emphasized to achieve precise classification of fine-grained learning behaviors and interpretable decision-making. Ultimately, behavior category labels for the corresponding image sequences are output. Through the integration of an end-to-end multi-task learning strategy, the three-layer architecture is jointly optimized, effectively linking the processes of image enhancement, feature fusion, and behavior recognition. As a result, a balanced performance in terms of accuracy, real-time capability, and interpretability is achieved, ensuring adaptability to the complex and dynamic conditions of smart classroom environments.

3.2 Unified multimodal image preprocessing and enhancement module

The unified multimodal image preprocessing and enhancement module is designed to address typical challenges encountered in smart classroom environments, including low-light degradation in RGB images, occlusion-induced missing regions in depth images, and insufficient contrast in thermal image temperature distributions. An integrated framework combining modality-specific fine-grained optimization with global spatial calibration is constructed. While modality-dependent defects are selectively corrected, precise spatial alignment across modalities is simultaneously achieved, thereby providing high-quality data for subsequent cross-modal feature alignment and fusion. The overall workflow is illustrated in Figure 1. To mitigate low-light degradation in RGB images, a Retinex-based decomposition enhancement method incorporating multi-scale weighted guided filtering is developed. According to the illumination imaging model, the original image is decomposed into reflectance and illumination components, which can be expressed as:

$I(x, y)=R(x, y) \cdot L(x, y)$           (1)

where, I(x,y) denotes the original low-light RGB image, R(x,y) represents the reflectance component encoding intrinsic texture information, and L(x,y) corresponds to the illumination component. Unlike conventional Retinex approaches based on fixed-scale filtering, large-scale guided filtering is utilized to estimate the global illumination distribution, and small-scale guided filtering is employed to capture local illumination details. Adaptive weighting is further introduced based on local pixel neighborhood characteristics, with the weighting function defined as:

$\omega(x, y)=\lambda_1 \cdot|\nabla G(x, y)|+\lambda_2 \cdot T(x, y)$           (2)

where, $|\nabla G(x, y)|$ denotes the gradient magnitude within the pixel neighborhood, T(x,y) represents texture complexity, and λ₁ and λ₂ are normalized adaptive coefficients. Regions with pronounced edges and rich textures are assigned higher weights. Finally, the enhanced image is reconstructed using the optimized reflectance component, enabling effective noise suppression while preserving critical details such as human posture and object boundaries.

A multi-stage joint restoration and enhancement strategy is adopted for depth images. Sensor noise is first suppressed using bilateral filtering while spatial structural edges are preserved. Subsequently, small-scale holes are filled through morphological closing operations. Finally, large-area depth missing regions caused by occlusion or sensor failure are restored using a region-filling algorithm, resulting in depth feature maps with spatial continuity and structural completeness. The optimization of thermal images is achieved through the combination of adaptive histogram equalization and pseudo-color mapping. The dynamic range of the temperature distribution is first expanded using adaptive histogram equalization, thereby enhancing temperature differences in critical regions such as the face and hands. Pseudo-color mapping is then applied to transform grayscale temperature information into high-discriminability color features, while the original temperature values are retained for subsequent attention-based feature extraction.

After modality-specific optimization has been completed, spatial alignment across modalities is achieved through scale normalization and pixel coordinate calibration. RGB, depth, and thermal images are unified into a common spatial resolution and pixel coordinate system, thereby eliminating spatial misalignment caused by sensor placement disparities and temporal acquisition offsets. As a result, precise pixel-level correspondence among the three modalities is ensured, effectively removing spatial discrepancies for subsequent cross-modal feature fusion.

Figure 1. Workflow of the unified multimodal image preprocessing and enhancement module

3.3 Deformable convolution and cross-attention alignment fusion network

To address the challenges of low alignment accuracy and insufficient exploitation of complementary information caused by multimodal feature heterogeneity, a deformable convolution and cross-attention alignment fusion network is developed. RGB features are adopted as the reference baseline, and non-rigid spatial alignment is achieved through deformable convolution. In parallel, a cross-attention mechanism is employed to capture complementary information across modalities. Finally, a gated adaptive fusion strategy is applied to generate unified cross-modal feature representations, thereby providing high-quality feature support for subsequent fine-grained learning behavior recognition. The detailed architecture is illustrated in Figure 2. The proposed application of deformable convolution extends beyond its conventional use in object detection and is adapted for cross-modal feature alignment. Non-rigid matching is achieved by learning pixel-level two-dimensional spatial offsets. The offset learning process is formulated as:

$\Delta p_{i j}=W_{\text {offset }} \cdot F_{c a t}\left(F_{r e f}, F_{s r c}\right)$               (3)

where, Δp_ij denotes the two-dimensional spatial offset of the j-th sampling point in the i-th convolution kernel, W_offset represents the learnable offset weights, F_ref denotes the RGB reference feature map, and F_src represents the source feature map to be aligned (i.e., depth or thermal features). The operator F_cat denotes channel-wise feature concatenation. The offset is adaptively learned by jointly considering the local correlations between source and reference features as well as image gradient information. This enables the sampling points of the features to be aligned to undergo non-rigid deformation in response to variations in human poses and object boundaries within classroom scenes. Consequently, cross-modal spatial misalignment caused by sensor disparity and student movement is effectively eliminated, achieving pixel-level precise alignment.

Figure 2. Architecture of the deformable convolution and cross-attention alignment fusion network (DCA-Net)

The aligned multimodal features are further processed through a parallel cross-attention module to exploit complementary information across modalities. This module is designed with two parallel attention branches, enabling feature interaction and weight modulation between depth–RGB and thermal–RGB pairs, respectively. In contrast to conventional single-attention mechanisms, this design allows modality-specific salient information to be selectively emphasized. The attention weight maps are generated based on pairwise feature similarity, which can be formulated as:

$A_k(x, y)=\operatorname{softmax}\left(\frac{F_{R G B}(x, y) \cdot F_k(x, y)^T}{\sqrt{d}}\right)$                (4)

where, k∈{D,T}, with D and T representing the depth and thermal modalities, A_k(x,y) represents the corresponding attention weight map, and d denotes the feature channel dimension. The softmax function is applied to normalize the weights. In the depth-RGB cross-attention branch, feature similarity between depth and RGB representations is computed to generate attention maps that focus on regions with significant depth variations, thereby enhancing features with weak RGB texture but strong spatial structural cues. In the thermal-RGB branch, attention is directed toward regions associated with physiological characteristics, such as facial temperature variations, resulting in corresponding attention maps. After L2 normalization, the two attention maps are used to modulate the RGB features, yielding depth-modulated and thermal-modulated feature representations. Through this process, complementary information across modalities is effectively extracted.

The gated adaptive fusion unit is responsible for integrating multimodal features and overcoming the limitations of conventional fixed-weight fusion strategies. A learnable weight vector is introduced to dynamically balance the contributions of the three modalities. The fusion process is defined as:

$F_{\text {fusion }}=\alpha \cdot F_{R G B}+\beta \cdot F_{D-\text { mod }}+\gamma \cdot F_{T-\text { mod }}$               (5)

where, F_fusion denotes the final cross-modal fused feature map, F_D-mod and F_T-mod represent the depth-modulated and thermal-modulated features, respectively, and α, β, and γ are learnable fusion weights. These weights are adaptively optimized through a sigmoid activation function and constrained such that α + β + γ = 1. This design enables flexible allocation of modality contributions according to dynamic classroom conditions and feature characteristics. As a result, the fused representation effectively preserves the texture details of RGB images, while incorporating the spatial structural information from depth data and the physiological cues from thermal imaging. Consequently, a comprehensive and discriminative feature representation is obtained, providing robust support for subsequent fine-grained learning behavior recognition.

3.4 Multi-scale spatiotemporal graph convolutional network

To overcome the limitations of conventional behavior recognition methods that rely solely on skeletal keypoints, and to enable fine-grained learning behavior recognition through multimodal information integration, a multi-scale spatiotemporal graph convolutional network is constructed. Through the coordinated integration of multimodal keypoint feature encoding, dynamic spatiotemporal graph construction, multi-scale spatiotemporal convolution, and a spatiotemporal attention mechanism, both spatial structural characteristics and temporal dynamics of behaviors are effectively captured, while model interpretability is simultaneously enhanced. The overall framework is illustrated in Figure 3. A key innovation in the keypoint extraction and feature encoding stage lies in the integration of multimodal information rather than reliance solely on keypoint coordinates. Specifically, a lightweight high-resolution network architecture, combined with depth information, is employed to extract 17 upper-body keypoints of students from the cross-modal fused feature maps. Thermal imaging information is further utilized to refine head-related key regions at the sub-pixel level, thereby ensuring high localization accuracy. Local region feature vectors for each keypoint are subsequently extracted using deformable RoIAlign operations, enabling the encoding of texture, depth, and thermal information. The encoding process is formulated as:

$f_i=\operatorname{RoIAlign}\left(F_{\text {fusion}}, p_i\right) \cdot W_{\text {enc}}+b_{\text {enc}}$                (6)

where, f_i denotes the encoded feature vector of the i-th keypoint, F_fusion represents the cross-modal fused feature map, p_i denotes the keypoint coordinates, and W_enc and b_enc correspond to the weights and bias of the encoding layer. This design significantly enhances the discriminative capacity of keypoint features and provides high-quality node representations for subsequent spatiotemporal graph construction.

Figure 3. Topology and feature extraction graph of the multi-scale spatiotemporal graph convolutional network (MST-GCN)

The dynamic spatiotemporal graph construction further extends beyond conventional fixed-topology approaches by incorporating both spatial and temporal information for adaptive topology optimization. The graph is decomposed into a spatial graph and a temporal graph. In the spatial graph, encoded keypoints are treated as nodes, and edge weights are determined by jointly considering Euclidean distance and feature similarity, thereby capturing both spatial proximity and feature consistency. The weight is defined as:

$w_{i j}=\alpha \cdot \exp \left(-\frac{\left\|p_i-p_j\right\|^2}{\sigma^2}\right)+(1-\alpha) \cdot \frac{f_i \cdot f_j^T}{\left\|f_i\right\| f_j \|}$               (7)

where, w_ij denotes the spatial edge weight between node i and node j; α is a balancing coefficient; and σ represents the distance decay parameter. The edge weights are further adaptively adjusted through an attention mechanism, enabling optimization of the spatial graph topology. In the temporal graph, edges are established between corresponding keypoints across adjacent frames, allowing the propagation of temporal dynamic information and facilitating the modeling of continuous behavioral variations. Multi-scale spatiotemporal graph convolution constitutes one of the core innovations of the network. In the spatial domain, graph convolution is employed to aggregate features from neighboring nodes. In the temporal domain, one-dimensional convolutional kernels with dilation rates of 1, 2, and 4 are applied in parallel to process node-wise temporal sequences. This design enables the capture of both rapid behaviors (e.g., hand-raising and page-turning) and slower behaviors (e.g., head-down writing and chin-resting). The multi-scale feature fusion process is expressed as:

$F_{S T}=B N\left(\operatorname{ReLU}\left(F_1+F_2+F_4\right)\right)$                (8)

where, F₁, F₂, and F₄ denote the output features corresponding to temporal convolutions with dilation rates of 1, 2, and 4, respectively, and BN represents batch normalization. This design effectively enhances the representational capacity of temporal features, enabling comprehensive modeling of behavior patterns at different temporal scales.

A spatiotemporal attention mechanism is further introduced to improve feature discriminability. Through dynamic weighting, attention is focused on keypoint pairs and temporal segments that are most relevant to the current behavior, while suppressing background noise and irrelevant features. The attention weighting process is defined as:

$F_{\text {att }}=\operatorname{softmax}\left(W_{\text {att }} \cdot F_{S T}\right) \cdot F_{S T}$                 (9)

where, W_att denotes the attention weight matrix, and F_att represents the attention-enhanced spatiotemporal features. The optimized features are subsequently fed into a fully connected classifier, and category probabilities for fine-grained learning behaviors are obtained through a softmax function, thereby completing the recognition process. Through the integration of multimodal feature fusion, adaptive topology optimization, and multi-scale temporal modeling, the proposed multi-scale spatiotemporal graph convolutional network effectively addresses the limitations of conventional methods, including insufficient temporal feature modeling and limited interpretability, enabling accurate fine-grained learning behavior recognition.

3.5 End-to-end multi-task learning and loss function design

To achieve coordinated optimization across the entire pipeline-including image enhancement, feature fusion, and fine-grained behavior recognition-and to improve overall model performance, an end-to-end multi-task learning strategy is adopted. The enhancement module based on multi-scale weighted guided filtering, the fusion module of the deformable convolution and cross-attention alignment fusion network, and the recognition module of the multi-scale spatiotemporal graph convolutional network are integrated into a unified training framework, enabling joint parameter optimization and mutual adaptation among modules. This design effectively avoids feature inconsistency and performance degradation commonly introduced by stage-wise training. Within this training paradigm, a single backpropagation process is employed to simultaneously propagate gradient information from all three sub-tasks to each module. As a result, the output of the enhancement module is adaptively aligned with the input requirements of the fusion module, while the fused features are further optimized to match the input distribution of the recognition module. A closed-loop optimization mechanism is thereby established, ensuring consistency and synergy across the entire processing pipeline.

To address the distinct optimization challenges associated with each sub-task, a multi-objective loss function is designed. Through the joint contribution of individual loss components, improvements in image enhancement quality, cross-modal fusion effectiveness, and fine-grained recognition accuracy are simultaneously achieved. The total loss is formulated as a weighted sum of the losses from each module:

$L_{\text {total }}=\lambda_1 L_{e n h}+\lambda_2 L_{\text {fus }}+\lambda_3 L_{\text {rec }}$                 (10)

where, λ₁, λ₂, and λ₃ denote adaptive weighting coefficients of the modules, and L_enh, L_fus, and L_rec represent the loss functions of the enhancement, fusion, and recognition modules, respectively. The loss function for the enhancement module is composed of a perceptual loss and an edge-preservation loss. The perceptual loss is computed by extracting features using a pre-trained convolutional neural network, enforcing consistency between the enhanced image and a well-illuminated reference image in the feature space. It is defined as:

$L_{\text {percep }}=\left\|\phi\left(I_{\text {enh }}\right)-\phi\left(I_{\text {ref }}\right)\right\|_2^2$              (11)

where, ϕ (·) denotes the feature extraction function, I_enh represents the enhanced image, and I_ref denotes the reference image. The edge-preservation loss is introduced to constrain the gradient information of the enhanced image, thereby preserving edge details. It is defined as:

$L_{\text {edge }}=\left\|\nabla I_{\text {enh }}-\nabla I_{\text {ref }}\right\|_1$              (12)

The weighted sum of the two is obtained as:

$L_{\text {enh }}=\mu L_{\text {percep }}+(1-\mu) L_{\text {edge }}$                 (13)

The fusion module loss consists of a mutual information maximization loss and a feature alignment loss. The mutual information maximization loss is defined as:

$L_{m i}=-M I\left(F_{R G B}, F_D, F_T\right)$             (14)

To maximize the complementary information among multimodal features, the feature alignment loss is defined as:

$L_{\text {align }}=\left\|a v g\left(F_D-F_{R G B}\right)\right\|_2^2+\left\|a v g\left(F_T-F_{R G B}\right)\right\|_2^2$                  (15)

The aligned features are constrained to maintain spatial consistency. For the recognition module, a weighted focal loss is adopted to alleviate the class imbalance problem. The loss function is defined as:

$L_{w-\text { focal }}=-\sum_{c=1}^C w_c y_c \log \left(p_c\right)$           (16)

where, w_c denotes the class weight, y_c is the label indicator variable, and p_c represents the predicted category probability. Higher weights are assigned to minority classes, such as hand-raising behaviors.

To ensure coordinated optimization across the three sub-tasks and to prevent dominance by any single loss component, an adaptive weighting optimization strategy is introduced. The coefficients λ₁, λ₂, and λ₃ are not fixed but are dynamically updated during training according to:

$\lambda_k=\frac{\exp \left(L_k / T\right)}{\sum_{i=1}^3 \exp \left(L_i / T\right)}$             (17)

where, k∈{1,2,3} corresponds to the three modules, and T denotes a temperature parameter that controls the sensitivity of weight updates. Through this design, sub-tasks associated with larger loss values are assigned higher weights, allowing current performance bottlenecks to be prioritized during optimization. As a result, a dynamic balance among the losses of different modules is achieved, ensuring the coordinated progression of the three sub-tasks-image enhancement, feature fusion, and behavior recognition-and ultimately improving the overall performance and generalization capability of the model.

The proposed multimodal image processing and fine-grained learning behavior recognition framework demonstrates substantial advancements across the three core stages-multimodal image enhancement, cross-modal feature fusion, and fine-grained behavior recognition-as evidenced by both quantitative results and visualization analyses. A comprehensive technical framework tailored for smart classrooms in higher education is thus established, exhibiting clear advantages over existing approaches. The multi-scale weighted guided filtering enhancement module extends beyond the limitations of conventional single-modality enhancement and fixed-scale filtering. By explicitly addressing challenges such as low illumination, depth incompleteness, and insufficient contrast in thermal imaging, unified optimization and spatial alignment of RGB, depth, and thermal modalities are achieved. This design effectively balances noise suppression and edge preservation. The deformable convolution and cross-attention alignment fusion network architecture introduces deformable convolution into cross-modal alignment tasks. Through the integration of parallel cross-attention mechanisms and a gated adaptive fusion strategy, limitations associated with conventional fusion methods-such as poor alignment accuracy and feature redundancy-are effectively mitigated. The multi-scale spatiotemporal graph convolutional network integrates multimodal keypoint features and employs dynamic spatiotemporal graph construction alongside multi-scale temporal convolution. This design effectively enhances the accuracy and interpretability of fine-grained behavior recognition. In addition, the end-to-end multi-task learning strategy combined with an adaptive multi-objective loss function enables coordinated optimization across the entire processing pipeline. The coordinated interaction among all modules forms a closed-loop optimization framework. Overall, the proposed framework is not only well-suited for smart classroom environments but also provides a generalizable paradigm for multimodal image processing and fine-grained behavior recognition in other complex scenarios. The methodological contributions are therefore of considerable academic significance and practical relevance.

Despite the superior overall performance demonstrated in the experiments and the effective adaptation to typical smart classroom scenarios, several limitations remain. In densely populated classroom environments with significant student overlap, inaccuracies in keypoint localization are likely to occur. Consequently, cross-modal feature alignment and fusion may be adversely affected, leading to reduced discrimination accuracy for similar fine-grained behaviors. This limitation is primarily attributed to insufficient feature modeling in overlapping regions. In addition, the temperature calibration accuracy of thermal images is constrained by sensor performance and environmental temperature variations. Although temperature differences are enhanced in the current framework, environmental fluctuations are not explicitly modeled, which may compromise the accuracy of thermal information and subsequently affect recognition accuracy. Furthermore, the degree of model lightweighting remains limited. While real-time requirements are satisfied, computational cost and memory consumption remain relatively high, posing challenges for efficient deployment on resource-constrained platforms such as embedded edge devices. Finally, although the constructed SmartClass-MM dataset reflects realistic classroom conditions, its scale and diversity remain limited, which may restrict generalization across different classroom types and student populations.

In light of these limitations and emerging trends in image processing and intelligent education, future work is expected to be pursued along three primary directions: feature modeling, data optimization, and model deployment. To address the challenge of overlapping student scenarios, Transformer-based architectures will be incorporated to enhance global modeling of cross-modal features. In addition, attention mechanisms specifically designed for overlapping regions will be developed, and keypoint extraction algorithms will be refined to enable accurate localization and separation of overlapping keypoints. To improve the accuracy of thermal information, adaptive temperature calibration methods based on environmental conditions will be investigated, and deep learning–based strategies will be employed to achieve dynamic compensation of temperature errors. To enhance model generalization, the dataset will be expanded in both scale and diversity, and advanced data augmentation techniques will be introduced. Model lightweighting will be further explored through techniques such as model quantization, pruning, and knowledge distillation, enabling compatibility with embedded edge devices. In addition, the integration of federated learning with multimodal techniques will be investigated to facilitate multi-scenario collaborative training while preserving student privacy. Finally, deeper integration with educational management systems will be pursued, allowing behavior recognition outputs to be directly linked with instructional optimization strategies, thereby promoting the development of precise and personalized smart classrooms in higher education.