Optimizing Multimodal Image Representation Learning with Knowledge Graph Semantic Constraints for Personalized Education

2.1 Overview of overall architecture

To meet the core requirements of personalized education scenarios for semantic enhancement, cognitive adaptation, and interpretability of instructional image representations, this paper proposes an optimized multimodal image representation learning method integrating knowledge graph semantic constraints. Its core objective is to construct image representations capable of accurately capturing the semantics of educational knowledge points and adapting to the cognitive differences of students. The overall framework is illustrated in Figure 1. The framework consists of three collaboratively linked core modules, forming an end-to-end trainable multimodal representation learning system: (1) The Multimodal Feature Extraction and Initial Alignment Module extracts initial features of instructional image patches and relevant textbook text via a ViT and a lightweight text encoder, respectively, achieving preliminary spatial alignment of features from the two modalities. (2) The Knowledge Graph Semantic Constraint Module encodes entities and relations from the educational domain knowledge graph into a differentiable semantic topology. Relying on a sparse graph attention mechanism and relational path contrastive learning, it imposes structured semantic constraints on visual features, strengthening the semantic logic and accuracy of the representations. (3) The Personalized Representation Optimization Module dynamically generates semantic masks based on student cognitive state vectors, adaptively modulating the representation weights of image regions to realize the personalized adaptation of image representations. The following sections will elaborate on the core technical details and mathematical formalizations of each module in turn, clarifying the collaborative working mechanisms and operational principles of each module. The overall architecture is shown in Figure 1.

Figure 1. Overall architecture diagram of multimodal image representation learning integrating knowledge graph semantic constraints

2.2 Multimodal feature extraction and semantic prototype initialization

The core of multimodal feature extraction is to obtain discriminative initial features of images and text and achieve preliminary spatial alignment between the two modalities, laying the foundation for subsequent semantic constraints and personalized optimization. Given an instructional image $I \in R^{H \times W \times 3}$, where H and W are the height and width of the image respectively, and 3 represents the RGB channels, it is divided into N non-overlapping 16 × 16 image patches. The selection of this patch size balances the local details and global correlations of knowledge points in instructional images, effectively preserving key visual information in scenarios such as textbook diagrams and experiment illustrations. Each image patch undergoes linear projection processing to convert features from the pixel space into high-dimensional feature embeddings. The linear projection process can be expressed as:

$v_j=W_{p r o j} \cdot x_j+b_{p r o j}$              (1)

where, $x_j \in \mathrm{R}^{16 \times 16 \times 3}$ is the pixel matrix of the $j$-th image patch, $W_{\text {proj }} \in \mathrm{R}^{D \times 768}$ and $b_{\text {proj }} \in R^D$ are the learnable weight and bias of the linear projection layer respectively, and $D$ is the feature embedding dimension. Finally, the image patch embedding sequence $V=\left[v_1, v_2, \cdots, v_N\right] \in R^{N \times D}$ is obtained. To improve feature stability, layer normalization is introduced after projection to standardize each image patch embedding, reducing interference caused by feature distribution differences.

The structured semantics contained in the knowledge graph are key to improving the semantic logic of image representations. However, the modality gap between entity embeddings and visual features prevents the effective fusion of semantic information. Therefore, it is necessary to construct a semantic prototype projection mechanism to achieve spatial alignment between the two. The entity set $\mathrm{E}=\left\{e_1, \ldots, e_M\right\}$ and relation triples ($e_h, r, e_t$) related to the current learning content are extracted from the educational knowledge graph, where $M$ is the number of entities, and $e_h, e_t$ are the head and tail entities of relation $r$, respectively. Each entity $e_i$ obtains an initial vector $e_i \in R^D$ through a pre-trained knowledge graph embedding method. On this basis, a learnable semantic prototype projection layer is introduced to map the entity embeddings to the visual feature space, generating semantic prototypes with the same dimension as the image features. This projection layer adopts a two-layer fully connected network structure, which can fully fit the complex mapping relationship between entity semantics and visual features. Its mathematical expression is:

$p_i=W_2 \cdot \operatorname{$R e L U$}\left(W_1 \cdot e_i+b_1\right)+b_2$            (2)

where, $W_1 \in R^{D \times D}$ are $W_2 \in \mathrm{R}^{D \times D}$ the weights of the fully connected layers, $b_1 \in R^D$ and $b_2 \in R^D$ are the bias terms, and $R e L U$ is the activation function used to enhance the nonlinear fitting capability of the model. $p_i$ is the semantic prototype of the $i$-th concept. The semantic prototype projection layer is jointly optimized with the overall network. During the training process, its parameters are updated synchronously with those of subsequent modules such as the attention mechanism and contrastive learning, ensuring that the semantic prototypes can accurately adapt to the visual feature space. This achieves deep alignment between knowledge graph entity semantics and image visual features, providing a reliable semantic benchmark for the subsequent semantic constraint module.

Semantic prototypes are not only the carrier of knowledge graph semantics mapped to the visual space but will also serve as the core reference for the subsequent semantically guided sparse graph attention mechanism, used to filter semantically related image regions. Through the above process of multimodal feature extraction and semantic prototype initialization, the discriminability of the initial image and text features is guaranteed, and a bridge between knowledge semantics and visual features is built through semantic prototypes. This effectively eliminates fusion barriers caused by modality differences, providing a solid foundation for semantic enhancement and personalized optimization of the entire multimodal image representation learning framework.

2.3 Semantically guided sparse graph attention mechanism

The visual attention mechanism is the core of multimodal image representation learning. Traditional self-attention mechanisms achieve feature interaction by calculating the affinity between all pairs of image patches, which not only suffers from high computational complexity of O(N²), but also lacks effective semantic prior guidance. This easily leads to attention being focused on irrelevant background regions, making it difficult to adapt to the characteristics of instructional images that are knowledge-intensive and strongly semantically structured. To address this issue, a semantically guided sparse graph attention mechanism is constructed. Relying on the structured semantics of the educational knowledge graph, it filters semantically related image regions for attention interaction. While reducing computational complexity, it achieves precise constraints of high-level knowledge semantics on visual attention, injecting structured semantic information into the image representations. This mechanism takes the semantic prototypes generated in Section 2.2 as the core reference, and establishes the association between image patches and knowledge concepts by constructing a region-concept bipartite graph, thereby realizing the sparsification and semanticization of attention interaction. Figure 2 illustrates the schematic diagram of the semantically guided sparse graph attention interaction and feature aggregation mechanism.

Figure 2. Schematic diagram of semantically guided sparse graph attention interaction and feature aggregation mechanism

The association strength between image patches and knowledge concepts is the core basis for sparse attention filtering, and it is necessary to quantify the semantic matching degree between the two through similarity calculation. For each image patch feature v_j and each concept semantic prototype p_i, cosine similarity is adopted to measure their semantic association degree. This metric can effectively avoid interference caused by feature scale differences and accurately capture the semantic consistency between the two. The calculation method is:

$s_{j, i}=\frac{v_j \cdot p_i}{\left\|v_j\right\|\left\|p_i\right\|}$             (3)

where, $s_{j, i} \in[0,1]$, and a larger value indicates a closer semantic association between the $j$-th image patch and the $i$-th knowledge concept. To ensure the sparsity and semantic specificity of attention interaction, for each image patch $j$, the top $K$ knowledge concepts with the highest similarity are selected as its strongly associated concepts. The range of $K$ is set to 3 to 5 , which can not only guarantee the semantic integrity of the image patch but also avoid semantic redundancy caused by introducing too many concepts. Based on this, a sparse adjacency matrix $A \in \mathbb{R}^{(N \times M)}$ is constructed, where $A_{j, i}=1$ indicates that the $i$-th concept is a Top-K strongly associated concept of the $j$-th image patch, otherwise $A_{j, i}= 0$. The sparsity of the adjacency matrix is determined by the value of $K$. Through this matrix, the knowledge concepts corresponding to each image patch can be clearly marked, providing a clear basis for the screening of subsequent attention interactions.

Based on the image patch-concept associations marked by the adjacency matrix, a sparse semantic attention mask is further constructed to achieve precise screening of semantically related image regions and attention weight allocation. The calculation of attention scores needs to take into account both the interactive affinity of image patch features and the relevance of knowledge semantics. Its mathematical expression is:

$\alpha_{j, 1}=\frac{\exp \left(\left(W_Q v_j\right)^{\top}\left(W_K v_1\right) / \sqrt{d}\right) \cdot I[\operatorname{path}(j, 1)]}{\sum_{m=1}^N \exp \left(\left(W_Q v_j\right)^{\top}\left(W_K v_m\right) / \sqrt{d}\right) \cdot I[\operatorname{path}(\overline{j, m})]}$             (4)

where, $W_Q \in \mathbb{R}^{(D \times d)}$ and $W_K \in \mathbb{R}^{(D \times d)}$ are the learnable weight matrices for the query and key vectors, respectively; $d$ is the feature dimension of the attention head, usually satisfying $D=$ head $\times d$ (where head is the number of attention heads); $\sqrt{ } d$ is used to alleviate the gradient vanishing problem during the attention score calculation process; $\mathbb{N}[p a t h(j, l)]$ is an indicator function whose value is determined by the semantic association relationship between image patche $j$ and $l$. When the two share at least one common strongly associated concept, or the shortest path length between their strongly associated concepts in the knowledge graph is $\leq 2, \mathbb{N}[p a t h(j, l)]=1$, otherwise it is 0 . This design fully integrates the topological structure of the educational knowledge graph, ensuring that only image patches that are semantically directly or indirectly associated can perform attention interaction. For example, image regions belonging to the same "Mechanics" concept group, or regions indirectly associated through concepts such as "Fulcrum" and "Lever," effectively excluding interference from background noise and irrelevant regions. Meanwhile, the computational complexity of attention interaction is reduced from $O\left(N^2\right)$ to $O(N K)$, significantly improving the model training and inference efficiency.

After the attention weight calculation is completed, the semantic-enhanced aggregation of image patch features is realized through weighted summation, obtaining the semantically constrained image patch representation v_j'. Its calculation method is:

$v_j^{\prime}=\sum_{l=1}^N \alpha_{j, l} W_V v_l$           (5)

where, $W_V \in \mathbb{R}^{(D \times d)}$ is the learnable weight matrix for the value vector, and $\alpha_{j, l}$ is the attention weight of the $j$-th image patch on the $l$-th image patch. A higher weight indicates a greater semantic contribution of that image patch to the current image patch. Through the above process, the semantically guided sparse graph attention mechanism not only realizes the sparsification and semanticization of attention interaction but also deeply integrates the structured semantics of the knowledge graph into the visual feature aggregation process, enabling image patch representations to accurately capture the semantic associations and logical relationships of knowledge points. The enhanced image patch representations generated by this mechanism will serve as the basis for subsequent contrastive learning and personalized optimization, providing strong support for improving the semantic logic and interpretability of multimodal image representations.

2.4 Contrastive learning under relational path constraints

Contrastive learning is a key means to achieve multimodal feature alignment and semantic enhancement. Its core lies in guiding the model to learn discriminative representations by constructing positive and negative sample pairs. Instructional images in educational scenarios often contain causal associations and partial-order evolutionary relationships among multiple knowledge points. Traditional contrastive learning methods can only achieve sample-level alignment between images and corresponding texts, failing to capture such structured semantic relationships. This leads to a lack of knowledge logic association in image representations, making it difficult to meet the requirements of personalized education for knowledge point semantic modeling. To this end, a contrastive learning mechanism under relational path constraints is constructed, integrating the relational path information from the knowledge graph into the design of the contrastive learning loss. This enables image representations to accurately encode the causal and partial-order relationships of educational knowledge points, further enhancing the semantic logic and discriminative ability of the representations. Figure 3 illustrates the framework diagram of the RPC-InfoNCE contrastive learning model under relational path constraints.

The accurate acquisition of image global representations and text representations is the foundation of contrastive learning. It is necessary to construct a unified dimensional multimodal representation space based on the semantically enhanced image patch representations and knowledge graph entity texts mentioned above. For the image patch representation processed by the semantically guided sparse graph attention mechanism v_j', the global average pooling operation is adopted to extract the image global representation z_I. This operation can effectively aggregate the semantic information of all knowledge point regions in the image, balancing local details and global correlations. Its mathematical expression is:

$z_I=\frac{1}{N} \sum_{j=1}^N v_j^{\prime}$             (6)

where, N is the number of image patches, and $z_I \in R^D$ is the image global representation. For each entity in the knowledge graph related to the image e, text fragments such as concept definitions and knowledge point descriptions are extracted from the textbook, and feature extraction is performed through a lightweight text encoder to obtain the entity text representation t_e$\in R^D$ . This ensures that the text representation and the image representation are in the same feature space, providing a basis for subsequent contrastive learning. Meanwhile, the relational path set P related to the current image is extracted from the knowledge graph. Each path corresponds to the causal evolution or partial-order relationship of knowledge points. The path length L is set to 3 to 5 according to the logical complexity of educational knowledge to ensure that the path can completely reflect the association logic between knowledge points.

Figure 3. Framework diagram of the Relation-Path-Constrained (RPC)- Information Noise-Contrastive Estimation (InfoNCE) contrastive learning model under relational path constraints

To enable image representations to encode the relational path information in the knowledge graph, a path ranking loss is designed. By constraining the similarity relationship between the image global representation and the entity text representations along the path, explicit encoding of the causal and partial-order relationships of knowledge points is achieved. The core of the path ranking loss is to force the image representation to maintain a higher similarity with the text representations of subsequent entities along the path direction, thereby restoring the evolution order of knowledge points in the representation space. Its mathematical expression is:

$\frac{1}{|P|} \sum_{\left(e_1, r_1, e_2, \ldots, e_L\right) \in P} \sum_{k=1}^{L_{p a t h}=}\left(0, \operatorname{sim}\left(z_I, t_{e_{k+1}}\right)-\operatorname{sim}\left(z_I, t_{e_k}\right)\right)$               (7)

where, $|P|$ is the number of valid relational paths in the training batch; γ is the margin parameter, ranging from 0.1 to 0.3, used to control the similarity difference between adjacent entity text representations and the image representation on the path, avoiding gradient vanishing or training instability; sim uses cosine similarity to quantify the semantic matching degree between the image global representation and the entity text representation. By introducing indicative constraints, this loss function produces a positive loss value when the similarity between the image representation and the subsequent entity text on the path is smaller than that with the preceding entity text. This forces the model to adjust parameters and optimize the image representation so that it can follow the causal evolution logic of knowledge points.

To ensure the basic alignment capability of multimodal features, the standard multimodal contrastive loss is retained, working synergistically with the path ranking loss to construct the complete total contrastive learning loss. The standard multimodal contrastive loss maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs, achieving sample-level alignment between images and corresponding texts. Its mathematical expression is:

$L_{C L}=-\frac{1}{B} \sum_{i=1}^B \log \frac{\exp \left(\operatorname{sim}\left(z_{I_i}, t_{T_i}\right) / \tau\right)}{\sum_{j=1}^B \exp \left(\operatorname{sim}\left(z_{I_i}, t_{T_j}\right) / \tau\right)}$           (8)

where, $B$ is the training batch size, $z_{I_i}$ is the global representation of the $i$-th image, $t_{T_i}$ is the representation of its corresponding text, and $\tau$ is the temperature parameter, ranging from 0.05 to 0.1 , used to adjust the concentration of the similarity distribution and enhance the discriminative effect of contrastive learning. The total contrastive learning loss fuses the path ranking loss and the standard contrastive loss through the weight parameter $\lambda_{\text {path}}$, achieving the dual goals of basic image-text alignment and knowledge path constraints. Its expression is:

$L_{\text {contrast }}=L_{C L}+\lambda_{\text {path }} L_{\text {path }}$           (9)

where, λ_path is the weight coefficient of the path ranking loss, ranging from 0.5 to 1.0, used to balance the contributions of the two losses. This ensures that the model can not only achieve accurate multimodal alignment but also effectively encode the relational path information of knowledge points. This contrastive learning mechanism deeply integrates the structured semantics of the knowledge graph into the representation learning process, enabling image representations to possess good discriminability while accurately capturing the causal and partial-order relationships of educational knowledge points, providing a semantic foundation for subsequent personalized representation optimization.

2.5 Cognitive state-driven dynamic mask optimization

The core requirement of personalized education is to achieve precise adaptation between teaching content and students' cognitive levels. Reflected in image representation learning, this requires that the same instructional image generates differentiated representations for students with different levels of knowledge mastery. It should strengthen the image regions related to students' knowledge weaknesses while suppressing the regions corresponding to knowledge points already mastered. To achieve this goal, student cognitive state vectors are introduced as prior information into the image representation optimization process. A dynamic semantic mask mechanism is constructed to realize personalized modulation at the attention level, enabling image representations to adaptively match students' cognitive differences. The cognitive state vector can be output in real-time by a knowledge tracing model; this paper takes it as model input to ensure the timeliness and accuracy of cognitive information. Figure 4 presents the computational flow diagram of cognitive state-driven dynamic semantic masking and personalized feature generation.

Figure 4. Computational flow diagram of cognitive state-driven dynamic semantic masking and personalized feature generation

The student cognitive state vector is used to quantify the student's mastery degree of each concept in the knowledge graph. It is defined as $c \in[0,1]^M$, where M is the total number of concepts in the knowledge graph, and c_i represents the probability that the student has mastered the i-th concept e_i. A value closer to 1 indicates a higher mastery level of the concept, while a value closer to 0 indicates that the concept is a knowledge weakness for the student. The generation of the dynamic semantic mask is based on the activation intensity between image patches and concepts, where the activation intensity s_j,i of image patch j on concept i follows the cosine similarity calculated in Section 2.3, which can accurately reflect the semantic association degree between the image patch and the corresponding concept. Based on this, the individual importance weight of the image patch is defined to measure the learning value of the image patch for the current student. Its mathematical expression is:

$w_j=\sigma\left(\sum_{i=1}^M s_{j, i} \cdot\left(1-c_i\right)\right)$             (10)

where, $\sigma$(·) is the sigmoid activation function, specifically $\sigma$(x)=1/1+e^-x, used to normalize the weight value to the [0,1] interval to achieve a soft mask effect. From a physical perspective, if an image patch strongly activates a knowledge weakness concept (i.e., s_j,i is large and c_i is small), then takes a larger value (1-c_i), causing w_j to approach 1, and the semantic information of that image patch will be preserved or even enhanced. If the image patch activates concepts that the student has already mastered, then w_j approaches 0, and the semantic information of that image patch will be softly suppressed. This design not only achieves personalized modulation but also avoids training instability and feature distortion caused by hard masks through the smoothing property of the sigmoid function.

To further improve training stability and avoid feature distortion caused by directly multiplying individual importance weights onto image patch features, the weights are integrated into the re-weighting process of attention scores, realizing the deep fusion of dynamic semantic masks and the attention mechanism. The re-weighted attention scores retain both the semantic-guided sparse characteristics from Section 2.3 and the personalized constraints of the student cognitive state. Its mathematical expression is:

$\tilde{\alpha}_{j, l}=\frac{\alpha_{j, l} \cdot \sqrt{w_j w_l}}{\sum_{m=1}^N \alpha_{j, m} \cdot \sqrt{w_j w_m}}$           (11)

where, α_j,l is the semantically guided sparse attention score obtained in Section 2.3, and $\sqrt{w_j w_m}$ is the geometric mean of the individual weights of image patches j and l, used to balance the contribution of the two image patches in attention interaction. This design allows the model to automatically reduce the attention weight from regions already mastered by the student during feature aggregation, while increasing the weight proportion of regions representing knowledge weaknesses, ensuring that the image representation focuses on the areas with the most learning value for the current student. The introduction of the geometric mean can effectively avoid extreme impacts caused by excessively high or low single image patch weights, ensuring the stability and rationality of the attention aggregation process.

Based on the re-weighted attention scores, feature aggregation is performed to obtain the personalized image patch representation. Its calculation method is:

$\tilde{v}_j=\sum_{l=1}^N \tilde{\alpha}_{j, l} W_V v_l$              (12)

where, $W_V$ is the learnable weight matrix for the value vector, consistent with Section 2.3, to ensure consistency in the feature space. To obtain the global personalized image representation adapted to the student's cognitive state, global average pooling is performed on all personalized image patch representations $\tilde{v}_j$ to obtain the global personalized representation $\tilde{z}_I$. Its expression is:

$\tilde{z}_I=\frac{1}{N} \sum_{j=1}^N \tilde{v}_j$            (13)

This global representation fully integrates the structured semantics of the knowledge graph and the cognitive differences of students, accurately matching the knowledge mastery levels of different students, providing strong support for downstream tasks such as personalized exercise recommendation and learning weakness diagnosis. The entire dynamic mask optimization process is trained collaboratively with other modules of the model, ensuring the precision and effectiveness of personalized modulation, and truly realizing the personalized adaptation of instructional image representations.

2.6 Semantic consistency-driven regularization terms

To effectively prevent the model from overfitting to pseudo-semantic correlations and enhance the semantic consistency and interpretability of image representations, two targeted regularization terms are introduced relying on the topological structure of the knowledge graph. These terms optimize the representation distribution from the dimensions of knowledge topology alignment and concept hierarchy modeling, respectively. They ensure that the image representations learned by the model accurately match the structured characteristics and hierarchical logic of educational knowledge, collaborating with the semantic constraint and personalized optimization modules mentioned above to achieve end-to-end joint optimization.

The core of Graph Laplacian Semantic Consistency Regularization is to drive the similarity relationship between image patch representations to be consistent with the semantic distance of corresponding concepts in the knowledge graph, thereby strengthening the semantic rationality of the representations. First, the activated concept set corresponding to all image patches C_active is extracted, and an undirected graph is constructed with image patches as nodes. The edge weights in this graph are determined by the semantic distance between the concepts activated by the corresponding image patches in the knowledge graph. The edge weight calculation adopts an exponential decay function, which can transform the semantic distance between concepts into discriminative weight values. Its expression is:

$A_{j l}^{k g}=\exp \left(-\delta \cdot d_{K G}\left(i_j, i_l\right)\right)$             (14)

where, $d_{K G}\left(i_j, i_l\right)$ represents the shortest path length between the concepts activated by image patches $j$ and $l$ in the knowledge graph. If an image patch activates multiple concepts, the minimum value among the shortest path lengths is selected to ensure that the weight can accurately reflect the core semantic association between image patches. $\delta$ is the scale parameter, ranging from 0.2 to 0.5 , used to adjust the influence degree of semantic distance on edge weights; the larger the $\delta$, the more significant the attenuation effect of semantic distance on the weight. Meanwhile, the cosine similarity matrix $S_{j l}$ between the personalized image patch representations $\tilde{v}_j$ and $\tilde{v}_l$ is calculated to quantify the semantic association degree of the image patch representations. Its calculation formula is:

$S_{j l}=\frac{\tilde{v}_j \cdot \tilde{v}_l}{\left\|\tilde{v}_j\right\|\left\|\tilde{v}_l\right\|}$          (15)

To achieve low-rank alignment of the two matrices, a Graph Laplacian regularization loss is constructed. By minimizing this loss, the similarity of image patch representations is forced to be consistent with the semantic distance of the knowledge graph. Its mathematical expression is:

$L_{l a p}=\sum_{j, l} A_{j l}^{k g}\left\|S_{j l}-I_{\{j=l\}}\right\|_F^2$             (16)

where, $1_{\{\mathrm{j}=1\}}$ is an indicator function, which takes the value of 1 when j = l, otherwise 0; ||·||_F² is the square of the Frobenius norm, used to measure the difference between matrices. This regularization term can effectively suppress interference caused by pseudo-semantic associations, ensuring that semantically similar image patch representations have higher similarity and improving the semantic consistency of the representations.

Hyperbolic geometric hierarchical constraint​ is designed for the hierarchical characteristics of educational concepts. Utilizing the natural advantage of hyperbolic space in modeling hierarchical inclusion relationships, it enables image representations to accurately match the hierarchical structure of knowledge concepts, conforming to educational cognitive laws. Educational knowledge concepts have clear hierarchical relationships, which are difficult to model effectively using Euclidean space due to their nested containment properties. In contrast, hyperbolic space can naturally represent the hierarchical distance of concepts through distance differences. Therefore, the image global representation and concept prototypes are projected into the Poincaré ball hyperbolic space. The projection of the image global personalized representation $\tilde{z}_I$ is achieved through exponential mapping, and its expression is:

$h_I=\exp _0\left(\tilde{z}_I\right)=\tanh \left(\left\|\tilde{z}_I\right\|\right)^{\frac{\tilde{z}_I}{\left\|\tilde{z}_I\right\|}}$           (17)

where, $\exp _0$ represents the exponential mapping centered at the origin of the hyperbolic space, and the tanh function is used to normalize the representation inside the Poincaré ball, ensuring that the projected data satisfies the hyperbolic space constraints. Similarly, each concept prototype p_i is projected into the same hyperbolic space via the same exponential mapping to obtain the concept prototype h_i in hyperbolic space. To constrain the distribution of image representations in hyperbolic space to conform to the concept hierarchy, a hyperbolic geometric hierarchical loss is constructed. It requires the hyperbolic distance between the image representation and the most specific concept to be minimal, and the distance to the upper-level generalized concepts to increase hierarchically. Its mathematical expression is:

$L_{h y p}=\sum_{k=1}^{L-1} \max \left(0, d_H\left(h_I, h_{e_{c_{k+1}}}\right)-d_H\left(h_I, h_{e_{c_k}}\right)+\mu\right)$           (18)

where, $d_H(u, v)$ is the distance metric in hyperbolic space, calculated as:

$d_H({u, v})=\operatorname{arcosh}\left(1+2 \frac{\|u-v\|^2}{\left(1-\|u\|^2\right)\left(1-\|v\|^2\right)}\right)$               (19)

$\mu$ is the margin parameter, ranging from 0.1 to 0.3, used to control the constraint strength of the distance difference between hierarchies to avoid hierarchy confusion; $e_r \rightarrow e_{c 1} \rightarrow \cdots \rightarrow e_{l e a f}$ is the hierarchical path from the root concept to the most specific leaf concept in the knowledge graph, and L is the path length. This constraint ensures that the image representation can be precisely located at the corresponding concept level in hyperbolic space, improving the hierarchical modeling capability and interpretability of the representation.

Combining the contrastive learning loss from the previous section, the Graph Laplacian semantic consistency regularization loss, and the hyperbolic geometric hierarchical constraint loss, the total model loss function is constructed to achieve collaborative optimization of each module. Its expression is:

$L_{\text {total }}=L_{\text {contrast }}+\lambda_{\text {lap }} L_{\text {lap }}+\lambda_{\text {hyp }} L_{\text {hyp }}$           (20)

where, $\lambda_{\text {lap}}$ and $\lambda_{\text {hyp}}$ are the weight coefficients of the two regularization losses, respectively, both ranging from 0.1 to 0.3. They are used to balance the contribution of each loss, ensuring that the model can effectively improve the semantic consistency, hierarchical modeling capability, and generalization of the representations while achieving multimodal alignment, semantic constraints, and personalized adaptation, thereby avoiding overfitting problems. The design of the total loss function balances the performance and stability of the model, providing a reliable optimization target for the end-to-end training of the entire multimodal image representation learning framework.

2.7 Training and inference pipeline

The multimodal image representation learning framework proposed in this paper adopts an end-to-end training mode. The entire architecture is fully differentiable, and the parameters of each module are co-optimized to ensure that the model can accurately capture knowledge graph semantic constraints and student cognitive differences, while balancing training stability and representation performance. In the training phase, a multimodal dataset containing textbook images, corresponding text fragments, and knowledge graph triples is used. The text fragments in the dataset are derived from textbook knowledge point definitions and exercise stems, and the knowledge graph triples cover core concepts and association relations in the education field, providing sufficient semantic supervision information for the model. During the training process, a batch of data is first randomly sampled and input into the model. The pre-trained ViT is used to extract the initial features of image patches, while a lightweight text encoder is used to extract the initial representations of text fragments and knowledge concepts. Subsequently, the semantically guided sparse graph attention mechanism calculates the semantically associated attention weights to generate semantically enhanced image patch representations. Next, the student cognitive state vector obtained offline by the knowledge tracing model is input to generate a dynamic semantic mask and re-weight the attention weights, yielding personalized image patch representations. Finally, the contrastive learning loss, Graph Laplacian semantic consistency regularization loss, and hyperbolic geometric hierarchical constraint loss are calculated. Backpropagation is used to update all learnable parameters of the model, realizing the collaborative optimization of each module until the model converges.

The inference phase focuses on the practical application requirements of personalized education scenarios. The process is concise and efficient, requiring no additional complex preprocessing steps. For a new student, their knowledge cognitive state vector is first obtained, and this vector is input into the trained model together with the instructional image to be processed. The model will automatically complete a series of operations including image feature extraction, semantic constraint enhancement, and personalized mask modulation, ultimately outputting the image global personalized representation adapted to the student's cognitive level. This representation can be directly used in downstream tasks such as cross-modal image retrieval, knowledge point clustering, personalized exercise recommendation, and visual diagnosis of learning weaknesses, providing precise multimodal semantic support for personalized education services, fully reflecting the engineering practicality and scenario adaptability of the model.

The experimental results presented above fully verify the effectiveness of the multimodal image representation optimization method integrating knowledge graph semantic constraints proposed in this paper. Its performance improvement stems from the synergistic effect of four core technical mechanisms, which compensate for the deficiencies of traditional image representation methods in personalized education scenarios from different dimensions. The sparse graph attention mechanism filters image regions strongly correlated with knowledge points through knowledge graph semantic guidance, effectively eliminating irrelevant background noise. While reducing computational complexity, it ensures the semantic specificity of the representations. This is also the core reason why the attention semantic matching degree of the proposed method is significantly better than that of the baseline models; the AME metric decreased by 19.1 percentage points compared to the traditional ViT, confirming the precise regulation effect of this mechanism on attention allocation. Relational path contrastive learning encodes the causal partial-order relationships of knowledge points into the image representations by constructing a path-level contrastive loss. This breaks through the limitation of traditional contrastive learning, which only achieves shallow image-text alignment, enabling the representations to possess stronger logical correlation and pushing the mAP of cross-modal retrieval to 88.9%. The introduction of cognitive state-driven dynamic masking achieves precise matching between representations and student cognitive levels. By adaptively modulating image region weights, the F1 score of the personalized recommendation task reached 80.9%, fully reflecting the scene adaptation capability. The hyperbolic-Laplacian joint regularization optimizes the representation distribution from the dimensions of topological alignment and hierarchical modeling, ensuring both the consistency between image patch representations and knowledge graph semantics, and accurately characterizing conceptual hierarchies through hyperbolic space, thereby enhancing the interpretability and stability of the representations and guaranteeing performance improvements in various downstream tasks. The four mechanisms collaborate with each other to form a complete chain of "semantic screening – logical encoding – personalized adaptation – distribution optimization," constructing a multimodal image representation system tailored to the needs of personalized education.

Although the proposed method demonstrates excellent performance in various experiments, considering the practical application requirements of personalized education scenarios, there are three limitations that need to be addressed. First, the model representation is susceptible to interference from instructional images containing complex overlapping knowledge points. When multiple highly correlated knowledge point regions overlap within an image, the sparse graph attention mechanism struggles to accurately distinguish the boundaries of each knowledge point, leading to deviations in semantic association judgments, which in turn affects cross-modal alignment and clustering performance. This is also the main reason why the model's performance in Chemistry scenarios is slightly lower than in Physics. Second, there are deficiencies in modeling high-order knowledge relational paths. The current model can only effectively encode single-chain relational paths and finds it difficult to achieve comprehensive semantic encoding for complex knowledge point relationships involving multiple branches and cross-associations, leading to a decline in representation accuracy in certain complex scenarios. Third, the computational overhead of hyperbolic space is relatively high. Compared to Euclidean space, the calculation of hyperbolic distance involves inverse hyperbolic functions, increasing the model's inference time. Especially in resource-constrained scenarios like mobile devices, it is difficult to meet real-time processing requirements, limiting the scope of engineering deployment. These limitations are closely related to the particularity of personalized education scenarios and represent the core directions for subsequent optimization.

Addressing the limitations of the proposed model and combining them with the deployment requirements of personalized education scenarios, future research will proceed in four directions to promote the practicality and performance improvement of the method. First, research will be conducted on lightweight visual representation deployment schemes. Through model pruning, quantization, and knowledge distillation techniques, the overhead of hyperbolic space computation and sparse attention will be reduced to achieve real-time inference on mobile devices, adapting to real-time application scenarios such as classroom teaching and online Q&A. Second, large models will be introduced to enhance knowledge graph semantic modeling. Leveraging the semantic understanding capabilities of pre-trained large models in the education domain, the relational mining and representation learning of knowledge graphs will be optimized to improve the encoding accuracy of high-order complex knowledge point relationships and solve the problem of representation interference from overlapping knowledge points. Third, multi-scale image feature fusion technology will be explored. Combining fine-grained local features with global semantic features will enhance the representation capability for complex instructional images, further improving the performance of knowledge point clustering and cross-modal retrieval. Finally, research will be conducted on real-time instructional image analysis on the mobile side. End-side applications adapted to personalized education will be developed to achieve real-time semantic parsing and personalized feedback for instructional images, promoting the deep deployment of multimodal image representation technology in personalized education scenarios and providing more efficient and precise technical support for smart education.