Screen Image Understanding, Interface Element Detection, and Intelligent Analysis of Visual Operational Behaviors for Adult Digital Literacy Education

To address the critical challenges in adult digital literacy assessment scenarios—namely, poor adaptability of interface detection, absence of structured interface semantic representation, and insufficient cognitive reasoning capabilities for operations—a four-tiered end-to-end visual intelligence analytical architecture, comprising perception, modeling, reasoning, and evaluation, is established to enable the precise transformation of screen operation visual data into quantitative digital literacy indicators. The proposed approach takes as fundamental inputs the continuous screen recording video sequences and synchronized mouse trajectory data of adult learners, accepting time-series screen image frames at a resolution of 1920 × 1080, and progressively accomplishes visual feature extraction, structural semantic modeling, cognitive behavioral reasoning, and quantitative literacy evaluation through hierarchically organized functional modules. At the perception tier, focused on the differentiated interaction behavioral characteristics of adult users, an adaptive interface element detection mechanism is constructed, in which feature learning in high-frequency interaction regions is reinforced, thereby effectively enhancing the detection accuracy and robustness of multi-scale interface elements within complex real-world operational contexts. At the modeling tier, relying on the interface element information acquired through detection, the spatial affiliation relationships and functional semantic attributes among interface units are mined, enabling the structured organization and high-order semantic representation of discrete visual elements, which compensates for the deficiency of existing methods in interface functionality cognition. At the reasoning tier, dynamic interface state transition characteristics are integrated with temporal contextual regularities of sequential operations to establish a complete reasoning logic that ascends from basic operation recognition to high-level behavioral intention inference, thereby achieving deep interpretation of learners' cognitive logic in digital operations. At the evaluation tier, the visual detection, structural modeling, and behavioral reasoning results from the preceding modules are synthesized to construct a multidimensional quantitative digital literacy evaluation system and a visual feedback mechanism, forming a closed-loop technical architecture that encompasses data acquisition, intelligent analysis, literacy assessment, and instructional intervention, and providing systematic algorithmic support for automated, fine-grained, and objective adult digital literacy assessment.

2.1 Adaptive interface element detection network

To address the characteristics of adult learners' interface interaction behaviors that differ from those of professional users, as well as the issues of high miss-detection rates for small-scale interface elements and weak feature adaptability exhibited by conventional detection models in real adult operational scenarios, an adaptive interface element detection network is constructed based on YOLO11-Large. The architecture of the adaptive interface element detection network is illustrated in Figure 1. Four groups of multi-scale feature maps are extracted by the network backbone, corresponding to down-sampling rates of fourfold, eightfold, sixteenfold, and thirty-twofold, respectively, thereby covering multi-dimensional feature information of screen interfaces ranging from micro-level textures to global semantics. To address the problem of small-scale user interface feature vanishing within deep networks, a bidirectional feature pyramid structure is introduced at the feature fusion stage. Through a weighted bidirectional cross-scale connection mechanism, adaptive multi-level feature propagation and reconstruction are accomplished, enabling efficient fusion of shallow fine-grained textural features and deep high-level semantic features, and effectively enhancing the foundational perceptual capability of the model for miniature interactive controls. The reconstructed multi-scale feature representations are subsequently utilized for feature aggregation and detection inference, with the corresponding computational formulation given as follows:

$\widetilde{F}_l=\operatorname{BiFPN}\left(F_1, F_2, F_3, F_4\right)$          (1)

where, $F_1, F_2, F_3$, and $F_4$ denote the four-layer raw multi-scale feature maps output by the backbone network, and $\widetilde{F}_l$ represents the enhanced feature maps obtained through bidirectional weighted feature fusion.

tu_pian_1.png

Figure 1. Architecture of the adaptive interface element detection network

To accommodate the visual characteristics of graphical interface elements—namely, their variable scales, irregular morphologies, and dense arrangements—a deformable attention mechanism is employed to accomplish adaptive aggregation of multi-scale features, thereby overcoming the limitations of fixed receptive field-based feature fusion. Global contextual information of images is extracted through global average pooling and a multi-layer perceptron, and dynamic fusion weights for features at each scale are generated via normalization processing, enabling differentiated weighted aggregation of features across hierarchical levels. During feature sampling, multi-point offset sampling is performed around reference positions, and learnable attention weights along with feature projection matrices are leveraged to adaptively capture the local structural characteristics of interface controls, precisely accommodating the morphological distribution features of irregular user interface elements. The overall computational process of multi-scale feature fusion and the deformable attention sampling formulation can be expressed as:

$F_{\text {fusion }}=\sum_{l=1}^4 \alpha_l \cdot$ $DeformAttn\left(\widetilde{F}_l, P_l\right)$       (2)

$\operatorname{DeformAttn}\left(\widetilde{F_l}, P_l\right)=\sum_{k=1}^K W_k \widetilde{F_l}\left(p+p_{\text {ref }}+\Delta p_{l, k}\right) \cdot \Delta m_{l, k}$        (3)

where, $\alpha_l$ denotes the dynamic fusion weight for each scale's enhanced features, adaptively generated from the global context of the image; $P_l$ represents the positional encodings corresponding to features at each hierarchy; $p$ denotes the feature query position, and $p_{\text {ref }}$ is the local reference position; $\Delta p_{l, k}$ denotes the learnable sampling offsets, designed to accommodate irregular control morphologies; $\Delta m_{l, k}$ is the spatial attention weight scalar; $W_k$ is the feature projection matrix, employed for dimensional alignment and feature refinement; and $K$ is the number of sampling points per reference point. Based on the aggregated refined features, the network is capable of outputting bounding box coordinates, category labels, and interactivity confidence scores for interface elements, thereby comprehensively characterizing the visual compositional information of screen interfaces.

To align with the interaction behavioral regularities of adult learners—characterized by cursor hesitation, slow movement, and dispersed click landing points—a temporal decay heatmap guidance strategy is designed based on time-series mouse trajectory data, enabling behavior-adaptive optimization of the detection model. A fixed temporal sliding window is established to integrate cursor trajectory information across consecutive frames, through which a temporal decay factor is employed to attenuate the interfering effects of long-past trajectories while reinforcing the feature weights of interactions proximate to the current moment, and a global screen attention heatmap is generated via two-dimensional Gaussian kernel density estimation. The diffusion scale of the Gaussian kernel is dynamically adjusted by the real-time cursor movement speed: under slow or stationary conditions, the kernel size is contracted to focus on precise interaction regions, whereas under fast-moving conditions, the kernel size is expanded to accommodate the attentional dispersion characteristics of behavior, thereby faithfully reflecting real adult operational habits. Finally, the normalized attention heatmap is fused with the aggregated features through element-wise weighted fusion, guiding the model to preferentially learn feature representations from high-frequency user interaction regions while suppressing feature responses in ineffective background areas. The computational formulations for heatmap generation and feature guidance are given as follows:

$A_{\text {heat }}(u, v)=\sum_{\tau=t-\Delta T}^t \gamma^{t-\tau} \cdot N\left((u, v) ; p_\tau^{\text {cursor }}, \sigma_\tau^2 I_2\right)$       (4)

$\sigma_\tau=\sigma_{\text {base }}+\eta \cdot v_\tau$       (5)

$F_{\text {guided }}=F_{\text {fusion }} \odot N o r m\left(A_{\text {heat }}\right)$        (6)

where, $\Delta T$ denotes the temporal sliding window length; $\gamma$ is the temporal decay factor used to regulate the weight attenuation rate of historical trajectories; $u, v$ are the spatial coordinates of the screen image; $p_\tau^{\text {cursor}}$ is the cursor coordinate at time $\tau; N$ denotes the twodimensional Gaussian distribution function; $I_2$ is the second-order identity matrix; $\sigma_\tau$ is the adaptive Gaussian kernel diffusion scale; $\sigma_{\text {base}}$ is the base kernel size; $\eta$ is the velocity adjustment coefficient; $v_\tau$ is the instantaneous cursor movement speed; ⊙ denotes elementwise multiplication of features; and Norm denotes the min-max normalization operation.

A multi-component joint loss function is constructed for holistic optimization of the detection network, coordinating the multi-dimensional training objectives of interface element localization, classification, and attention feature constraints to accommodate the complex characteristics of adult interface detection tasks. The overall loss function is composed of bounding box regression loss, category classification loss, and attention regularization loss, with the specific formulation given as:

$L_{\text {det }}=L_{\text {bbax }}+L_{\text {cls }}+\lambda \cdot L_{\text {attn }}$          (7)

For bounding box regression, the complete intersection over union loss is adopted, which comprehensively accounts for the overlap area, center distance, and aspect ratio discrepancy between predicted and ground-truth boxes, thereby enhancing localization accuracy for densely arranged interface elements. For the classification task, the Focal Loss function is introduced to suppress the training weights of numerous simple background samples while focusing optimization on small-scale and rare controls, thereby alleviating the class imbalance problem among interface elements. The attention regularization loss, through constraining feature responses in non-interactive regions, aligns with the heatmap-guided feature learning mechanism and further reinforces the model's detection capability for effective interaction regions. The three loss components are jointly optimized through gradient coordination via fixed balancing weights, ensuring stability and precision of the network in adult screen interface detection tasks. The specific computational rules for each sub-loss are given as follows:

$L_{\text {CIoU }}=1-I o U\left(B, B^{g t}\right)+\frac{\rho^2\left(b, b^{g t}\right)}{c^2}+\beta \cdot v$        (8)

$L_{c l s}=-\sum_{i=1}^N \alpha_c\left(1-p_i(c)\right)^\kappa \log p_i(c)$        (9)

$L_{\text {attn }}=\left\|\left(1-N o r m\left(A_{\text {heat }}\right)\right) \odot F_{\text {fusion }}\right\|_2$     (10)

where, $\lambda$ denotes the attention loss balancing weight; $B$ and $B^{g t}$ represent the predicted box and the ground-truth annotation box, respectively; $b$ and $b^{g t}$ are the center coordinates of the corresponding boxes; $\rho$ denotes the Euclidean distance function; $c$ is the diagonal length of the minimum enclosing bounding box; $v$ is the aspect ratio consistency measure; $\beta$ is the aspect ratio trade-off coefficient; $N$ is the total number of interface elements in a single frame; $\alpha_c$ is the class balancing weight; $p_i(c)$ is the model's predicted probability for the $i$-th element belonging to class $c ; \kappa$ is the focusing parameter, employed to suppress gradients from easily classified samples; and 1 denotes the all-ones matrix.

2.2 Hierarchical screen semantic graph

Interface element detection alone yields only discrete control entities and visual features within screen images, lacking a structured representation of the overall spatial topology and functional associations of the interface, and thus cannot adequately support cognitive-level interpretation of user operational behaviors. To compensate for the limitations of flattened element representation, a hierarchical screen semantic graph model is constructed in this section, transforming scattered interface controls into a structured topological architecture endowed with spatial hierarchies and functional semantic associations. The detected interface elements are taken as fundamental nodes, and spatial nesting relationships along with local neighborhood association features are incorporated, enabling the screen interface to be elevated from a mere visual collection to a functionally structured system, thereby providing semantic-level support for subsequent operational intention reasoning and fine-grained digital literacy assessment. The construction and classification workflow of the hierarchical screen semantic graph is illustrated in Figure 2.

tu_pian_2.png

Figure 2. Construction and classification workflow of the hierarchical screen semantic graph

To construct a robust interface topological structure, a dual-edge association mechanism is designed by integrating the spatial distribution characteristics of interface elements, effectively circumventing the problem of structural hierarchy fragmentation caused by minor detection box errors. All detected interface controls are defined as a set of graph nodes, with each node carrying dedicated spatial coordinates, category attributes, and visual feature information. To address the prevalent container nesting structures within interfaces, a soft containment score is introduced to quantify the affiliation relationships between nodes, thereby mitigating the structural noise induced by hard-threshold decisions. When the computed containment degree between elements exceeds a preset threshold, a directed container association edge is established to characterize the hierarchical nesting relationships of the interface. The specific computational formulation is given as:

$s_{i j}^{\text {contain }}=\operatorname{IoU}\left(b_i, b_j\right) \cdot 1\left[\begin{array}{c}x_i \leq x_j, y_i \leq y_j, x_i+w_i \geq x_j+w_j, y_i+h_i \geq y_j+h_j\end{array}\right]$        (11)

For densely arranged sibling interface elements without nesting associations, a spatial proximity decision rule is further introduced to construct adjacency edges, where the degree of element association is measured through the relative ratio of node center distances to control dimensions, thereby comprehensively covering the topological association relationships of all elements within the interface. The corresponding spatial edge decision rule is formulated as follows:

$e_{i j}^{\text {spatial }}=1\left[\frac{\| \text {center}_i-\text {center}_j \|_2}{\min \left(w_i, h_i\right)+\min \left(w_j, h_j\right)}<\gamma\right]$        (12)

where, $x_i, y_i, w_i$, and $h_i$ denote the horizontal coordinate, vertical coordinate, width, and height of the bounding box of the $i$-th interface element, respectively; $center_i$ denotes the center coordinates of the element box; $\gamma$ is the spatial association threshold, with a fixed value of 1.5, which is adapted to the typical arrangement regularities of conventional interface controls; and 1 denotes the indicator function, which outputs 1 when the condition is satisfied and 0 otherwise. By integrating the two types of association edges, a complete hierarchical screen semantic graph topological structure is formed.

To achieve multi-dimensional feature representation of graph nodes, three types of information—visual semantics, geometric structure, and category semantics—are fused to accomplish node feature initialization, thereby constructing high-dimensional and refined initial embedding features for nodes. Control-specific visual features are extracted from the fused guided feature maps through region feature alignment, characterizing the appearance and textural information of controls, while the geometric parameters of interface elements, including coordinates, width, and height, are normalized to encode the spatial structural attributes of interface controls. Additionally, vector embedding encoding is performed on element categories to supplement the inherent category semantic information of controls. After dimensionality transformation via a multi-layer perceptron, the three types of features are concatenated and fused to form a unified node feature of 256 dimensions. The overall initialization process is formulated as follows:

$\begin{aligned} h_i^{(0)}= & \operatorname{MLP}\left(\text { RoIAlign }\left(F_{\text {guided }}, b_i\right)\right) \oplus \operatorname{MLP}\left(\left[b_i ; c_i\right]\right) \oplus \text { Embed }\left(c_i\right)\end{aligned}$        (13)

where, $h_i^{(0)}$ denotes the initial feature vector of the $i$-th node; RoIAlign denotes the region feature alignment operation; $b_i$ denotes the element bounding box features; $c_i$ denotes the element category label; ⊕ denotes the vector concatenation operation; and Embed denotes the category embedding encoding function. The multi-source feature fusion approach simultaneously preserves the visual appearance, spatial structure, and semantic attributes of the interface, thereby providing comprehensive initial feature support for the graph model to mine deep functional associations.

A three-layer graph attention network is employed to accomplish deep encoding of the semantic graph, where adaptive attention mechanisms are leveraged to mine the differentiated semantic weights of distinct nodes and association edges, thereby achieving high-order representation of interface hierarchical semantics. Attention coefficients are dynamically computed based on the nodes’ intrinsic features and neighborhood association features, while edge type embedding vectors are introduced to differentiate the semantic distinctions between container containment edges and spatial adjacency edges, enabling the model to precisely identify both the hierarchical structure and sibling-level association relationships of the interface. The iterative update rules for node features and the computational formulation for attention coefficients are expressed as:

$h_i^{(l+1)}=\sigma\left(\sum_{j \in N(i)} \alpha_{i j}^{(l)} W^{(l)} h_j^{(l)}\right)$       (14)

$\alpha_{i j}=\frac{\exp \left(\operatorname{LeakyReLU}\left(a^T\left[W h_i \oplus W h_j \oplus \phi\left(e_{i j}\right)\right]\right)\right)}{\sum_{k \in N(i)} \exp \left(\operatorname{LeakyReLU}\left(a^T\left[W h_i \oplus W h_k \oplus \phi\left(e_{i k}\right)\right]\right)\right)}$       (15)

where, $h_i^{(l)}$ denotes the feature vector of node $i$ at the $l$-th network layer; $\sigma$ denotes the non-linear activation function; $N(i)$ denotes the set of neighboring nodes of node $i ; \alpha^{(l)}{ }_{i j}$ denotes the layer-wise normalized attention coefficient; $W^{(l)}$ denotes the learnable weight matrix; $a$ denotes the attention scoring vector; and $\phi\left(e_{i j}\right)$ denotes the learnable edge type embedding features, employed to differentiate nested associations from spatial adjacency associations. The network is configured with an eight-head attention mechanism and 256-dimensional hidden layer features. Following iterative encoding, the node feature dimensions are compressed to 128 dimensions, thereby accomplishing deep condensation of structured semantic features.

Based on the encoded high-order node features, semantic role classification of interface elements is accomplished, enabling automatic partitioning of interface functional semantics. Interface elements are uniformly categorized into six functional roles: navigation elements, content display areas, action triggers, input controls, status indicators, and containers. The semantic role probability distribution for each node is output through a linear classifier and a softmax function, with the optimal label being selected as the final classification result. The specific computational formulation is given as:

$\hat{r}_i=\arg \max _{r \in R} \operatorname{Softmax}\left(W_r h_i^{(L)}+b_r\right)$      (16)

To optimize the accuracy of semantic role classification, a node-level cross-entropy loss function is employed to supervise network training, constraining the deviation between node prediction labels and ground-truth functional labels on a per-node basis. The overall loss function is defined as follows:

$L_{H S S G}=-\frac{1}{N} \sum_{i=1}^N \sum_{r \in \mathrm{R}} y_{i, r} \log \hat{y}_{i, r}$       (17)

where, $R$ denotes the predefined set of semantic role labels; $W_r$ and $b_r$ denote the classification layer weight and bias parameters, respectively; $\hat{r}_i$ denotes the predicted semantic role of the node; $y_{i, r}$ denotes the one-hot label of the ground-truth semantic role; $\hat{y}_{i, r}$ denotes the model-predicted probability; and $N$ denotes the total number of graph nodes. Through end-to-end training, the model is capable of precisely distinguishing the functional attributes of each interface element, thereby establishing a comprehensive interface functional semantic system and achieving effective transformation from screen visual information to functional cognition.

2.3 Visual operational behavior intention reasoning module

After structured modeling of interface spatial structures and functional semantics is accomplished through the hierarchical screen semantic graph, further cognitive-level interpretation of learners' dynamic interaction behaviors can be performed. Existing interface behavior analysis methods are only capable of recognizing and classifying basic interaction actions, without being able to incorporate interface functional attributes and operational temporal logic to infer the task objectives underlying user behaviors, and thus cannot adequately support the cognitive capability assessment required for digital literacy. To address this limitation, a multi-level visual operational behavior intention reasoning module is constructed, establishing a progressive reasoning pipeline composed of operation type recognition, operation target localization, and operation intention inference. Through the integration of spatiotemporal interface variation features, structured semantic information, and historical operation contexts, precise interpretation from surface-level behavioral observation to deep cognitive intention is achieved. The framework of the multi-level visual operational behavior intention reasoning architecture is illustrated in Figure 3.

Operation type recognition serves as the foundational tier of behavioral reasoning. To address the issue that single-frame images are incapable of distinguishing between similar interaction actions such as clicking and dragging, a long-short-term temporal difference mechanism is introduced to quantify the visual variation patterns of interfaces across consecutive frames. By computing the image difference features between adjacent frames and between frames with a five-frame interval, instantaneous operation actions and delayed interface feedback are captured separately, thereby effectively differentiating short-duration static interactions from sustained dynamic interactions. Consecutive three-frame images are selected as the fundamental input, and a lightweight 3D convolutional network is employed to extract spatiotemporal joint features. Furthermore, the cursor position heatmap is fused to reinforce feature responses in interaction regions, thereby achieving precise discrimination of six categories of basic interface operations. The overall recognition process is formulated as follows:

$D_t^{\text {short }}=\left|I_t-I_{t-1}\right|, D_t^{\text {long }}=\left|I_t-I_{t-5}\right|$       (18)

$\hat{a}_t=\arg \max _{a \in A} p\left(a_t \mid I_{t-2: t}, D_t^{\text {short }}, D_t^{\text {long }}, A_t^{\text {cursor }}\right)$       (19)

where, $I_t$ denotes the screen image at frame $t ; D_t^{\text {short }}$ denotes the short-term inter-frame difference feature, employed to characterize instantaneous interaction changes; $D_t^{\text {long }}$ denotes the long-term inter-frame difference feature, employed to characterize delayed interface feedback; $I_{t-2: t}$ denotes the consecutive three-frame input image sequence; $A$ denotes the predefined set of operation types; and $\hat{a}_t$ denotes the operation category at the current moment predicted by the model.

tu_pian_3.png

Figure 3. Framework of the multi-level visual operational behavior intention reasoning architecture

Upon clarification of the operation type, precise localization of the operation target is accomplished by integrating the structured semantic information of the interface. Different from traditional localization approaches that rely solely on geometric coordinate matching, the interaction coordinates are mapped into the node system of the hierarchical screen semantic graph, and the target control node corresponding to the current operation is identified by computing the Euclidean distance between the operation coordinates and the center positions of each interface element, thereby precisely matching the target object of the user interaction behavior. The screening rule for the target node is defined as:

$v_{\text {target }}=\arg \min _{v_i \in V} \| p_t^{\text {action }}$-center $_i \|_2$       (20)

where, $p_t^{\text {action}}$ denotes the interaction operation coordinates at time $t ; V$ denotes the set of all interface nodes in the semantic graph; and $v_{\text {target}}$ denotes the optimally matched operation target node. By leveraging the topological constraints of the semantic graph, this localization approach effectively circumvents localization deviations caused by complex interface layouts and control occlusions. Upon completion of localization, the functional semantic roles of the target node and its neighborhood node features are simultaneously extracted, thereby providing interface semantic contextual support for subsequent high-level intention inference.

Operation intention inference constitutes the core component of behavioral cognitive analysis, through which high-level task objective reasoning is accomplished by integrating interface state transitions, individual operation attributes, and historical operational temporal logic. Visual difference features of the interface before and after operations are first extracted, where visual feedback information—including interface content updates and control state transitions—is captured via frame differencing between adjacent images, and fixed-dimensional state transition feature representations are generated through a lightweight convolutional network:

$\Delta I_t=\left|I_{t+1}-I_{t-1}\right|$       (21)

To mine the potential logical associations among sequential operations, a temporal sequence constructed from five consecutive historical operations is selected, and the action type, semantic role, and control geometric features of each operation are uniformly encoded to form a structured temporal feature vector:

$E_\tau=\operatorname{Embed}\left(a_\tau\right) \oplus \operatorname{Embed}\left(r_{\text {target }, \tau}\right) \oplus M L P\left(b_{\text {target, }, \tau}\right)$       (22)

Global modeling of the temporal sequence is performed through a two-layer Transformer encoder, capturing causal relationships and task logic among different operations, and temporal contextual features are output as follows:

$h_{\text {context }}=\operatorname{Transformer}\left(E_{t-K}, \ldots, E_{t-1}\right) \in \mathrm{R}^{256}$     (23)

The temporal contextual features, interface state transition features, and target semantic role features are finally fused, and classification discrimination of seven categories of operation intentions is accomplished through a softmax function. The complete inference formulation is given as:

$\begin{gathered}p\left(z_t \mid H_{t-K: t}, \Delta I_t, r_{\text {target }}\right)= \operatorname{Softmax}\left(W_z\left[h_{\text {context }} \oplus f_{\Delta t} \oplus E_{\text {mbed }}\left(r_{\text {target }}\right)\right]+b_z\right)\end{gathered}$      (24)

where, $\Delta I_t$ denotes the interface state transition feature map; $E_\tau$ denotes the single-step operation temporal embedding feature; $H_{t-K: t}$ denotes the set of historical operation sequences; $K$ denotes the temporal context window length; $h_{\text {context }}$ denotes the global temporal contextual features; $f_{\Delta t}$ denotes the deep interface transition features; $W_z$ and $b_z$ denote the intention classification layer weight and bias parameters; and $z_t$ denotes the predicted operation intention. The corresponding intention categories encompass core task scenarios of digital operations, including file management, format adjustment, information retrieval, and others, which comprehensively align with the task framework of adult digital literacy assessment.

To ensure training stability for multi-task collaborative optimization, a multi-task joint learning strategy is adopted, through which the three sub-tasks—operation recognition, target localization, and intention classification—are uniformly optimized. For the operation target localization task, the information noise-contrastive estimation contrastive loss is introduced, where the interaction position features are taken as anchors, the feature distance to the ground-truth target node is pulled closer, and the feature representations of irrelevant controls are pushed farther, thereby enhancing localization accuracy:

$\begin{array}{r}L_{\text {target }}=-\log \frac{\exp \left(\operatorname{sim}\left(e_{\text {action }}, h_{\text {target }}\right) / \tau\right)}{\exp \left(\operatorname{sim}\left(e_{\text {action }}, h_{\text {target }}\right) / \tau\right)} +\sum_{v_{\text {neg }} \in V_{\text {neg }}} \exp \left(\operatorname{sim}\left(e_{\text {action }}, h_{\text {neg }}\right) / \tau\right)\end{array}$      (25)

where, $e_{\text {action}}$ denotes the interaction position feature vector; $h_{\text {target}}$ denotes the ground-truth target node features; sim denotes the cosine similarity function; $\tau$ denotes the temperature coefficient, employed to scale the feature distribution and regulate the gradient update magnitude; and $V_{\text {neg}}$ and $h_{\text {neg}}$ denote the set of negative sample nodes and the corresponding negative sample features, respectively.

To address the training bias issue caused by magnitude discrepancies among multi-subtask losses, a homoscedastic uncertainty mechanism is introduced, through which the weight coefficients of each loss function are adaptively and dynamically adjusted, replacing the manual configuration of fixed weights. The overall joint loss function of the model is defined as:

$L_{\text {intent }}=L_{\text {action }}+L_{\text {target }}+L_{\text {intent_cls }}$     (26)

$\begin{aligned} & L_{\text {intent }}=\frac{1}{2 \sigma_1^2} L_{\text {action }}+\frac{1}{2 \sigma_2^2} L_{\text {target }} +\frac{1}{2 \sigma_3^2} L_{\text {intent_cls }}+\log \sigma_1 \sigma_2 \sigma_3\end{aligned}$     (27)

where, $L_{\text {action}}, L_{\text {target}}$, and $L_{\text {intent_cls}}$ denote the operation recognition loss, the target localization contrastive loss, and the intention classification loss, respectively; and $\sigma_1, \sigma_2$, and $\sigma_3$ denote the learnable uncertainty parameters, through which the model adaptively balances the optimization weights of each subtask via backpropagation, thereby ensuring multi-task collaborative convergence and effectively enhancing the robustness and accuracy of behavioral intention reasoning in complex scenarios.

2.4 Digital literacy visual assessment and feedback system

Based on the outputs of the aforementioned interface perception, semantic modeling, and behavioral reasoning modules, an end-to-end digital literacy visual assessment and feedback system is constructed, enabling quantitative evaluation and visual instructional guidance of adult digital operational behaviors. Existing digital literacy assessment systems predominantly rely on macro-level questionnaire scoring and manual subjective judgment, lacking the capability for quantitative decomposition of micro-level operational behaviors and deficiency traceability, and thus cannot precisely characterize learners' interface cognitive proficiency or operational shortcomings. By leveraging the complete temporal operation sequences, interface semantic information, and behavioral intention data, a multi-dimensional quantitative assessment model is established, and visual feedback results are generated through image overlay rendering techniques, forming a complete technical closed loop from behavioral analysis, literacy scoring, to instructional intervention. The four-dimensional quantitative digital literacy assessment and visual feedback system is illustrated in Figure 4.

tu_pian_4.png

Figure 4. Four-dimensional quantitative digital literacy assessment and visual feedback system

By leveraging the comparison relationship between learners' actual operation sequences and standardized task operation paths, a four-dimensional integrated quantitative digital literacy indicator system is established, through which comprehensive assessment is accomplished across four dimensions: operational efficiency, interface comprehension, operational standardization, and error recovery capability. Operational efficiency quantifies the degree of conciseness in task completion by learners, where the efficiency of task execution is characterized through the deviation between the actual number of operation steps and the standard path step count. The computational formula is given as follows:

Efficiency $=1-\frac{\left|T-T^*\right|}{\max \left(T, T^*\right)}$     (28)

where, T and T* denote the length of the learner's actual operation sequence and the length of the task standard operation sequence, respectively. Interface comprehension, grounded in the functional semantic labels of the hierarchical screen semantic graph, measures the matching degree between the learner's operation targets and the standard semantic roles, directly reflecting the learner's cognitive proficiency regarding the interface functional structure. The computational formulation is as follows:

Understanding $=\frac{1}{T} \sum_{t=1}^T 1\left[r_{\text {target }, t}=r_{\text {target }, t}^*\right]$      (29)

where, $r_{\text {target,} t}$ and $r_{\text {target,} t} *$ denote the actual target semantic role and the standard semantic role for the single operation at time, respectively. Operational standardization quantifies the degree of alignment between the overall operation flow and the software's predefined operational logic, where statistical evaluation is performed through temporal alignment comparison between operation behaviors and the standard operation specification library:

$Normality =\frac{1}{T-1} \sum_{t=1}^{T-1} 1\left[\left(a_t, v_{\text {target}, t}\right)\right.$ conforms to the predefined specification]         (30)

Error recovery capability is employed to measure learners' autonomous correction and problem-solving abilities, where the ratio between the number of erroneous operations and the number of successfully corrected operations during the task process is statistically computed, with a small constant introduced to ensure computational stability:

Recovery $=\frac{N_{\text {corrected }}}{N_{\text {errors }}+\epsilon}$         (31)

where, $N_{\text {errors}}$ denotes the total number of detected erroneous operations, $N_{\text {corrected}}$ denotes the number of successfully corrected erroneous operations, and $\epsilon$ is set to 10^-6 to avoid computational anomalies caused by a zero denominator.

Based on the four individual evaluation metrics, a weighted fusion comprehensive digital literacy scoring model is constructed, where the weight for each dimension is assigned according to the competency composition characteristics of adult digital literacy, with particular emphasis placed on the central role of interface functionality cognition in literacy evaluation. The comprehensive score is computed as follows:

$\begin{gathered}{ Score}=\alpha \cdot { Efficiency }+\beta \cdot { Understanding } +\gamma \cdot { Normality }+\delta \cdot { Recovery }\end{gathered}$   (32)

The model weight parameters are set to $\alpha=0.25, \beta=0.30$, $\gamma=0.25$, and $\delta=0.20$, corresponding to the contribution weights of operational efficiency, interface comprehension, operational standardization, and error recovery capability, respectively. The highest weight is assigned to interface comprehension, which aligns with the evaluation logic in digital literacy assessment that prioritizes cognitive capability over operational form. The remaining weights are assigned in a balanced manner according to the literacy contributions of operational efficiency, operational standardization, and error correction capability, thereby ensuring the scientific rigor and rationality of the comprehensive score.

To enable the visual implementation of quantitative assessment results and instructional intervention, a screen visual feedback overlay technique is designed, through which abstract assessment data are transformed into intuitive on-screen visual prompts. All inefficient and erroneous operation samples are filtered by the system, and corresponding optimization prompts and standard operation guidelines are overlaid onto the original screen images. Multi-type visual elements are fused and overlaid through customized rendering functions, with the specific computational formulation given as:

$I_{\text {feedback }}=I_{\text {end }}+\sum_{t \in \mathrm{E}} \lambda_t \cdot {Render}\left(\right.type\left._t, b_{{target},t}\right)$       (33)

where, $I_{\text {end}}$ denotes the original final screen image; E denotes the index set corresponding to inefficient and erroneous operations; ${type}_t$ denotes the visual feedback type, which includes highlighting annotations, path guidance, and text prompts; $b_{\text {target,}t}$ denotes the interface element box corresponding to the anomalous operation; and $\lambda_t$ denotes the visual overlay transparency parameter, with a fixed value of 0.6, which preserves the original interface information in its entirety while clearly presenting the learner's operational deficiencies and optimization solutions. This mechanism enables precise identification of learners' individualized operational shortcomings, providing intuitive and implementable technical support for targeted training and fine-grained instruction in adult digital literacy.