Skill Spectrum Construction and Task Allocation Optimization Driven by Image Recognition for Digital Organization Management

2.1 Overall framework

A comprehensive image-driven framework for skill spectrum construction and task allocation optimization is proposed, targeting the human resource management requirements of operational roles within digitally transforming organizations. With employee operation videos serving as the sole raw input, the framework proceeds sequentially through three progressive stages—skill spectrum node quantification, skill-task association graph generation, and dynamic prediction with allocation optimization—culminating in the final outputs of optimal task allocation schedules and employee skill evolution forecasts. Throughout the entire pipeline, image representation and visual computation serve as the core technical paradigm, eliminating the need for manually predefined skill dimensions and task requirement labels and enabling end-to-end automated computation from raw video data to management decision outputs.

These three stages constitute a fully integrated technical chain with sequential dependencies. In the skill spectrum node quantification stage, discrete human keypoint trajectories are transformed into continuous multi-channel image representations through spatiotemporal heatmap encoding. A spatiotemporal graph convolutional network is then employed to extract fine-grained skill features, from which a structured set of skill spectrum nodes is constructed, thereby achieving unsupervised objective quantification of employee operational skills. In the skill-task association graph generation stage, three distinct scales of visual features are fused to establish a unified task representation. A cross-attention mechanism is leveraged to uncover latent mappings between skill features and task features, and singular value decomposition is subsequently applied to generate an interpretable bipartite association graph, through which the correspondences between skill dimensions and task requirements are automatically established. In the dynamic prediction and allocation optimization stage, temporal skill data are reconstructed as image sequences, and temporal contrastive learning is used to learn semantically consistent skill evolution representations. A temporal prediction network is then integrated to enable forward-looking estimation of future skill deficits. Finally, a bi-objective greedy policy is employed for task scheduling, thereby establishing a closed-loop optimization mechanism that jointly optimizes task execution efficiency and employee skill development.

2.2 Skill spectrum node quantification via spatiotemporal heatmap encoding

Skill spectrum node quantification serves as the foundational prerequisite for association modeling and task allocation, with the core objective of extracting objective, fine-grained, structured skill representations from unstructured operation videos. Modeling approaches that directly operate on keypoint coordinate sequences are inherently limited in their capacity to fully capture the spatiotemporal distribution characteristics of motion trajectories, and consequently provide only restricted utilization of fine-grained information such as motion rhythm and spatial coverage. The complete skill quantification pipeline based on spatiotemporal heatmap encoding is illustrated in Figure 1. First, a high-resolution pose estimation network is employed to extract two-dimensional coordinates of K human keypoints from each frame of the input video. The continuous motion trajectories are then transformed into standardized two-dimensional heatmap image representations. For the k-th keypoint, the heatmap trajectory image ${{\text{H}}_{\text{k}}}{{\text{R}}^{\text{T }\!\!\times\!\!\text{ L}}}$ is defined, with pixel values computed via a Gaussian kernel function:

${{\text{H}}_{\text{k}}}\text{(t,l)=exp}\left( \text{-}\frac{{{\text{(}{{\text{d}}_{\text{k,t}}}\text{-}{{\text{ }\!\!\mu\!\!\text{ }}_{\text{l}}}\text{)}}^{\text{2}}}}{\text{2}{{\text{ }\!\!\sigma\!\!\text{ }}^{\text{2}}}} \right)$                    (1)

where, T denotes the total number of video frames; L denotes the number of discretized bins for spatial positions or joint angles; ${{\text{d}}_{\text{k,t}}}$ denotes the spatial coordinate or computed joint angle value of the k-th keypoint at frame t; ${{\text{ }\!\!\mu\!\!\text{ }}_{\text{l}}}$ denotes the center value of the l-th bin; and $\text{ }\!\!\sigma\!\!\text{ }$ denotes the Gaussian kernel width. Through this encoding scheme, the one-dimensional temporal sequence of each individual keypoint is mapped to a two-dimensional spatiotemporal image, in which pixel intensity corresponds to the probability density of the keypoint's presence at the corresponding spatiotemporal location. The complete spatiotemporal distribution pattern of the motion is thereby preserved.

Figure 1. Flowchart of skill quantification based on spatiotemporal heatmap encoding

The heatmap trajectories of all keypoints are stacked along the channel dimension to form a multi-channel heatmap trajectory map $\text{H}{{\text{R}}^{\text{T }\!\!\times\!\!\text{ L }\!\!\times\!\!\text{ K}}}$, which serves as the unified input for subsequent feature extraction. To simultaneously incorporate the topological constraints of the human skeleton and the temporal evolution patterns of the motion, the multi-channel heatmap trajectory map is formulated as a spatiotemporal graph structure. Graph nodes correspond to individual human keypoints, and the edge set is partitioned into two categories: spatial edges and temporal edges. Spatial edges are constructed according to the natural connectivity of the human skeleton, capturing the spatial relationships among keypoints within the same frame. Temporal edges connect corresponding keypoints across adjacent frames, capturing the temporal evolution of each individual keypoint. On this basis, a spatiotemporal graph convolutional network is employed to perform multi-layer feature extraction. The node feature update rule for the l-th convolutional layer is given by:

${{\text{f}}^{\text{(l+1)}}}\text{(v)=}\mathop{\sum }_{\text{u}\in \text{N(v)}}\frac{\text{1}}{{{\text{Z}}_{\text{v,u}}}}{{\text{f}}^{\text{(l)}}}\text{(u)}\cdot {{\text{W}}^{\text{(l)}}}\text{(label(u,v))}$                    (2)

where, v denotes the current target node; $\text{N(v)}$ denotes the spatiotemporal neighborhood set of node v; ${{\text{Z}}_{\text{v,u}}}$ denotes the neighborhood normalization coefficient; and ${{\text{W}}^{\text{(l)}}}$ denotes the learnable weight matrix corresponding to the neighborhood type. The function $\text{label(u,v)}$ is used to partition the neighboring nodes into three categories—root nodes, centripetal nodes, and centrifugal nodes—to accommodate the physical characteristics of human skeletal motion. Through multi-layer spatiotemporal convolutions, spatial topological information and temporal contextual information are progressively aggregated, enabling the extraction of deep action features.

Following multi-layer convolutional operations and global average pooling, a D-dimensional skill feature vector ${{\text{s}}_{\text{i}}}$ is generated as the final output, corresponding to the comprehensive skill representation of the i-th employee. The individual dimensions of this vector implicitly correspond to skill attributes such as motion standardization, rhythm fluency, and operational precision. To further characterize the proportion of non-contributory action during the operation, a gradient-based redundant action ratio metric is introduced, through which the density of gradient discontinuities in the heatmap images is used to quantify the extent of corrective adjustments and non-contributory motion. The redundant action ratio is computed as:

$\text{Redundanc}{{\text{y}}_{\text{i}}}\text{=}\frac{\text{1}}{\text{T}\cdot \text{L}}\mathop{\sum }_{\text{t,l}}\|\nabla {{\text{H}}_{\text{i}}}\text{(t,l)}{{\|}_{\text{2}}}\cdot {{\text{1}}_{\|\nabla {{\text{H}}_{\text{i}}}\text{(t,l)}{{\|}_{\text{2}}}\text{ }\!\!\tau\!\!\text{ }}}$                   (3)

where, $\nabla {{\text{H}}_{\text{i}}}\text{(t,l)}$ denotes the gradient operator applied to the heatmap image of the i-th employee at location (t,l); $\text{ }\!\!\tau\!\!\text{ }$ denotes the gradient magnitude threshold; and the indicator function is used to select regions where the gradient magnitude exceeds the threshold. A higher value of this metric indicates a greater proportion of non-contributory corrective adjustments and redundant actions during the operation, corresponding to lower overall operational proficiency. The skill feature vectors of all employees, together with their corresponding redundancy metrics, collectively constitute the node set of the skill spectrum, completing the transformation from raw video to structured skill representations and providing a unified input foundation for subsequent skill-task association modeling.

2.3 Skill-task association graph generation via multi-scale cross-attention

Skill-task association modeling serves as the critical bridge connecting skill quantification results to task allocation decisions. Conventional matching approaches, which rely on manually predefined skill dimensions and task requirements, are incapable of capturing the substantial latent association information embedded within operational processes, and incur prohibitively high manual costs when adapting to new tasks. To construct a comprehensive visual representation of tasks, complementary features are extracted at three distinct scales—microscopic, mesoscopic, and macroscopic—enabling multi-dimensional characterization of operational details, semantic patterns, and temporal regularities. The complete multi-scale feature fusion and skill-task association modeling pipeline is illustrated in Figure 2. At the microscopic scale, the focus is placed on local operational details. Local binary patterns and the histogram of oriented gradients are employed to capture fine-grained information such as workpiece surface textures and operational edge trajectories. The local binary pattern descriptor, which encodes local texture structures through grayscale differences between neighboring pixels, is computed as:

$\text{LB}{{\text{P}}_{\text{P,R}}}\text{(}{{\text{x}}_{\text{c}}}\text{,}{{\text{y}}_{\text{c}}}\text{)=}\mathop{\sum }_{\text{p=0}}^{\text{P-1}}\text{sgn(}{{\text{g}}_{\text{p}}}\text{-}{{\text{g}}_{\text{c}}}\text{)}{{\text{2}}^{\text{p}}}\text{,sgn(x)=}\left\{ \begin{matrix}   \text{1,x}\ge \text{0}  \\   \text{0,x＜0}  \\ \end{matrix} \right.$                 (4)

where, ${{\text{g}}_{\text{c}}}$ denotes the grayscale value of the center pixel; ${{\text{g}}_{\text{p}}}~$denotes the grayscale value of the p-th sampling point within a circular neighborhood of radius R; and P denotes the total number of neighborhood sampling points. The two types of hand-crafted features are concatenated to form a ${{\text{d}}_{\text{1}}}$-dimensional microscopic feature vector, which precisely reflects task attributes related to operational precision and procedural detail.

Figure 2. Schematic diagram of multi-scale feature fusion and skill-task association modeling

At the mesoscopic and macroscopic scales, the task representation is supplemented from semantic and temporal perspectives, respectively. At the mesoscopic scale, convolutional layer features are extracted using a pre-trained ResNet-50 network, and global average pooling is applied to obtain a 2048-dimensional semantic feature vector, which characterizes mid-level semantic patterns of operational scenes and action phases, reflecting the task type and core workflow. At the macroscopic scale, a SlowFast network is employed to process 16-frame continuous video segments, fusing the spatial semantic information from the slow pathway with the motion temporal information from the fast pathway to obtain a 2048-dimensional global temporal feature vector, which characterizes the overall rhythm and motion evolution patterns of the task. For each task, the three types of features extracted from all sampled frames are averaged separately and then concatenated along the dimension to form a multi-scale fused feature vector of dimension ${{\text{d}}_{\text{1}}}\text{+4096}$. This representation comprehensively covers the multi-level visual attributes of the task—from local details to global processes—providing a unified feature foundation for subsequent association modeling.

To establish latent mappings between employee skill features and task features, a cross-attention mechanism is employed to compute the association weights between the two feature sets. Let the employee skill matrix be denoted as $\text{S}\in {{\text{R}}^{\text{N }\!\!\times\!\!\text{ D}}}$, where N is the total number of employees and D is the skill feature dimension, and let the task feature matrix be denoted as $\text{T}{{\text{R}}^{\text{J }\!\!\times\!\!\text{ (}{{\text{d}}_{\text{1}}}\text{+4096)}}}$, where J is the total number of tasks. Through learnable projection matrices, the two types of features are mapped into a common latent space, and the cross-attention weight matrix $\text{A}{{\text{R}}^{\text{N }\!\!\times\!\!\text{ J}}}$ is computed as:

${{\text{A}}_{\text{i,j}}}\text{=softma}{{\text{x}}_{\text{j}}}\left( \frac{\text{(}{{\text{s}}_{\text{i}}}{{\text{W}}_{\text{Q}}}\text{)(}{{\text{t}}_{\text{j}}}{{\text{W}}_{\text{K}}}{{\text{)}}^{\top }}}{\sqrt{{{\text{d}}_{\text{k}}}}} \right)$                   (5)

where, ${{\text{W}}_{\text{Q}}}{{\text{R}}^{\text{D }\!\!\times\!\!\text{ }{{\text{d}}_{\text{k}}}}}$ and ${{\text{W}}_{\text{K}}}{{\text{R}}^{\text{(}{{\text{d}}_{\text{1}}}\text{+4096) }\!\!\times\!\!\text{ }{{\text{d}}_{\text{k}}}}}$ denote the query projection matrix and the key projection matrix, respectively; ${{\text{d}}_{\text{k}}}$ denotes the latent space dimension; and $\sqrt{{{\text{d}}_{\text{k}}}}$ serves as the scaling factor to stabilize the training process. After row-wise softmax normalization, the value of ${{\text{A}}_{\text{i,j}}}$ corresponds to the skill-task compatibility between the i-th employee and the j-th task. Through the attention mechanism, the contributions of different feature dimensions are automatically weighted, enabling the discovery of latent associations that cannot be captured by manually defined rules.

To enhance the interpretability of the association results and reduce the subsequent computational complexity, singular value decomposition is applied to the cross-attention weight matrix to extract a low-rank latent cluster structure. The decomposition is formulated as:

$\text{A=U }\!\!\Sigma\!\!\text{ }{{\text{V}}^{\top }}$                  (6)

where, $\text{U}{{\text{R}}^{\text{N }\!\!\times\!\!\text{ R}}}$ denotes the latent skill cluster representation; $\text{V}{{\text{R}}^{\text{J }\!\!\times\!\!\text{ R}}}$ denotes the latent task cluster representation; R denotes the reduced latent space dimension satisfying $\text{R}\ll \text{min(N,J)}$; and $\text{ }\!\!\Sigma\!\!\text{ }$ denotes the diagonal singular value matrix. Based on these two latent representations, a skill-task bipartite association graph is constructed, in which nodes on the two sides correspond to skill clusters and task clusters, respectively, and edge weights are determined by the corresponding elements of ${{\text{U}}^{\top }}\text{V}$, reflecting the association strength between the two types of clusters. This graph provides an intuitive visualization of the intrinsic association structure between skill dimensions and task requirements. When combined with t-distributed stochastic neighbor embedding dimensionality reduction for visualization, the graph can further support management decision analysis, while the low-rank representation simultaneously provides an efficient computational foundation for the subsequent task allocation algorithm.

2.4 Skill prediction via temporal contrastive learning and bi-objective allocation optimization

Static task allocation approaches, which perform matching based solely on the current skill state without incorporating the temporal evolution patterns of employee skills, are incapable of supporting the coordinated optimization of human resource planning and skill development from a long-term perspective. The complete framework for temporal skill prediction and bi-objective task allocation is illustrated in Figure 3. To integrate the skill evolution process into a unified visual computing framework, the temporal skill representations are reconstructed into image formats, transforming the problem into a standard image sequence learning task. Let the employee skill matrix at the $\text{ }\!\!\tau\!\!\text{ }$-th time step be denoted as:

${{\text{S}}^{\text{( }\!\!\tau\!\!\text{ )}}}{{\text{R}}^{\text{N }\!\!\times\!\!\text{ D}}}$               (7)

where, N is the total number of employees, and D is the skill feature dimension. For the i-th employee, the skill vector is reshaped according to the correspondence between skill categories and sub-dimensions to form a two-dimensional skill image $\text{G}_{\text{i}}^{\text{( }\!\!\tau\!\!\text{ )}}{{\text{R}}^{{{\text{d}}_{\text{h}}}\text{ }\!\!\times\!\!\text{ }{{\text{d}}_{\text{w}}}}}$, satisfying ${{\text{d}}_{\text{h}}}\text{ }\!\!\times\!\!\text{ }{{\text{d}}_{\text{w}}}\text{=D}$, where the rows and columns correspond to different hierarchical levels of skill attributes, thereby preserving the topological associations among dimensions. After stacking along the temporal dimension, the skill evolution process of a single employee is represented as the image sequence $\text{ }\!\!\{\!\!\text{ G}_{\text{i}}^{\text{( }\!\!\tau\!\!\text{ )}}\text{ }\!\!\}\!\!\text{ }_{\text{ }\!\!\tau\!\!\text{ =1}}^{\text{T}}$, where $\text{T}$ is the total number of time steps. This representation format is directly compatible with established image sequence prediction and self-supervised learning methods.

Figure 3. Framework diagram of temporal skill prediction and bi-objective task allocation

Based on the current skill state and the future skill deficit predictions, a bi-objective greedy task allocation strategy is constructed to achieve the joint optimization of task execution efficiency and employee skill development. Let the set of pending tasks be denoted as ${{\text{J}}_{\text{pending}}}$, and let the skill requirement vector ${{\text{r}}_{\text{j}}}$ for the j-th task be obtained through mapping from the previously constructed skill-task association graph. The comprehensive matching score between the employee and the task is defined as:

$\text{Score(i,j)=s}_{\text{i}}^{\top }{{\text{r}}_{\text{j}}}\text{+ }\!\!\lambda\!\!\text{ }\mathop{\sum }_{\text{d:}{{\text{ }\!\!\delta\!\!\text{ }}_{\text{i,d}}}＞\text{0}}{{\text{ }\!\!\delta\!\!\text{ }}_{\text{i,d}}}\cdot {{\text{r}}_{\text{j,d}}}$                (10)

where, the first term denotes the current skill-task compatibility, computed as the inner product between the employee's current skill vector and the task requirement, reflecting the immediate efficiency of task execution. The second term denotes the skill development gain, which is accumulated only over skill dimensions in which the employee exhibits deficits, reflecting the developmental value of task execution for the employee's deficient skill dimensions. $\text{ }\!\!\lambda\!\!\text{ }$ denotes a balancing hyperparameter used to adjust the weight between immediate efficiency and long-term skill development. The allocation process employs a greedy iterative strategy, in which the unassigned employee-task pair with the highest comprehensive score is selected for matching in each round, until all pending tasks have been assigned. This strategy simultaneously ensures task delivery efficiency while directionally reinforcing employees' skill deficits. Through this mechanism, a closed-loop management system is established—encompassing data acquisition, skill modeling, predictive allocation, and capability development—that satisfies the long-term human resource operational requirements of digitally transforming organizations.