Image Recognition-Based Automatic Detection of Rule Violations in Soccer Match Videos

2.1 Overview of the proposed framework

A three-module serial-feedback coupled multi-view framework for automatic soccer rule violation detection is proposed. Synchronized multi-view match video sequences are taken as the input, and three types of outputs are generated in an end-to-end manner, including foul category classification, severity assessment, and interpretable energy evolution curves. The overall architecture of the proposed framework is illustrated in Figure 1. During forward inference, the framework sequentially passes through three core modules: a rule-guided spatiotemporal graph convolutional network, a dynamically weighted dual-branch spatiotemporal query fusion network, and a geometric potential field-based fine-grained assessment module. In the first module, competition rules are formalized as computable semantic constraints and embedded into the feature aggregation process of the spatiotemporal graph convolutional network, thereby generating frame-level semantic representations enriched with rule-based prior knowledge. In the second module, adaptive fusion of multi-view features is performed through dual-branch learnable query embeddings, and preliminary predictions of foul category and severity assessment are produced. In the third module, a multi-component geometric potential field is constructed based on high-precision human body keypoints, through which quantitative severity assessment and interpretable visual explanations are generated.

Figure 1. Operational mechanism of the rule-guided spatiotemporal graph convolutional network

Unlike conventional sequential detection architectures, a closed-loop optimization mechanism is established through three feedback pathways connecting the three modules. Rule-constrained features are utilized to provide initialization biases for the query vectors, classification predictions are employed to dynamically adjust the weights of the potential energy field, and backward calibration is triggered whenever inconsistencies arise between the energy-based quantitative assessment and the classification results. Through this design, bidirectional interaction among low-level visual representations, mid-level semantic representations, and high-level rule-based knowledge is achieved. Consequently, detection accuracy is improved while the consistency and interpretability of the final decision are simultaneously ensured, thereby satisfying the stringent requirements for both accuracy and reliability in professional sports officiating.

2.2 Rule-guided spatiotemporal graph convolutional network

The objective of this module is to explicitly incorporate domain knowledge derived from competition rules into the visual feature aggregation process, thereby providing semantic guidance for rule violation detection. The operational mechanism of the rule-guided spatiotemporal graph convolutional network is illustrated in Figure 1. Human body keypoint coordinates of all players and the soccer ball location within each frame are first extracted using a pretrained pose estimation network and an object detection network, respectively, and are subsequently used as the fundamental inputs for rule quantification. The parameters of the pretrained models are kept fixed throughout the entire training process. Competition rules are then decomposed into a triplet representation consisting of visual preconditions, threshold conditions, and decision outcomes. Consequently, officiating criteria originally expressed in natural language are transformed into frame-wise computable semantic propositions. A Boolean rule trigger is defined as:

$I_{k, t}(i, j)=1\left[f_{{visual }}\left(p_{i, t}^{(j)}, p_{i, t}^{(l)}, \mathrm{q}_t\right) \geq \theta_k\right]$          (1)

where, k denotes the rule index, t denotes the frame index, and i and j denote the player-pair indices. The function f_visual represents the geometric computation function, which takes the player keypoint coordinates $p$ and the soccer ball position $q_t$ as inputs and produces quantitative measurements, including spatial distance, joint angle, and relative position. The parameter $\theta_k$ denotes the decision threshold associated with the k-th rule, and 1[ ] denotes the indicator function, which returns 1 when the specified condition is satisfied and 0 otherwise. The geometric computation function can accommodate multiple rule criteria, including inter-player distance, limb contact distance, leg swing amplitude, and ball possession determination. In this manner, unstructured competition rule text is converted into structured binary trigger signals, thereby providing computable semantic constraints for subsequent feature aggregation.

A multi-player spatiotemporal interaction graph is subsequently constructed based on the rule trigger signals. Human body keypoints of all players within each frame are treated as graph nodes, whereas the edge set is partitioned into three categories. Spatial edges correspond to the natural skeletal connections of an individual player and encode topological constraints of human body pose. Interaction edges connect keypoint pairs belonging to different players whose spatial distance is smaller than a predefined threshold and encode inter-player interaction relationships. Temporal edges connect the corresponding keypoints across adjacent frames and encode the temporal evolution of actions. The initial node representations are obtained by concatenating the local visual representations extracted by the backbone network with the corresponding keypoint coordinate features. Unlike the standard graph attention network, in which attention weights are computed solely from visual representations, a rule constraint term is incorporated into the attention computation in the proposed module, thereby enabling domain knowledge to directly modulate information propagation. The node feature update rule for the l-th graph convolutional layer is formulated as:

$h_v^{(l+1)}=\sigma\left(W_0^{(l)} h_v^{(l)}+\sum_{u \in N(v)} \alpha_{v u}^{(l)} \cdot W_1^{(l)} h_u^{(l)}\right)$          (2)

where, $h_v^{(l)}$ denotes the feature vector of node v at the l-th layer; $N(v)$ denotes the spatiotemporal neighborhood of node v; $W_0^{(l)}$ and $W_1^{(l)}$ denote the learnable weight matrices; $\sigma$ denotes the nonlinear activation function; and $\alpha_{v u}^{(l)}$ denotes the attention weight assigned to neighboring node u with respect to node v.

The attention weights are jointly determined by visual similarity and rule constraints and are computed as:

$\alpha_{v u}^{(l)}=\frac{\exp \left(\beta_{v u}^{(l)}+\gamma \sum_{k=1}^K \lambda_k I_{k, t}(v, u)\right)}{\sum_{w \in N(v)} \exp \left(\beta_{v w}^{(l)}+\gamma \sum_{k=1}^K \lambda_k I_{k, t}(v, w)\right)}$          (3)

where, $\beta_{v u}^{(l)}$ denotes the baseline attention score computed from the original visual representations and reflects the feature similarity between nodes. The parameter $\gamma$ denotes the global scaling coefficient for rule constraints, whereas  denotes the weight associated with the k-th competition rule. Both parameters are automatically optimized during model training. The term $I_{k, t}(v, u)$ represents the rule trigger associated with the corresponding node pair. When the interaction between two nodes satisfies the rule activation condition, the attention weight of the corresponding edge is positively enhanced, thereby increasing the contribution of the associated interaction features during feature aggregation. Neighborhoods in which no rule is activated retain their baseline attention weights, resulting in a relative suppression effect. Through this mechanism, the feature aggregation process is guided to focus preferentially on interaction patterns that are semantically consistent with competition rules, thereby effectively improving both feature discriminability and semantic specificity. Following multiple layers of spatiotemporal graph convolution and global average pooling, a frame-level semantic representation enriched with rule-based prior knowledge, denoted as $F_t^{ {rule }}$, is generated and subsequently propagated to the downstream modules to support foul classification and severity assessment.

2.3 Dynamically weighted dual-branch spatiotemporal query fusion network

To satisfy the requirements of spatiotemporal feature aggregation for multi-view match videos, spatiotemporal feature encoding is first performed independently for each view. A video vision Transformer network employing factorized spatiotemporal attention is adopted as the backbone for single-view feature extraction. Synchronized video sequences from each camera view are encoded separately to obtain the initial spatiotemporal feature sequences. To preserve both the spatial positional attributes and the identity information associated with each view, a learnable view embedding vector is added to the feature representation of each view, yielding the view-aware feature representation:

$Z_v^{{(view) }}=Z_v+e_v^{ {view }}$          (4)

where, $Z_v$ denotes the original spatiotemporal feature representation extracted from the v-th view, and $Z_v^{(view)}$ denotes the learnable embedding vector corresponding to the v-th camera view. Conventional multi-view fusion methods have generally relied on static fusion strategies, such as direct feature concatenation or fixed-weight feature aggregation. Consequently, fusion weights cannot be adaptively adjusted according to the occlusion status of the action region or the visual quality of individual viewpoints. In densely contested match scenarios, the fusion process is therefore susceptible to interference from low-quality views, thereby limiting further improvements in detection accuracy. The architecture of the dynamically weighted dual-branch spatiotemporal query fusion network is illustrated in Figure 2.

Figure 2. Architecture of the dynamically weighted dual-branch spatiotemporal query fusion network

To emulate the decision-making process by which human referees verify events from multiple viewpoints and preferentially select the most informative observations, a dual-branch spatiotemporal query decoding architecture is constructed. Two categories of learnable query vectors, corresponding to action category and severity assessment, are designed to address the two subtasks of foul classification and severity evaluation, respectively. Adaptive aggregation of multi-view features is achieved through a cross-attention mechanism. During the decoding process, a dynamic view weighting mechanism is introduced. At each decoding layer, the fusion weight associated with each view is dynamically computed according to the matching degree between the current query state and the corresponding view representation:

$Score _v=\frac{\exp \left(s_v^{(l)} \cdot {Attention}\left(Q^{ {act }(l-1)}, Z_v^{ {(view) }}\right)\right)}{\sum_{v^{\prime}=1}^V \exp \left(s_{v^{\prime}}^{(l)} \cdot {Attention}\left(Q^{ {act }(l-1)}, Z_{v^{\prime}}^{{(view })}\right)\right)}$          (5)

where, l denotes the number of the decoder layers, V denotes the total number of camera views, $Q^{a c t(l-1)}$ denotes the action query vector generated by the previous layer, $Attention$(·) denotes the cross-attention matching score between the query and the view representation, and $s_v^{(l)}$ denotes the learnable confidence factor associated with the v-th view at the l-th layer. After Softmax normalization, $Score _v$ represents the fusion weight assigned to the v-th view during the current decoding step. When the target action region is occluded in a particular view, the matching score between the query and the corresponding view representation is reduced, causing the associated fusion weight to decrease automatically. Conversely, views providing unobstructed observations receive correspondingly higher weights, thereby enabling adaptive adjustments of fusion weights. Following multiple decoding iterations, preliminary probability distributions of foul category classification and severity classification probabilities are generated.

To achieve semantic continuity between the preceding rule-guided module and the current module, a feature bias coupling strategy is adopted to incorporate rule-constrained semantic information into the initial decoder state. Unlike strongly coupled approaches that directly replace the query representations, the proposed design incorporates rule-based representations as a bias term added to the learnable query vectors. The initialization of the action query is formulated as:

$Q^{{act }(0)}=Q_{ {learnable }}^{ {act }}+M L P\left(f_t^{ {rule }}\right)$          (6)

where, $Q_{ {learnable }}^{ {act }}$ denotes the randomly initialized learnable action query vector, and $f_t^{ {rule}}$ denotes the frame-level semantic representation enriched with rule-based constraints generated by the preceding module. The latter is projected into the same feature space as the query through a multilayer perceptron. Through this design, the initial decoder state is endowed with prior semantic knowledge derived from competition rules, thereby guiding the attention mechanism to focus more rapidly on player interaction regions that are semantically consistent with rule violations and consequently accelerating model convergence. Furthermore, under few-shot training scenarios, the rule-based prior knowledge provides effective semantic regularization, thereby reducing the dependence of the model on large volumes of annotated data while improving the generalization capability for rare foul categories. The severity query is initialized using the same bias strategy, thereby ensuring that both types of tasks are consistently guided by rule-based prior knowledge.

2.4 Geometric potential field-driven fine-grained assessment and interpretation

To satisfy the requirements for quantitative severity assessment and interpretability of rule-violating actions, multi-scale keypoint refinement is first performed within the candidate action regions identified by the second module. An improved You Only Look Once version 8 Pose (YOLOv8-Pose) network is employed to extract the 17 human body keypoints of the interacting players. A reparameterized multi-scale feature fusion module is incorporated to improve the recall of small-scale limb keypoints. In the reduced candidate regions, high-precision detection is conducted, ensuring positioning accuracy while simultaneously reducing the computational cost. Instantaneous velocity vectors of all keypoints are subsequently computed through frame-to-frame differencing of keypoint coordinates across consecutive frames. These velocity vectors are then combined with the corresponding keypoint detection confidence scores to generate weighted velocity estimates, which serve as the fundamental kinematic data for the subsequent calculation of the geometric potential field. The construction process of the geometric potential field and the closed-loop calibration procedure are illustrated in Figure 3.

Figure 3. Workflow of geometric potential field construction and the closed-loop calibration

To transform the abstract notion of foul hazardousness into quantifiable and interpretable physical indicators, a geometric potential field model consisting of three potential energy components is constructed. The model characterizes the severity of rule violations from three complementary perspectives: action intensity, physical contact intensity, and hazardous posture severity. The motion potential is designed to characterize the overall intensity of a player's movement and is computed as a weighted aggregation of the velocities of all detected keypoints. Higher values indicate larger motion amplitudes and stronger physical impacts. The motion potential is defined as:

$E_{k i n, i, t}=\frac{1}{2} m_i \sum_{j=1}^{17} c_{i, t}^{(j)} \cdot\left\|v_{i, t}^{(j)}\right\|_2^2$         (7)

where, $m_i$ denotes the equivalent mass coefficient of player i; c^(j)_i,t denotes the detection confidence of the j-th keypoint; and v^(j)_i,t denotes the instantaneous velocity vector of the corresponding keypoint. The contact potential characterizes the intensity of physical contact between two players. It is computed by mapping the spatial distances between corresponding keypoint pairs through a Gaussian kernel, while joint-pair weighting coefficients are introduced to distinguish the hazardousness associated with different contact locations. The contact potential is formulated as:

$E_{ {contact }, i, i^{\prime}, t}=\sum_{j=1}^{17} \sum_{j^{\prime}=1}^{17} c_{i, t}^{(j)} c_{i^{\prime}, t}^{\left(j^{\prime}\right)} \cdot w_{j j^{\prime}} \cdot \exp \left(-\frac{\left\|k_{i, t}^{(j)}-k_{i^{\prime}, t}^{\left(j^{\prime}\right)}\right\|_2^2}{2 \sigma_{ {contact }}^2}\right)$         (8)

where, k^(j)_i,t denotes the coordinates of the j-th keypoint of player i; $w_{j j^{\prime}}$ denotes the hazardousness weighting coefficient associated with the corresponding joint pair; and $\sigma_{contact}$ denotes the Gaussian bandwidth governing the contact distance. The hazardous posture potential characterizes the degree of postural rule violation. It is computed from the deviation between critical joint angles and their corresponding reference values under normal postures. Potential energy is accumulated only when the angular deviation exceeds a predefined safety threshold. The hazardous posture potential is expressed as

$E_{ {pose }, i, t}=\sum_{p \in P} \phi_p \cdot \max \left(0, \frac{\left|\theta_p\left(k_{i, t}\right)-\theta_p^{ {norm }}\right|}{\theta_p^{ {thr }}}-1\right)$         (9)

where, $P$ denotes the set of critical joints, $\theta_p\left(k_{i, t}\right)$ denotes the instantaneous angle of the p-th joint, $\theta_p^{norm}$ denotes the corresponding joint reference value under a normal posture, $\theta_p^{ {thr}}$ denotes the predefined safety threshold, and $\phi_p$ denotes the weighting coefficient associated with the joint.

The three potential energy components are subsequently fused to obtain the frame-level total potential energy, thereby forming an energy evolution sequence spanning the entire action. The weighting coefficients of the three energy components are not treated as fixed constants. Instead, they are dynamically generated from the severity query vector produced by the second module through a lightweight mapping network. Consequently, different foul categories are associated with distinct energy-component weight allocations. For example, the weight assigned to the contact potential is increased for charging fouls, whereas the contribution of the hazardous posture potential is emphasized for tripping fouls. In this manner, high-level classification priors are allowed to adaptively guide the underlying quantitative assessment process. The total potential energy is computed as:

$\begin{gathered}E_{{total }, t}=\alpha_{ {kin}} \sum_i E_{{kin }, i, t}+\alpha_{{contact }} \sum_{i<i^{\prime}} E_{ {contact }, i, i^{\prime}, t}  +\alpha_{ {pose }} \sum_i E_{ {pose }, i, t}\end{gathered}$         (10)

where, $\alpha_{ {kin}}, \alpha_{{contact}}$, and $\alpha_{{pose}}$ denote the dynamically generated weighting coefficients for the motion potential, contact potential, and hazardous posture potential, respectively. Three key statistical features are extracted from the continuous energy sequence. The sequence peak reflects the intensity of the most hazardous moment during the action, the energy variation rate at the peak characterizes the abruptness of the movement, and the temporal integral represents the cumulative influence of the rule-violating action throughout its duration. These three features are subsequently fed into a multilayer perceptron, and the final severity assessment is obtained as:

$\begin{gathered}s=\arg \max _{s^{\prime}}\left[M L P_{ {sev }}\left(\max _t E_{ {total }, t},\left.\frac{d E_{ {total }}}{d t}\right|_{\max }, \int E_{ {total }, t} d t\right)\right]\end{gathered}$         (11)

Through this mechanism, not only is a discrete severity level produced, but the complete frame-level energy curve is also retained as visual evidence for officiating decisions, thereby achieving a unified framework for quantitative severity assessment and decision interpretability.

The proposed module and the preceding classification module are further integrated through a bidirectional calibration mechanism, forming a closed-loop collaborative framework. Along the forward propagation pathway, the foul category and the preliminary severity assessment predicted by the second module are utilized to dynamically adjust both the weighting coefficients of the total potential energy components and the hazardousness coefficients associated with individual joints. Consequently, the quantitative assessment process is adaptively aligned with the officiating criteria corresponding to the current action type, thereby improving the accuracy of severity assessment. Along the backward calibration pathway, a classification–energy consistency verification criterion is established. When a significant confidence discrepancy is detected between the statistical characteristics of the energy evolution sequence and the classification results, the meta-calibration module is activated to inversely map the energy features into feature bias representations. These feature biases are subsequently injected into the decoder layers of the second module to refine the query vectors and the attention distribution, after which the classification inference is performed again. Through this bidirectional feedback mechanism, local misclassifications produced by individual modules can be effectively corrected, thereby improving both the consistency and the robustness of detection results across diverse scenarios. As a result, a complete closed-loop rule violation detection framework is established, spanning visual representation extraction, rule-guided classification, and physics-based quantitative assessment.

2.5 Joint training strategy and loss functions

A two-stage joint training strategy is adopted to ensure both efficient model convergence and effective collaboration among the proposed modules. The keypoint detection module is first pretrained on a publicly available human pose estimation dataset. During the subsequent end-to-end training stage, the parameters of this module are kept fixed, whereas the rule-guided spatiotemporal graph convolutional network, the dual-branch spatiotemporal query fusion network, and the geometric potential field-based assessment module are jointly fine-tuned in an end-to-end manner. Through this strategy, semantic consistency across the feature spaces of different modules is maintained while collaborative optimization is facilitated. The overall training objective is formulated as a weighted combination of four loss terms:

$L_{ {total }}=L_{ {rule }}+L_{ {query }}+\lambda_{ {pose }} L_{ {pose }}+\lambda_{ {energy }} L_{ {energy }}$         (12)

where, $L_{ {rule}}$ is implemented using the focal loss formulation and is applied to the interaction-node classification task in the rule-guided module. By introducing the modulation factor, the loss contribution of the abundant normal interaction samples is reduced, enabling the model to focus preferentially on the relatively few challenging rule-violation samples and thereby accommodating the severe class imbalance characteristic of soccer match scenarios, in which rule-violating actions constitute only a small proportion of all interactions. The loss term $L_{ {query}}$ consists of two components. The primary component is the cross-entropy loss for action classification and severity assessment, which is employed to optimize the classification accuracy of the dual-branch decoder. An additional L1 sparsity regularization term is imposed on the view fusion weights to constrain their distribution, encouraging the model to preferentially select a limited number of informative viewpoints while suppressing the contribution of low-quality views. The loss term $L_{pose}$ is formulated as an object keypoint similarity loss and is used to fine-tune keypoint refinement within the candidate regions. This formulation accommodates pose estimation for players at different spatial scales while ensuring the input accuracy of the potential field computation. Finally, $L_{energy}$ is formulated as an ordinal regression loss based on the cumulative link model and is applied to the severity assessment task. By explicitly exploiting the ordinal relationship among severity levels, this loss function more accurately reflects the hierarchical nature of officiating decisions than conventional classification losses.

All experiments were conducted using a single NVIDIA A100 graphics processing unit under the PyTorch deep learning framework. The AdamW optimizer was employed with a weight decay coefficient of 1 × 10⁻⁴. The initial learning rate was set to 3 × 10⁻⁵, and a linear warm-up strategy followed by cosine annealing scheduling was adopted. The model was trained for 30 epochs with a batch size of 8. The loss balancing coefficients $\lambda_{ {pose }}$ and $\lambda_{ {energy }}$ were set to 0.5 and 0.8, respectively, to balance the magnitudes of different task-specific losses. This configuration prevented any individual task from dominating the gradient update process, thereby ensuring the stability of multi-task training. During training, gradient clipping was simultaneously applied by constraining the gradient norm to a maximum of 1.0, thereby suppressing potential gradient fluctuations introduced by the closed-loop feedback pathways. During inference, the backward calibration module employed a confidence discrepancy threshold of 0.6. Re-inference was triggered only when the confidence discrepancy between the classification results and the energy-based quantitative assessment exceeded this threshold. Consequently, the consistency of the final decision was maintained while the additional computational overhead introduced by the re-inference process was effectively controlled.

Based on the comprehensive experimental results, the proposed framework achieved the expected improvements in detection accuracy, officiating consistency, and generalization capability. Nevertheless, three major limitations remain. First, the reliability of the proposed framework remains limited under scenarios involving severe full-body occlusion. When multiple players engage in dense physical interactions and the critical limbs required for rule violation assessment are completely occluded, both the accuracy and the recall of human pose estimation are substantially degraded. The resulting errors are subsequently propagated to the rule triggering and geometric potential field computation stages, thereby increasing the likelihood of missed detections and false predictions. Second, the adaptability of the proposed framework to extremely rare foul categories remains constrained. The current rule formalization strategy primarily covers high-frequency rule-violating actions encountered in competitive matches. For uncommon or highly infrequent foul categories, both the available rule-based priors and the corresponding training samples remain limited. Consequently, a certain degree of degradation in the model's generalization capability is still observed. Third, the depth ambiguity introduced by two-dimensional image projection cannot be completely eliminated. All geometric measurements are currently performed on image-plane coordinates without access to genuine three-dimensional spatial information. As a result, inherent projection errors are introduced into the estimation of player-to-player distances and limb contact distances, thereby limiting the absolute accuracy of both contact potential and hazardous posture assessment.

To address the aforementioned limitations, future research can be directed toward targeted improvements in several key technical aspects. First, multi-view three-dimensional human reconstruction can be integrated to recover the three-dimensional body pose and spatial positions of players through stereo correspondence across synchronized multi-view video streams. By replacing two-dimensional keypoints with three-dimensional pose representations as the input for geometric computation, projection errors can be fundamentally eliminated while the robustness of human pose estimation under severe occlusion can be further improved. Second, audio-modal features can be incorporated to assist contact assessment. A multimodal detection framework can be established by integrating collision sounds, body contact sounds, and other synchronously acquired acoustic signals, thereby compensating for the inherent limitations of purely vision-based approaches in contact determination and further improving the confidence of the detection results.

From a broader perspective, the proposed technical framework possesses considerable potential for extension to a wider range of competitive sports. The rule-based knowledge encoding, dynamic multi-view fusion, and physics-based quantitative interpretation strategies validated in the soccer scenario can be readily adapted to rule violation detection tasks in other multiplayer competitive sports, including basketball, handball, and hockey. Such adaptation can be accomplished by replacing the corresponding competition rule base and the associated quantitative parameter thresholds for each sport. In the long term, the integration of symbolic knowledge guidance with data-driven learning has the potential to facilitate the establishment of a unified intelligent officiating framework applicable to multiple sports, thereby providing sustained technical support for fair officiating and the intelligent advancement of competitive sports.