Adaptive Scenario Control in Intelligent Training Simulators Using Fuzzy Logic: Integration of Eye-Tracking and Performance Error Analysis

Intelligent training simulators (ITS) can be viewed as human-in-the-loop information systems that continuously acquire multimodal behavioral data, transform these data into state estimates, and apply decision rules to regulate task conditions. In this setting, adaptation is not limited to final scoring. Instead, the system performs a closed-loop information-processing function in which sensing, integration, estimation, and control are linked in real time under uncertainty and inter-user variability. Immersive simulators support this approach because scenario parameters such as visibility, environmental disturbances, and target behavior can be adjusted in real time while preserving the task structure.

Eye tracking provides a non-invasive stream of behavioral information that can be collected continuously during interaction. Recent reviews describe its use in immersive and VR settings for assessing workload, attention, and fatigue [1-5]. Fixation duration and saccade dynamics capture aspects of information processing and visual search [6-8]. Blink-related metrics and PERCLOS are often associated with vigilance and fatigue and have been validated in applied monitoring contexts [9-13].

A second issue concerns the decision layer. Many adaptive training approaches rely on black-box machine learning models, which can reduce transparency and complicate validation in contexts where interpretability and reliability are especially important. Interpretable rule-based decision models remain relevant when the system must expose its reasoning and allow expert adjustment. Fuzzy inference systems provide a framework to integrate heterogeneous indicators, represent uncertainty, and encode expert knowledge through linguistic rules [14-16].

In this work, we propose a real-time fuzzy scenario-control loop that combines robustly normalized eye-tracking features with a simulator-derived performance error score. The features are aggregated in sliding windows and processed by a Mamdani-type fuzzy model to estimate cognitive load, stress level, and fatigue. The subsystem combines eye-tracking features aggregated in sliding windows with a simulator-derived performance error score. A Mamdani-type fuzzy model outputs estimates of cognitive load, stress level, and fatigue, which are then used by a scenario manager to regulate difficulty within predefined limits. The approach is evaluated in a controlled study with 20 participants by comparing fixed-scenario training with adaptive fuzzy-driven training. The main contributions of this work are threefold. First, we introduce a task-oriented fusion strategy that combines personalized, robustly normalized eye-tracking indicators with a simulator-derived integrated error score on a common decision scale. Second, we define an interpretable scenario-control logic in which the inferred trainee states are translated into bounded difficulty adaptation through explicit rules and stability constraints. Third, the experimental validation is designed to assess the practical value of the proposed adaptive control loop, specifically whether it can stabilize inferred user-state trajectories and maintain task performance more effectively than a fixed training policy.

From an information-systems perspective, the proposed framework can be interpreted as a real-time human-in-the-loop architecture for multimodal data integration, latent state estimation, and adaptive decision management in simulator-based training.

3.1 Overall architecture concept

The proposed ITS is implemented as a closed-loop training system in which the trainee’s actions in a physical training room are continuously mapped to the evolution of a virtual Unity scenario and used for adaptive difficulty control. Training is conducted in a dedicated room where the simulator view is projected onto a screen surface, as shown in Figure 1, while the trainee holds a training mock-up that serves as the input device for aiming and triggering a shot.

Figure 1. Hardware layout of the training room with infrared eye tracking

The system operates on two synchronized data streams. The first stream is produced by the eye tracking subsystem and captures the dynamics of the trainee’s visual behavior. In real-time, oculomotor measures including fixations, saccades, blinks, and PERCLOS are computed. They are then aggregated in sliding windows and normalized relative to an individualized baseline. The second stream is generated by the Unity simulator and reflects task execution within the current scenario. It includes shot timestamps, grenade impact events, temporal deviations in user response, and attempt outcomes. These data are used to compute an integral error indicator, ErrScore, which serves as a compact behavioral proxy of performance.

Psychophysiological and behavioral information is fused within a joint processing module, where a unified feature representation of the user’s state is formed. This representation is then provided to a Mamdani-type fuzzy subsystem that outputs interpretable estimates of learner-relevant states. The inferred states include cognitive load, stress level, and fatigue. Based on these estimates, the scenario manager adjusts difficulty parameters within predefined bounds. It increases or decreases scenario demands as a function of the current state and execution quality. The architecture therefore supports continuous feedback and enables an adaptive learning trajectory, while preserving decision interpretability through explicit fuzzy rules.

3.2 Architecture of the fuzzy subsystem

Figure 2 illustrates the ITS architecture. The central element is the user, the trainee, who performs tasks within a selected scenario and continuously interacts with the ITS. During operation, the simulator records user actions, registers outcomes, and generates execution events, including errors.

Figure 2. Intelligent training simulator (ITS) architecture with fuzzy subsystem

In the diagram, the Scenario corresponds to a specific training mission in which the user practices target skills. The ITS initiates the scenario, supervises its execution, and receives behavioral data from the user. For training adaptation, an oculomotor metrics logging module provides measures such as saccade rate, blink rate, fixation duration, and PERCLOS. These features reflect the temporal dynamics of attention, workload, and fatigue, and are subsequently processed in the interpretation module. In addition, simulator-derived error events are used that is events extracted from ITS logs when the user acts prematurely or too late, skips steps, or responds in an untimely manner.

Eye tracking data and error events are fed into the Eye Tracking and Error Analysis block, where joint analysis is performed. This includes window-based aggregation, computation of derived indices, and formation of a feature representation of the user’s state. The output of this block is a set of diagnostic features that are forwarded to the Fuzzy Decision Block.

The Fuzzy Decision Block implements a fuzzy decision-making logic: based on the aggregated features, the system infers a user state relevant to the learning process, including cognitive load, attention, and fatigue. The inferred user state is then passed to the Scenario Manager, which is responsible for applying control actions to the scenario component of the training simulator.

Within the proposed model, the Scenario Manager performs one of the following actions:

1. Generation of a new scenario and assignment of a difficulty level.
2. Regularization of the current scenario parameters.
3. No action, if the system determines that the current training mode is optimal.

After the decision is made, the Scenario Manager applies the selected control action to the Scenario for subsequent adaptation.

3.3 Data acquisition layer

Data acquisition is performed in a training room using the simulator’s integrated hardware and software configuration in Figure 3. The Unity visual scene is displayed on a screen via a projector, while the control PC simultaneously runs the simulation and logs all events. The eye tracking subsystem is built around a monochrome IR camera operating at 120 FPS (1080p) with 850 nm IR illumination. This enables robust eye capture that is largely independent of ambient lighting. From the video stream, the following oculomotor measures are computed in real-time: saccade rate, fixation duration, blink rate, and PERCLOS. All measurements are stored as time series with precise timestamps to support subsequent aggregation into windows ${{W}_{k}}$ and alignment with simulator events.

Figure 3. Oculomotor data acquisition setup

An additional aiming module is based on a 980 nm IR laser mounted on the weapon mock up and a monochrome 60 FPS (4K) IR camera that tracks the laser spot on the projection surface. On the processing side, the spot is converted into normalized coordinates $\left( U,~V \right)\in \left[ 0,~1 \right]$ and transmitted via UDP. Unity interprets these coordinates as the current on-screen aiming point. The shot event is generated through a hardware trigger. A microcontroller detects the press and transmits a signal to Unity, where it is interpreted as the shot initiation time. This provides a direct link between the trainee’s physical action and the virtual process.

After a shot, Unity spawns a grenade that follows a realistic ballistic model and reaches the target object with a delay determined by flight dynamics. The impact event is registered by Unity’s collision component. A direct hit is defined as the grenade colliding with the armored target collider. Collision with any other scene object is treated as a miss. In parallel, the system logs the moment a target appears on screen and computes the user’s reaction time deviations. This enables discrimination between premature and delayed shots relative to the target detection moment.

To synchronize all data sources, a unified timeline is maintained on the control PC. Oculomotor measurements and simulator events are aligned into a common observation structure using sliding windows ${{W}_{k}}$ with duration $T=10$ s. On this basis, ErrScore is computed and fuzzy inference is subsequently performed.

The error analysis module evaluates task execution quality using Unity simulator events and shot telemetry from the RPG-7V2 training mock-up. The system records the shot outcome, hit or miss, and the temporal deviation of the user response, rushing or being late. All metrics are computed in sliding windows and synchronized by timestamps with the oculomotor features so that they can be jointly used in fuzzy inference.

In the setup, the aiming direction is determined by an IR beam that produces an aiming spot on the projection screen. An IR camera detects the spot and transmits normalized coordinates $\left( U,V \right)\in \left[ 0,1 \right]$ via UDP. On the PC, these values are mapped to screen coordinates ${{x}^{aim}}\left( t \right)=V\left( t \right)\left( Screen~width \right),~~{{y}^{aim}}\left( t \right)=U\left( t \right)\left( Screen~height \right)$.

These coordinates are treated as the observed aiming point at the moment of firing. In the software implementation, the firing moment is registered by a trigger press event sent by the microcontroller and recorded in Unity.

Each shot $sho{{t}_{i}}$ is characterized by the trigger time $t_{i}^{fire}$ and the grenade impact event time $t_{i}^{impact}$, generated by the projectile hit handler. A target hit is defined strictly by a direct collision with the armored target collider. We set $hi{{t}_{i}}=1$ if the grenade collides with the target object, the tank, and $hi{{t}_{i}}=0$ if the collision occurs with any other scene object. The outcome error is therefore defined as

$E_{i}^{miss}=\left\{ \begin{matrix}   1,~~hi{{t}_{i}}=0  \\   0,~~hi{{t}_{i}}=1  \\ \end{matrix} \right..$                      (5)

To estimate reaction time, a target visibility event is introduced. For each target $j$ is recorded as the first moment when the target becomes observable on the screen. Given the current target position ${{p}_{j}}\left( t \right)$, the screen projection ${{s}_{j}}\left( t \right)=\left( {{x}_{j}}\left( t \right),{{y}_{j}}\left( t \right) \right)$ is computed using the standard Unity transformation WorldToScreenPoint. The visibility time $t_{j}^{vis}$ is defined as the first $t$ for which the target lies in front of the camera and ${{s}_{j}}\left( t \right)$ falls within the screen region.

Since multiple targets can be present simultaneously and the order of engagement is not fixed, each shot is associated with the target that the user was effectively aiming at when the trigger was pressed. Let $a\left( t_{i}^{fire} \right)$ denote the aiming point at firing time obtained from the IR spot, and let ${{s}_{j}}\left( t_{i}^{fire} \right)$ be the screen projection of target $j$ at the same time. The associated target is then selected as

${{j}^{*}}\left( i \right)=\arg  \underset{j\in V\left( t_{i}^{fire} \right)}{\mathop{\min }}\, || a\left( t_{i}^{fire} \right)-{{s}_{j}}\left( t_{i}^{fire} \right)|| $                    (6)

where, $V\left( t \right)$ denotes the set of targets visible on the screen at time $t$. After selecting the associated target, reaction time is computed as

$R{{T}_{i}}=t_{i}^{fire}-t_{{{j}^{*}}(i)}^{vis}$                   (7)

Based on $R{{T}_{i}}$, two classes of temporal errors are defined, with thresholds depending on scenario difficulty $D$.

$E_{i}^{late}=\left\{ \begin{matrix}   1,~R{{T}_{i}}>{{\tau }_{late}}\left( D \right)  \\   0,~~else  \\ \end{matrix} \right. \\ E_{i}^{rush}=\left\{ \begin{matrix}   1,~~R{{T}_{i}}<{{\tau }_{rush}}\left( D \right)  \\    0,~~else  \\ \end{matrix} \right.$                  (8)

The thresholds ${{\tau }_{late}}\left( D \right)$ and ${{\tau }_{rush}}\left( D \right)$ are specified as a function of difficulty level. As difficulty increases, the admissible reaction time range becomes narrower, reflecting higher demands on decision speed.

Within each window ${{W}_{k}}$, relative error frequencies are computed over shots falling into the window

$MissRat{{e}_{k}}=\frac{\mathop{\sum }_{sho{{t}_{i}}\in {{W}_{k}}}E_{i}^{miss}}{{{N}_{k}}+\varepsilon },~~\\ LateRat{{e}_{k}}=\frac{\mathop{\sum }_{sho{{t}_{i}}\in {{W}_{k}}}E_{i}^{late}}{{{N}_{k}}+\varepsilon }, \\ RushRat{{e}_{k}}=\frac{\mathop{\sum }_{sho{{t}_{i}}\in {{W}_{k}}}E_{i}^{rush}}{{{N}_{k}}+\varepsilon },$                      (9)

where, ${{N}_{k}}$ is the number of shots in the window.

A single integrated error indicator is then formed and used as the only linguistic variable derived from the simulator.

$ErrScore_k= w_m \cdot MissRate_k+W_m \cdot LateRate_k+w_r \cdot RushRate_k,w_m+w_l+w_r=1$.                      (10)

Since the error rates lie in $\left[ 0,1 \right]$ and the weights are normalized, $ErrScor{{e}_{k}}\in \left[ 0,1 \right]$. In the baseline configuration, expert-defined weights are used with ${{w}_{m}}=0.6,~~{{w}_{l}}=0.25,~{{w}_{r}}=0.15$, where misses have the largest contribution as an outcome error. To confirm robustness, a sensitivity analysis is performed. When weights vary within plus or minus 0.1 while preserving the priority constraint${{w}_{m}}\ge {{w}_{l}},{{w}_{r}}$, the qualitative conclusions in the comparison of fixed and adaptive modes remain unchanged. If there are no shots in the window, ${{N}_{k}}=0$, the error rates are set to zero and $ErrScor{{e}_{k}}$ is not increased.

For real-time stability, $ErrScor{{e}_{k}}$ is additionally smoothed using exponential averaging

${{\overline{ErrScore}}_{k}}=\alpha \ ErrScor{{e}_{k}}+\left( 1-\alpha  \right)\ {{\overline{ErrScore}}_{k-1}}$                    (11)

The smoothed value $ErrScor{{e}_{k}}\in \left[ 0,1 \right]$ is then provided to the fuzzy subsystem input and interpreted using the linguistic terms Low, Medium, and High. The smoothed quantity $ErrScor{{e}_{k}}$ is updated only as new events become available.

Mamdani fuzzy inference is used as an interpretable mechanism for transforming observed oculomotor metrics and error indicators into estimates of trainee states, namely cognitive load, stress level, and fatigue.

Below, a compact rule set is provided that is sufficient for real-time model operation. Notation is as follows. FixDur is denoted as FD, SaccRate as SR, BlinkRate as BR, and ErrScore as ES.

The presented rule base yields consistent model behavior under typical simulator operating regimes. An increase in PERCLOS and blink rate raises F. An increase in ES strengthens both CL and SL. A high visual search intensity SR under unstable performance increases SL. Rules 22 to 25 target mixed cases where individual features may yield ambiguous interpretations, and they enforce consistent coordination among model outputs. The rule base was constructed to cover dominant monotonic and interaction effects observed during pilot experiments. Table 3 presents the compact fuzzy rule set used for real-time model operation.

Table 3. Fuzzy rules

For each output variable, after all rules are applied and aggregated, a resulting membership function ${{\mu }_{out}}\left( u \right)$. The crisp output value is then computed using the centroid method.

${{y}_{k}}=\frac{\int_{0}^{1}{{{\mu }_{out}}}\left( u \right)\ u\ du}{\int_{0}^{1}{{{\mu }_{out}}}\left( u \right)\ du}$                (12)

where, ${{y}_{k}}\in \left\{ C{{L}_{k}},S{{L}_{k}},{{F}_{k}} \right\}.$ The resulting values are subsequently used to generate user feedback and to adapt scenario parameters within the training difficulty control module.

(a) Baseline training scenario with a single armored target under favorable environmental conditions

(b) Multi target scenario with increased environmental complexity

Figure 7. Training scenarios used in both conditions

In the first scenario, shown in Figure 7(a), a single armored target was present. The target was controlled by an agent that was pre-trained using deep reinforcement learning. The agent performed continuous maneuvering to minimize the probability of being hit. Environmental conditions were favorable, with no fog and no wind, and clear weather. A realistic ballistic model was used, and the weapon was a virtual analogue of the RPG-7V2.

In the second scenario, shown in Figure 7(b), three armored targets were present. They were controlled by a multi-agent system that supported coordinated evasive behavior. Environmental conditions were substantially more challenging due to fog, precipitation, and wind. This reduced visibility and increased aiming difficulty while preserving the same realistic ballistics and the same weapon type.

The transition to the second scenario occurred only after all targets in the first scenario had been hit. Throughout task execution, oculomotor data were continuously recorded as described in Section 4, along with performance measures including hit accuracy as described in Section 5. These data were used for timestamp-based alignment with oculomotor features and subsequent analysis, and for computation of ES in both conditions.

The overall training structure was preserved. The participant completed the baseline scenario and then proceeded to the advanced scenario. However, unlike the fixed condition, the advanced scenario parameters were not predefined and were automatically updated at control intervals based on smoothed estimates of CL, SL, F, and ES. At each step, the Scenario Manager produced a control action by updating the difficulty level $D$, by no more than one level per cycle, or by applying soft parameter tuning within the current level inside the allowed ranges. As a result, the adaptive condition produced a personalized variant of the advanced scenario that was consistent with the user’s current psychophysiological and behavioral characteristics. This ensured smooth adaptation and reduced frequent difficulty switching.

The reported findings suggest that the proposed adaptive controller supports smoother task-related eye-tracking trajectories and more stable inferred state dynamics than the fixed-condition policy under the controlled conditions of the present study. At the group level, the Adaptive condition was associated with narrower confidence intervals for several normalized eye-tracking measures and for the fuzzy outputs, suggesting reduced between-participant dispersion during task execution. In this sense, the adaptive subsystem appears to support more stable regulation of the training process under the controlled conditions of the present study.

At the same time, the interpretation of the inferred state variables requires caution. The variables CL, SL, and F are model-derived latent state estimates produced by the fuzzy inference system rather than independently validated physiological ground-truth measures. Therefore, the reduced variability observed in these trajectories should be interpreted primarily as stabilization of controller-estimated user states. This pattern is consistent with the intended effect of adaptive regulation, but it does not by itself establish a verified reduction in true cognitive load, stress, or fatigue. In addition, part of the observed smoothing may reflect the structure of the inference and control pipeline itself.

The participant-level performance results also indicate that the observed effects are not uniform in magnitude across individuals. Table 7 shows generally consistent directional changes between the Fixed and Adaptive conditions, but with noticeable heterogeneity in effect size across participants. This is consistent with the personalized design of the normalization and control framework, in which a perfectly uniform response across all users would not be expected. Accordingly, the main result should be interpreted not as an identical participant-level benefit for every trainee, but as a tendency toward more stable individualized trajectories under adaptive regulation.

The practical size of the observed performance gains remains modest. HitRate increased slightly, while RushRate and ErrScore decreased, and LateRate changed only minimally. Thus, statistical significance should be distinguished from practical significance. Nevertheless, in an adaptive simulator context, even small but stable gains may still be meaningful when they are accompanied by improved control stability and do not come at the expense of task execution quality. From this perspective, the present findings are better understood as evidence of incremental but consistent benefit, rather than a large operational performance shift.

The present evaluation should also be interpreted as a proof-of-concept validation rather than as a fully developed comparative study. Although the fixed-versus-adaptive comparison is sufficient to examine whether the proposed framework can produce stable and directionally favorable changes under controlled conditions, it does not isolate the individual contribution of its main components. In particular, the current design does not yet distinguish the relative role of personalized normalization, temporal smoothing, the fuzzy inference layer, or adaptive control more generally. Therefore, the observed gains should be interpreted at the level of the integrated system rather than as evidence for the unique effect of any single module. A more comprehensive comparative analysis, including ablation studies and stronger alternative baselines, remains an important direction for future work.

From an information-systems perspective, the proposed framework functions as a human-in-the-loop architecture that integrates multimodal behavioral inputs, transforms them into interpretable latent state estimates, and supports adaptive decision management within a closed control loop. In this sense, the contribution of the study is not limited to simulator control alone, but also to the design of a real-time information-processing structure for sensing, state estimation, and transparent rule-based adaptation in training environments. This distinguishes the present approach from prior work focused mainly on monitoring, offline assessment, single-modality analysis, or less interpretable adaptive decision layers.

Several limitations should also be considered. The sample size was modest, and the scenario set and session structure were limited, which constrains external validity and broader generalization. The study was conducted under controlled experimental conditions and therefore does not yet establish that the proposed controller will remain equally reliable across different user groups, longer sessions, noisier sensing conditions, or a wider variety of operational tasks. Accordingly, the present results should be interpreted as evidence of potential rather than established deployment suitability in high-risk or safety-critical training settings. In addition, the membership functions, fuzzy rules, and error-score weights were specified using expert knowledge; although qualitative conclusions were stable under reasonable parameter variations, more systematic parameter identification may improve robustness and portability. Eye-tracking data quality also remains a practical constraint, particularly under head motion, occlusions, and transient tracking loss.

Another important limitation is that the latent variables CL, SL, and F were not externally validated against independent physiological or subjective reference measures. As a result, the present study cannot determine to what extent the observed stabilization of these trajectories corresponds to true changes in cognitive load, stress, or fatigue beyond the controller’s internal state representation. The study also employed a single initial baseline for normalization. While this approach is common in real-time adaptive systems, future work should examine alternative strategies such as multi-stage baselines, explicit familiarization periods, or adaptive baseline updates across the session. Broader validation with larger cohorts, more diverse scenarios, and longitudinal protocols will also be necessary to assess learning efficiency, retention, and transfer to more complex missions. An especially important next step will be an ablation-based evaluation that separately tests the contribution of personalized normalization, temporal smoothing, and the fuzzy decision layer against simpler adaptive and non-adaptive alternatives.