Proximal Policy Optimization–Based Deep Reinforcement Learning for Intelligent Test Case Generation in Autonomous Vehicles

Test generation based on a rule and scenario library has been a popular technique in early AV-based validation. Such methods are based on manually specified traffic policies, professionally defined scenarios, or prescribed combinatorial sets of parameters to test the behavior of the system in known driving conditions. Although these approaches are interpretable and easily applicable, they have low scalability and subpar generalization [22]. Hand-operated scenario libraries are limited by human assumptions by nature and rarely represent the rare, compound, and emergent safety-critical scenarios caused by complex vehicle interaction. During the process of moving AV systems to open-ended and uncertain environments, rule-based testing has not been found to be adequate in unearthing unforeseen failure modes [23].

Search-based and optimization-based testing methods have been suggested to enhance coverage of the scenario by performing a systematic search of the scenario parameter space. Other popular techniques are genetic algorithms, particle swarm optimization, and Bayesian optimization, which are applied to find the failure-causing inputs by maximizing the risk or violation measures specified by the user [24]. These techniques have proved to be effective in revealing the corner cases in the simulation environments. They usually, however, depend on handwritten fitness functions, handwritten representation of a static scenario, and offline optimization, which hinder their dynamism in dealing with changes in system behaviour. Search-based algorithms typically have difficulty in scaling to high-dimensional continuous spaces, and can tend to prematurely underscore local optimality [25].

Adversarial scenario generation and robustness testing are concerned with identifying vulnerabilities in perception and decision-making modules by adding small but specific perturbations to sensor inputs or environmental parameters. Adversarial attacks using gradients and falsification methods have been used to detect unsafe behaviours in neural network constituents and closed-loop control systems [26]. Although these methods yield important information on the sensitivity of models, they are generally component-based or a partial observer of a system. Consequently, they can miss the failure modes due to long-term interactions, temporal dependencies, or multi-agent dynamics that appear in the driving situation in the real world [27].

The test case generation based on RL has received growing popularity because of its sequential decision-making process representation of scenario generation. Challenging traffic behaviors, adversarial agents, and environment settings that are maximally likely to cause collisions or safety violations have been generated by RL agents. Value-based algorithms (DQN or actor-critic (A3C) algorithms were the most used in early studies [28]. Despite potential advantages, such methods are typically unstable to training, inefficient to explore, and sensitive to reward design, especially in high-dimensional and continuous simulation tasks such as autonomous driving. Such constraints restrict their use to scalable and reliable AV safety validation [29].

In contrast to existing RL-based testing systems, the proposed one uses PPO to attain stable and sample-efficient as well as scalable scenario generation. In contrast to acting on top of value or asynchronous action critic, such as PPO, policy updates are limited to avoid the destructive policy update, leading to robust training in challenging driving conditions. The given framework is specifically focused on system-level safety-critical event discovery, where scenario parameters are dynamically adjusted to observed closed-loop behaviors as opposed to fixed risk measures. This can be used to better explore rare and high-impact cases, better test coverage, and better fault detection than existing RL-based and optimization-driven testing strategies. Therefore, the PPO framework is an important breakthrough in adaptable, learning-based AV test case generation.

4.6 Proximal Policy Optimization

The PPO method is used as the baseline in this study due to its ease of implementation and consistent policy improvement. Although PPO is based on Trust Region Policy Optimization (TRPO), it is more efficient because of two key innovations: Generalized Advantage Estimation (GAE) and an unconstrained surrogate objective function.

4.6.1 Unconstrained surrogate objective function

To achieve stable and consistent policy improvement, TRPO limits policy updates to avoid significant deviations. PPO streamlines this approach by using a clipped surrogate objective function, which penalizes updates that stray too far from the existing policy, eliminating the need for complex constraints. The PPO objective function is defined as:

$\begin{gathered}J_{P P O}(\theta)=E_{s, a}\left[\min \left(\rho_t(\theta) \widehat{A}_t, \operatorname{clip}\left(\rho_t(\theta) 1-\epsilon_1\right.\right.\right. \left.\left.+\epsilon) \widehat{A}_t\right)\right]\end{gathered}$                (15)

where, $\rho_t(\theta)=\frac{\pi_\theta(s \mid a)}{\pi_{\theta_{\text {old}}}(s \mid a)}$ the probability ratio between the new and old policies. $\widehat{A}_t$: The estimated advantage at time step t. $\epsilon$ is a small hyperparameter (e.g., 0.2) that restricts the update range.

4.6.2 Generalized advantage estimation

The advantage function is critical for calculating policy gradients in PPO. It is estimated using the difference between the action-value function and the state-value function:

$\widehat{A}^\pi\left(s_t, a_t\right)=\widehat{Q}^\pi\left(s_t, a_t\right)-V\left(s_t, w\right)$              (16)

where, $\widehat{Q}^\pi\left(s_t, a_t\right)$ is the estimated action-value function. $V\left(s_t, w\right)$ is the estimated value of state $s_t$ with parameters $w$.

Monte Carlo Estimate (used in TRPO):

$\widehat{Q}^\pi\left(s_t, a_t\right)=\sum_{l=t}^{\infty} \gamma^{l-t} r_l$                   (17)

This estimate is unbiased but exhibits high variance.

One-Step Bootstrapping (used in A3C):

$\widehat{Q}^\pi\left(s_t, a_t\right)=r_t+\gamma V\left(s_{t+1}, w\right)$                    (18)

This method has lower variance but introduces bias. To strike a balance between bias and variance, GAE uses a weighted sum of multi-step temporal difference errors:

$\left(s_t, a_t\right)=\sum_{l=t}^{\infty} \gamma \lambda^{l-t} \delta_l$                  (19)

where the temporal difference (TD) error $\delta_l$ defined as:

$\delta_l=r_l+\gamma V\left(s_{l+1}, w\right)-V\left(s_l, w\right)$                   (20)

Here, $\lambda \in[0,1]$ a hyperparameter that governs the biasvariance trade-off. A lower $\lambda$ introduces more bias with less variance, while a higher $\lambda$ does the opposite.

4.7 Intelligent test case generation

The challenge of identifying program flaws through execution on carefully selected input data is a central focus in testing. A program P is considered to contain an error if its output does not match the expected requirements or is deemed incorrect by the tester. Verifying output correctness and generating relevant test scenarios requires some knowledge of the correct program, which may or may not be sufficiently complete. Ideally, program testing involves using input/output samples to demonstrate or probabilistically validate the program's equivalence to a correct version or executable specification. Testing DRL agents remains in its early stages. Similar to DL agents trained through supervised learning, DRL agents are also susceptible to adversarial attacks that generate conflicting scenarios. The approach for DRL differs significantly, as it emphasizes configuring the environment for each scenario encountered by the agent, rather than altering raw sensor inputs. The sampling strategy used as a baseline in the existing study closely aligns with this approach. Empirical findings demonstrate that the search-based method performs better than others in inducing a broader and more diverse range of failures across multiple case studies. A method has been proposed to assess the adaptability of DRL agents by reinitializing learning in environments different from those encountered during the initial training phase. This enables the construction of an adaptation frontier distinguishing between scenarios in which the agent adapts successfully and those in which it fails.

Although other methodologies also generate new environmental configurations, the existing approach does not retrain the agent. Instead, the focus is on testing to identify vulnerabilities in the agent during evaluation. Explored training and testing DRL agents in procedurally generated environments. Specifically, their study utilized a set of algorithmically generated 3D mazes to train agents, followed by a local search process that modified the mazes during evaluation based on agent performance. This technique produced out-of-distribution configurations not represented in the original training dataset. In contrast to directly testing agent performance in these altered environments, the existing study employs a failure predictor, a proxy representation of the environment is to reduce the computational cost of the search.

In more complex environments than mazes, it becomes prohibitively expensive to execute the DRL agent within the environment at every search iteration. To address evaluation challenges, a recent search-based strategy was proposed to assess the quality of DRL agents. Their approach involves sampling the environment to identify a reference trace that successfully solves the RL task. This trace is constructed using a depth-first search algorithm and consists of all states not pruned during the search’s backtracking process.

The key aspects being measured include fault detection effectiveness, scenario diversity, safety-critical coverage, and comfort in vehicle trajectories. The null hypothesis (H₀) assumes that the PPO-generated test cases are no better or inferior in performance compared to baseline methods (such as random or rule-based test generation).

This is expressed mathematically as:

$H_0: \mu_{P P O} \leq \mu_{\text {baseline }}$

where, $\mu$ represents the average effectiveness score, which is a weighted sum of fault detection rate F, scenario diversity D, and safety event coverage C. These weights $w_1, w_2, w_3$ are chosen based on the Importance of each factor to the test generation goals.

In contrast, the alternative hypothesis (H₁) asserts that the PPO-based framework outperforms the baseline, aiming to generate more useful, diverse, and fault-revealing test cases:

$H_1: \mu_{P P O} \leq \mu_{\text {baseline }}$

To deepen the analysis, sub-hypotheses are introduced:

• H₁.1 checks whether PPO-generated cases trigger more faults, indicating stronger stress-testing capability.
• H₁.2 evaluates behavioral diversity using variance across features like traffic density, weather, and obstacle types.
• H₁.3 measures coverage of safety-critical scenarios, ensuring the policy does not miss important corner cases.
• H₁.4 focuses on trajectory comfort, using mean jerk as a proxy for smoothness in the vehicle's movement.

Each equation ties directly to a measurable property of the system, enabling objective evaluation using statistical tests (e.g., t-test or Wilcoxon test) over generated test data. The reward function in the PPO model encourages policies that optimize these objectives, and testing these hypotheses validates that the optimization successfully aligns with real-world testing needs.

4.8 Proximal Policy Optimization-driven Deep Reinforcement Learning framework for intelligent test case generation in Autonomous Vehicles

Objective: Maximize the expected cumulative reward for generating test cases that challenge AV's decision-making under safety, efficiency, and comfort constraints.

Input: Environment model $\varepsilon$; Initial policy network $\pi_\theta(a \mid s)$; Value network $V(s ; w)$; Hyperparameters: Discount factory $\gamma$, GAE factor $\lambda$, Clipping threshold $\epsilon$, batch size B , epochs U

Output: Optimized policy $\pi_\theta$ for generating effective test cases

Initialization: 1. Randomly initialize policy parameters $\theta$ and value function parameters $w$; Initialize replay buffer $B \leftarrow \emptyset$

Step 1. Collect Environment Rollouts

Generate episodes using the existing policy $\pi_\theta$ and collect a batch mathcal:

$D=\left\{\left(s_t, a_t, r_t, s_{t+1}\right)\right\}_{t=1}^B$                 (21)

Step 2: Estimate the Temporal Difference (TD) Error

For each step t, compute:

$\delta_t=r_t+\gamma V\left(s_{t+1}, w\right)-V\left(s_t, w\right)$                     (22)

Step 3: Generalized Advantage Estimation (GAE)

$\widehat{A}_t=\sum_{t=0}^{\infty}(\gamma \lambda)^l \delta_{t+1}$                  (23)

This balances bias and variance in the estimation of the advantage function.

Step 4: Value Function Target

$\widehat{V}_t=\widehat{A}_t+V\left(s_t, w\right)$                 (24)

Step 5: Policy Update with PPO Surrogate Objective

Define the probability ratio:

$\rho_t(\theta)=\frac{\pi_\theta\left(a_t \mid s_t\right)}{\pi_{\theta_{\text {old}}}\left(a_t \mid s_t\right)}$                 (25)

Clip the objective to avoid large policy updates:

$\begin{gathered}J_{P P O}(\theta)=E_{s, a}\left[\min \left(\rho_t(\theta) \widehat{A}_t, \operatorname{clip}\left(\rho_t(\theta) 1-\epsilon_1\right.\right.\right.\left.\left.+\epsilon) \widehat{A}_t\right)\right]\end{gathered}$                   (26)

Update policy network:

$\theta \leftarrow \theta+\alpha_\theta \nabla_\theta J_{P P O}(\theta)$                   (27)

Step 6: Value Network Update: Minimize squared error between predicted and target value:

$J_v(w)=\frac{1}{B} \sum_{t=1}^B\left(V\left(s_t, w\right)-\widehat{V}_t\right)^2$                    (28)

Update value network:

$w \leftarrow w-\alpha_v \nabla_w J_V(w)$                     (29)

Step 7: Repeat Steps 1-6: Repeat for multiple iterations until convergence or maximum episodes reached.

Reward Function Design: The reward $R\left(s_t, a_t\right)$ balances three objectives:

$R\left(s_t, a_t\right)=\lambda_1 r_{\text {comfort}}+\lambda_2 r_{\text {efficiency}}+\lambda_3 r_{\text {safety}}$                   (30)

With the help of this method, the PPO agent may learn a lane-change decision policy that emphasizes realism, variety, and edge-case exposures while producing difficult yet secure test scenarios for self-driving vehicles.

4.9 Test case generation

The testing process begins with analysis, identifying key factors—such as ego vehicle acceleration—that influence driver assistance technologies. Based on these criteria, relevant test scenarios are developed. The third phase involves executing these scenarios in a Software-in-the-Loop (SiL) simulation environment. In the evaluation phase, scenario criticality is assessed using metrics like time-to-collision, which serve as reward signals for the DRL agent to guide scenario refinement. The exploration phase follows, leveraging prior knowledge or probing new environmental conditions. Parameter adjustment is performed using a ϵ-greedy algorithm that balances optimal and random actions to improve scenario diversity. Each action corresponds to modifying a parameter, prompting the generation of a new test instance, and restarting the cycle. Finally, the save critical test cases phase retains high-impact scenarios for future analysis.

Step 1: Analysis: This initial step identifies the key influencing factors on the automated driving function under investigation, these may include: Speed of the ego vehicle $v_e$; Relative positions and velocities of surrounding vehicles $\left\{v_x, d_x\right\}$; Road curvature $\rho$, weather $W$, and traffic density $T$. These parameters form the input feature vector $i \in R^n$ which defines the scenario space.

Step 2: Test Case Generation: Using the parameters identified in Step 1, initial test scenarios are generated. The test cases are defined as a vector:

$P=\left\{v_e, v_0, d_0, \rho, W, T, \ldots\right\}$                   (31)

These parameters are input into a simulation environment. Initially, parameter values can be chosen randomly or based on predefined distributions, as the RL agent will iteratively refine them.

Step 3: Test Run Execution: Each test case from Step 2 is executed using a SiL simulation framework. The simulation simulates vehicle dynamics, sensor models, and the autonomous driving software stack. During this phase, key runtime metrics are collected, such as: Time-to-Collision TTC; Lane deviation; Braking and acceleration behavior.

Step 4: Test Evaluation: The criticality of each scenario is evaluated using metrics like Time-to-Collision (TTC):

$T T C=\frac{d_{r e l}}{v_{r e l}}$ where $v_{r e l}=v_e-v_{o b s t a c l e}$                    (32)

A reward function is constructed to encourage the generation of dangerous but realistic cases:

$R\left(s_t, a_t\right)=-\alpha T T C^{-1}+\beta$ comfort penalty                    (33)

Scenarios with low TTC values are rewarded more as they are closer to critical conditions.

Step 5: Exploration: Using the PPO RL agent, new test scenarios are explored. To balance exploration and exploitation, an epsilon-greedy strategy is used:

$a=\left\{\begin{array}{l}\operatorname{random}(A), \text { with probability } \epsilon \\ \arg \max _a Q(s, a), \text { with probability } 1-\epsilon\end{array}\right.$                    (34)

where, $Q(s, a)$: Estimated action-value; A: Action space, i.e, parameter modifications; $\epsilon \in[0,1]$: Exploration rate (typically decaying over time).

Step 6: Parameter Change: Based on exploration, test parameters are adjusted. An action here refers to increasing or decreasing a parameter:

$P_x^{\prime}=P_x+\Delta P, \quad$ where $\Delta P \in\{-\delta, 0,+\delta\}$                      (35)

This modifies the scenario to potentially make it more critical. For example: Increasing vehicle speed $v_e$; Decreasing following distance $d_0$; Changing road curvature $p$. The newly generated parameter set becomes the next test case.

Step 7: Save Critical Test Cases: Scenarios that meet or exceed a criticality threshold (eg, TTC < 2s) are stored for future safety validation and regression testing. A critical test case repository ensures: Reusability in future training and validation. Coverage of edge conditions and failure-prone cases. Save if: $R\left(s_t, a_t\right)>R_{\text {threshold}}$.

The cycle repeats by feeding back the modified parameters into Step 2, forming a closed-loop system for intelligent test case generation. The RL agent continuously adapts its strategy to find edge cases that challenge the autonomous driving stack, while ensuring diversity, realism, and safety-critical exposure.