ExamPPO: Adversarial Testing for AVs

• ExamPPO is an interactive adversarial testing framework that uses a PPO-trained surrounding vehicle as a dynamic examiner to evaluate autonomous vehicle robustness.
• The framework employs multi-head attention and strength-conditioned policy encoding to generate scenario-adaptive interventions under varying confrontation intensities.
• Experimental results show that graded confrontation levels significantly impact metrics like collision rate and decision failure, enabling comprehensive robustness analysis.

ExamPPO is an interactive adversarial testing framework for autonomous vehicles in which a learned surrounding vehicle functions as an intelligent examiner rather than as a cooperative traffic participant. The framework is designed to evaluate, not improve, the autonomous vehicle under test: the surrounding vehicle reacts online to the tested policy, escalates or attenuates adversarial pressure through a scalar confrontation factor, and uses a PPO-trained policy with multi-head attention to generate scenario-adaptive interventions. In the paper’s formulation, ExamPPO is a PPO-based testing framework whose novelty lies in strength-conditioned adversarial testing, attention-based policy encoding, reward design, and structured evaluation, rather than in proposing a new foundational policy-optimization algorithm (Guo et al., 29 Jul 2025).

1. Conceptual role and problem setting

ExamPPO addresses a validation problem that standard autonomous-driving test pipelines handle only imperfectly: the generation of rare, high-risk, highly interactive scenarios that can expose weaknesses in tactical decision-making. The paper distinguishes its setting from passive dataset replay, fixed scenario suites, random perturbation methods, and optimization-based adversarial trajectory generation. Hand-crafted and replay-based methods are reproducible but static; randomized perturbations are broad but often untargeted; optimization-based adversarial trajectories can target collision-like costs but are typically fixed and must be re-solved for new settings. ExamPPO instead implements closed-loop examination: the surrounding vehicle observes the tested autonomous vehicle and adapts its behavior online to sustain pressure under evolving traffic conditions (Guo et al., 29 Jul 2025).

The framework is organized around two asymmetric agents. The autonomous vehicle is the test subject and remains fixed during evaluation. The surrounding vehicle is the examiner and is trained to produce adversarial, scenario-aware interventions. This asymmetry is central: the objective is not joint multi-agent coordination, but systematic probing of autonomous-vehicle robustness through a controlled adversary. The paper characterizes the framework as interactive because the examiner responds online rather than following a precomputed trajectory; adversarial because the examiner is optimized to disrupt or pressure the tested policy; scenario-adaptive because the examiner conditions on traffic context and autonomous-vehicle behavior; and intensity-controllable because a scalar confrontation factor explicitly modulates how aggressive the examiner should be (Guo et al., 29 Jul 2025).

Methodologically, ExamPPO retains the standard PPO clipped surrogate rather than altering the proximal-update mechanism itself. This distinguishes it from PPO-adjacent work that changes the trust-region surrogate, such as POP3D’s point-probability penalty (Chu, 2018), and from analyses that reinterpret PPO-Clip as a per-sample KL-regularized update (Colletti et al., 22 Jun 2026). In ExamPPO, PPO serves as the optimizer for the examiner policy; the substantive contribution is the adversarial testing formulation built around that optimizer (Guo et al., 29 Jul 2025).

2. POMDP formulation and state-action structure

The paper models adversarial testing from the surrounding vehicle’s perspective as a partially observable Markov decision process,

$$(\mathcal{S}, \Omega, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma, \rho_0),$$

where $\mathcal{S}$ is the state space, $\mathcal{A}$ the action space, $\Omega:\mathcal{S}\to\mathcal{O}$ the observation mapping, $\mathcal{O}$ the observation space, $\mathcal{T}$ the transition distribution, $\mathcal{R}$ the reward function, $\gamma\in[0,1]$ the discount factor, and $\rho_0$ the initial-state distribution. The surrounding vehicle acts according to a stochastic policy

$\pi(a_t \mid o_t),$

and optimizes

$\mathcal{S}$0

The paper also provides the associated value and action-value functions,

$\mathcal{S}$1

$\mathcal{S}$2

and the Bellman optimality relation

$\mathcal{S}$3

Partial observability arises because the surrounding vehicle has only local and partial observations of the traffic scene (Guo et al., 29 Jul 2025).

The observation is represented as a structured matrix $\mathcal{S}$4, where $\mathcal{S}$5 is the number of observed agents and $\mathcal{S}$6 is the number of features per vehicle. For each observed vehicle $\mathcal{S}$7, the feature vector is

$\mathcal{S}$8

Here $\mathcal{S}$9 is a presence indicator, $\mathcal{A}$0 are Cartesian coordinates, $\mathcal{A}$1 are velocity components, $\mathcal{A}$2 is the relative heading angle with respect to the surrounding vehicle, and $\mathcal{A}$3 is the confrontation intensity. The explicit inclusion of $\mathcal{A}$4 in every vehicle feature vector makes confrontation strength part of the observation model rather than only an external evaluation tag (Guo et al., 29 Jul 2025).

The action space is discrete and deliberately coarse: $\mathcal{A}$5 The paper does not numerically specify the episode horizon $\mathcal{A}$6, but the sequential structure is clear: the surrounding vehicle repeatedly observes the tested autonomous vehicle, selects a longitudinal action, receives a shaped reward, and continues the interaction over time. During training, episode initialization is randomized over both scenario and autonomous-vehicle policy, so the examiner is exposed to multiple traffic configurations and controller types (Guo et al., 29 Jul 2025).

3. Strength-conditioned policy architecture and PPO optimization

ExamPPO’s policy network combines PPO with a multi-head attention-enhanced encoder. The paper states that the policy input contains surrounding-vehicle kinematics, road context, and confrontation strength $\mathcal{A}$7. Rather than flattening all entities into a single vector and applying only a multilayer perceptron, the model computes vehicle-wise embeddings and applies scaled dot-product attention,

$\mathcal{A}$8

with multiple heads in parallel. The intended function of attention is relational rather than merely compressive: it allows the examiner to focus dynamically on salient interaction cues such as proximity, velocity, and spatial alignment with the autonomous vehicle (Guo et al., 29 Jul 2025).

The implementation details are partially specified. Input observations pass through two fully connected layers of 64 units with ReLU activation, followed by a two-head self-attention module with 64-dimensional queries and keys. The attention output is concatenated and projected to policy and value heads, with layer normalization and residual connections used for stability. The paper does not provide layer-by-layer actor parameterization beyond this point, nor does it write out a specialized critic loss, but it is explicit that the surrounding vehicle is trained with PPO rather than with a custom optimizer (Guo et al., 29 Jul 2025).

The policy is strength-aware in two senses. First, $\mathcal{A}$9 is part of the observation. Second, the pseudocode writes the encoder and policy as

$\Omega:\mathcal{S}\to\mathcal{O}$0

This is a conditional policy in which confrontation intensity is an explicit control variable, not a latent nuisance factor. The strength signal is externally specified by the tester and remains fixed within an episode; it is not learned online and is not scheduled adaptively inside a rollout (Guo et al., 29 Jul 2025).

For optimization, the paper adopts the standard PPO clipped objective,

$\Omega:\mathcal{S}\to\mathcal{O}$1

with the intended importance ratio

$\Omega:\mathcal{S}\to\mathcal{O}$2

Training uses generalized advantage estimation with $\Omega:\mathcal{S}\to\mathcal{O}$3, an entropy coefficient of $\Omega:\mathcal{S}\to\mathcal{O}$4, rollout length $\Omega:\mathcal{S}\to\mathcal{O}$5, discount factor $\Omega:\mathcal{S}\to\mathcal{O}$6, learning rate $\Omega:\mathcal{S}\to\mathcal{O}$7, batch size $\Omega:\mathcal{S}\to\mathcal{O}$8, clip range $\Omega:\mathcal{S}\to\mathcal{O}$9, and Adam as the optimizer. ExamPPO is trained for $\mathcal{O}$0 time steps per scenario (Guo et al., 29 Jul 2025).

4. Confrontation intensity and reward structure

The confrontation factor is one of ExamPPO’s defining mechanisms. It is a scalar

$\mathcal{O}$1

used to modulate adversarial intensity continuously. During training, $\mathcal{O}$2 is sampled per episode from

$\mathcal{O}$3

and these five levels are denoted $\mathcal{O}$4, from weakest to strongest confrontation. Because $\mathcal{O}$5 is included both in the observation and in the reward, it changes not only how the examiner is evaluated but also what it perceives as the task. Low-$\mathcal{O}$6 behavior is intended to be closer to ordinary traffic participation; high-$\mathcal{O}$7 behavior increasingly prioritizes disruption of the autonomous vehicle (Guo et al., 29 Jul 2025).

The total reward is

$\mathcal{O}$8

This decomposition is the paper’s main confrontation-control equation. Since $\mathcal{O}$9 increases and $\mathcal{T}$0 decreases on $\mathcal{T}$1, larger $\mathcal{T}$2 shifts the examiner toward adversarial disruption and away from self-centered efficient motion. The adversarial component is further decomposed as

$\mathcal{T}$3

The exact weights $\mathcal{T}$4 are not reported in the paper (Guo et al., 29 Jul 2025).

The distance term is

$\mathcal{T}$5

where $\mathcal{T}$6 is described as a smooth bounded shaping function based on a scaled sigmoid: it is negative below $\mathcal{T}$7, zero beyond $\mathcal{T}$8, and transitions smoothly in between. The paper does not provide an explicit analytical form for $\mathcal{T}$9. The velocity-disruption term is

$\mathcal{R}$0

The aggressive-maneuver term is

$\mathcal{R}$1

and the path-blocking term is

$\mathcal{R}$2

Together these terms reward close interaction, autonomous-vehicle slowdown, aggressive timing, and occupation of the autonomous vehicle’s forward path (Guo et al., 29 Jul 2025).

The efficiency term preserves plausible motion: $\mathcal{R}$3 with

$\mathcal{R}$4

The penalty term $\mathcal{R}$5 penalizes invalid or implausible behavior, specifically off-road deviation, but the paper does not state a closed-form expression. The collision term is intensity-dependent: for

$\mathcal{R}$6

a collision yields reward $\mathcal{R}$7, whereas for higher-intensity testing it yields $\mathcal{R}$8. This rule makes low-intensity testing contact-averse and high-intensity testing explicitly risk-tolerant (Guo et al., 29 Jul 2025).

5. Evaluation methodology, baselines, and metrics

The experimental platform is a modified highway-env simulator extended with lane-level geometric constraints, conflict-zone tagging, and directional vehicle initialization. The paper evaluates three scenarios chosen for conflict and negotiation structure: unsignalized intersection, highway lane change, and ramp merging. The autonomous vehicle under test is instantiated with three controller families: RPID, described as a rule-based method; a pre-trained PPO policy; and a pre-trained RecurrentPPO policy. This yields comparisons across heuristic, feedforward learned, and memory-based learned decision policies (Guo et al., 29 Jul 2025).

The surrounding-vehicle baselines are AdvDQN, ExamPPO-wo, and a standard PPO agent without adversarial strength conditioning or attention. ExamPPO-wo retains confrontation-strength conditioning but removes the attention module, making it the key architectural ablation. The paper emphasizes AdvDQN and ExamPPO-wo in the quantitative tables. All runs are seeded for reproducibility, and testing uses seeds $\mathcal{R}$9, $\gamma\in[0,1]$0, and $\gamma\in[0,1]$1 (Guo et al., 29 Jul 2025).

The metric suite is structured around both adversary-side and autonomous-vehicle-side outcomes. The examiner’s action entropy is

$\gamma\in[0,1]$2

used to characterize behavioral diversity or decisiveness. Confrontation Success Rate (CSR) is defined as

$\gamma\in[0,1]$3

where disruption includes significant autonomous-vehicle behavioral change such as abrupt braking, yielding, or task failure. Decision Failure Rate (DFR) is

$\gamma\in[0,1]$4

and is grounded in the RSS safety framework, especially the right-of-way rule in the intersection scenario. Additional reported metrics are collision rate, task success rate, average autonomous-vehicle speed, average surrounding-vehicle speed, and Post-Encroachment Time (PET), a temporal proximity measure for interaction risk (Guo et al., 29 Jul 2025).

6. Reported empirical behavior

In the unsignalized-intersection scenario with a PPO-controlled autonomous vehicle at fixed $\gamma\in[0,1]$5, the paper reports a large separation between baselines and the full method. AdvDQN yields surrounding-vehicle speed $\gamma\in[0,1]$6 m/s, autonomous-vehicle speed $\gamma\in[0,1]$7 m/s, PET $\gamma\in[0,1]$8 s, collision rate $\gamma\in[0,1]$9, DFR $\rho_0$0, and CSR $\rho_0$1. ExamPPO-wo raises these interaction metrics substantially, reaching surrounding-vehicle speed $\rho_0$2 m/s, autonomous-vehicle speed $\rho_0$3 m/s, PET $\rho_0$4 s, collision rate $\rho_0$5, DFR $\rho_0$6, and CSR $\rho_0$7. ExamPPO further increases the intensity of the interaction to surrounding-vehicle speed $\rho_0$8 m/s, autonomous-vehicle speed $\rho_0$9 m/s, PET $\pi(a_t \mid o_t),$0 s, collision rate $\pi(a_t \mid o_t),$1, DFR $\pi(a_t \mid o_t),$2, and CSR $\pi(a_t \mid o_t),$3. The paper interprets the AdvDQN-to-ExamPPO-wo gap as evidence for strength conditioning and the ExamPPO-wo-to-ExamPPO gap as evidence that attention materially improves sustained, temporally coordinated adversarial interaction (Guo et al., 29 Jul 2025).

The graded-confrontation experiments are the paper’s main validation of $\pi(a_t \mid o_t),$4 as an interpretable difficulty knob. For RPID, increasing intensity from $\pi(a_t \mid o_t),$5 to $\pi(a_t \mid o_t),$6 changes CSR from $\pi(a_t \mid o_t),$7 to $\pi(a_t \mid o_t),$8, DFR from $\pi(a_t \mid o_t),$9 to $\mathcal{S}$00, collision rate from $\mathcal{S}$01 to $\mathcal{S}$02, PET from $\mathcal{S}$03 s to $\mathcal{S}$04 s, and task success from $\mathcal{S}$05 to $\mathcal{S}$06. For the PPO autonomous vehicle, $\mathcal{S}$07 yields CSR $\mathcal{S}$08, DFR $\mathcal{S}$09, collision $\mathcal{S}$10, and task success $\mathcal{S}$11, whereas $\mathcal{S}$12 yields CSR $\mathcal{S}$13, DFR $\mathcal{S}$14, collision $\mathcal{S}$15, PET $\mathcal{S}$16 s, and task success $\mathcal{S}$17, and $\mathcal{S}$18 yields CSR $\mathcal{S}$19, DFR $\mathcal{S}$20, collision $\mathcal{S}$21, PET $\mathcal{S}$22 s, and task success $\mathcal{S}$23. For RecurrentPPO, the moderate-intensity region is less catastrophic—$\mathcal{S}$24 gives CSR $\mathcal{S}$25, DFR $\mathcal{S}$26, collision $\mathcal{S}$27, PET $\mathcal{S}$28 s, task success $\mathcal{S}$29, and $\mathcal{S}$30 gives CSR $\mathcal{S}$31, DFR $\mathcal{S}$32, collision $\mathcal{S}$33, PET $\mathcal{S}$34 s, task success $\mathcal{S}$35—but $\mathcal{S}$36 and $\mathcal{S}$37 still drive near-complete failure. The paper interprets this pattern as a robustness stratification in which RPID collapses earlier, PPO is somewhat stronger, and RecurrentPPO is more resilient at moderate confrontation levels (Guo et al., 29 Jul 2025).

The paper also reports entropy trends. At low confrontation levels, action entropy is generally higher, indicating more exploratory and less targeted behavior. As $\mathcal{S}$38 increases, entropy decreases, consistent with sharper adversarial specialization. Against stronger autonomous-vehicle policies such as RecurrentPPO, entropy remains comparatively higher, which the authors interpret as evidence that the examiner must deploy more varied tactics. This suggests that confrontation intensity and policy difficulty affect not only failure rates but also the strategic complexity of the generated tests (Guo et al., 29 Jul 2025).

Cross-scenario results support the claim that ExamPPO is not restricted to intersections. In the highway scenario with RPID, $\mathcal{S}$39 and $\mathcal{S}$40 produce DFR $\mathcal{S}$41, CSR $\mathcal{S}$42, collision $\mathcal{S}$43, and task success $\mathcal{S}$44, whereas $\mathcal{S}$45 yields DFR $\mathcal{S}$46, CSR $\mathcal{S}$47, collision $\mathcal{S}$48, autonomous-vehicle speed $\mathcal{S}$49 m/s, and task success $\mathcal{S}$50; $\mathcal{S}$51 yields DFR $\mathcal{S}$52, CSR $\mathcal{S}$53, collision $\mathcal{S}$54, autonomous-vehicle speed $\mathcal{S}$55 m/s, and task success $\mathcal{S}$56; and $\mathcal{S}$57 yields DFR $\mathcal{S}$58, CSR $\mathcal{S}$59, collision $\mathcal{S}$60, autonomous-vehicle speed $\mathcal{S}$61 m/s, and task success $\mathcal{S}$62. In the merge scenario, $\mathcal{S}$63 and $\mathcal{S}$64 again remain non-disruptive, while $\mathcal{S}$65, $\mathcal{S}$66, and $\mathcal{S}$67 progressively degrade performance, reaching DFR $\mathcal{S}$68, CSR $\mathcal{S}$69, collision $\mathcal{S}$70, autonomous-vehicle speed $\mathcal{S}$71 m/s, and task success $\mathcal{S}$72 at $\mathcal{S}$73. The effect sizes are smaller than at intersections, but the monotone degradation with increasing intensity remains present (Guo et al., 29 Jul 2025).

7. Limits, interpretation, and significance

The paper’s scope is deliberately controlled. All experiments are conducted in simulation, specifically a modified highway-env, across three structured scenarios. The action space is limited to three discrete longitudinal commands, which constrains the maneuver vocabulary available to the examiner. Several implementation and modeling details remain underspecified, including the exact shaping function $\mathcal{S}$74, the numerical reward weights, the complete combined PPO loss including value and entropy terms in a single expression, the exact episode horizon, hardware, and wall-clock cost. These omissions do not obscure the framework’s central design, but they do bound strict reproducibility and complicate interpretation of which reward components dominate in practice (Guo et al., 29 Jul 2025).

ExamPPO should therefore be read primarily as a framework for controllable adversarial evaluation rather than as a general solution to autonomous-driving verification. Its central contribution is procedural: it converts validation from passive replay or static falsification into a closed-loop examination process in which a learned surrounding vehicle can apply progressively stronger, behavior-aware tests. A plausible implication is that the framework is especially suited to comparative robustness studies, because the same scenario family and the same confrontation scale $\mathcal{S}$75–$\mathcal{S}$76 can be applied across heterogeneous autonomous-vehicle controllers. The paper explicitly identifies extension to more complex multi-agent traffic environments and to high-fidelity or real-world simulation platforms as future work (Guo et al., 29 Jul 2025).