PerlAD: Vector-Space RL for Autonomous Driving

• PerlAD is a pseudo-simulation-based reinforcement learning method that maps multi-camera inputs to planning actions, optimizing safety and efficiency in closed-loop driving.
• It replaces costly game-engine rendering with a vector-space pseudo-simulation, using a Prediction World Model to generate reactive agent trajectories.
• The method decouples planning by using imitation learning for lateral path generation and reinforcement learning for longitudinal speed control, achieving significant benchmark improvements.

Searching arXiv for the specified paper to ground the article in the current record. PerlAD is a pseudo-simulation-based reinforcement learning method for closed-loop end-to-end autonomous driving that is designed to learn policies mapping multi-camera observations $X$ to planning actions $A$ while optimizing driving metrics such as safety and efficiency rather than only open-loop imitation losses (Gao et al., 16 Mar 2026). The method is explicitly motivated by the mismatch between open-loop imitation objectives and closed-loop driving requirements, and by the rendering gap and computational cost associated with game-engine-based reinforcement learning. Its core design consists of three coupled elements: a rendering-free pseudo-simulation in vector space, a Prediction World Model (PWM) that generates reactive agent trajectories conditioned on the ego vehicle’s plan, and a hierarchical decoupled planner that uses imitation learning for lateral path generation and reinforcement learning for longitudinal speed optimization.

1. Research setting and problem definition

PerlAD addresses closed-loop end-to-end autonomous driving under a formulation in which the policy directly maps surround-view sensory inputs to planning actions. The stated objective is to optimize real driving metrics, specifically safety and efficiency, rather than relying purely on open-loop imitation losses. In the problem framing, conventional imitation learning minimizes an imitation error of the form $\|\hat a - a^*\|$, but this objective ignores interactive effects and causal dependencies. The reported consequence is that policies trained only on static datasets may drift or behave unsafely when exposed to feedback in deployment.

The method is also positioned against two further constraints. First, rendering realistic sensor inputs in game-engine simulators introduces input-domain mismatch, denoted as the rendering gap. Second, such simulators are computationally expensive for large-scale reinforcement learning. PerlAD therefore replaces rendered interaction with an offline, vector-space pseudo-simulation. A third challenge identified in the formulation is data sparsity for reinforcement learning: purely offline RL with logged data cannot capture dynamics beyond slow, static replay unless a dynamic world model is introduced.

This framing places PerlAD at the intersection of imitation learning, offline reinforcement learning, world modeling, and end-to-end planning. A plausible implication is that the method is not only an alternative optimizer for a fixed perception stack, but also a specific answer to the question of how to recover interactive closed-loop behavior from offline autonomous driving corpora without online simulator rollouts.

2. Vector-space pseudo-simulation

PerlAD defines a POMDP $(\mathcal{X,S,A,T,O,R},\gamma)$. In this specification, $\mathcal{X}$ is formed by surround-view images from $N_{\rm cam}=6$ cameras, $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$ contains decoupled lateral-path and longitudinal-speed actions, $\mathcal{S}$ is a vector-space state containing ego kinematics and agent trajectories, $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$ is the pseudo-simulation transition, $R$ is a reward function from collision, lane-keeping, progress, and distance terms, and the discount factor is $A$0.

The ego state $A$1 includes position $A$2, heading $A$3, and speed $A$4. Each agent state $A$5 has the same structure for a traffic participant, and the full state is

$A$6

The lateral action is a sequence of $A$7 future waypoint coordinates in the map frame. The longitudinal action is a discrete target speed

$A$8

with $A$9 levels up to $\|\hat a - a^*\|$0.

The pseudo-simulation transition is written as

$\|\hat a - a^*\|$1

where $\|\hat a - a^*\|$2. Ego dynamics are produced by a bicycle model with two PID controllers that track the lateral and longitudinal commands and generate a high-frequency ego trajectory over $\|\hat a - a^*\|$3 steps at $\|\hat a - a^*\|$4. Agent motion is derived from the PWM’s Top-1 predicted low-frequency path at $\|\hat a - a^*\|$5 with $\|\hat a - a^*\|$6, which is then interpolated to $\|\hat a - a^*\|$7 with constant-speed segments. The cumulative pseudo-simulation return is

$\|\hat a - a^*\|$8

with

$\|\hat a - a^*\|$9

This vector-space construction is central to the method’s rendering-free character. The intended effect is to preserve interactive planning structure while avoiding the cost and domain-shift liabilities of pixel-level simulator rendering.

3. Prediction World Model

The Prediction World Model is the component that converts a static offline corpus into a reactive pseudo-environment. Its inputs are structured scene queries from the end-to-end model: agent queries $(\mathcal{X,S,A,T,O,R},\gamma)$0, map queries $(\mathcal{X,S,A,T,O,R},\gamma)$1, and the ego plan, which consists of lateral waypoints and target speed. The PWM uses an autoregressive GRU to generate multi-modal trajectories for $(\mathcal{X,S,A,T,O,R},\gamma)$2 agents, conditioned on the ego plan.

The hidden-state initialization uses modified agent queries together with anchor embeddings and positional encoding. At each recurrent step, the GRU receives an ego-plan embedding formed from positional encoding of ego-plan increments. Each step produces predicted agent waypoints and a modality score $(\mathcal{X,S,A,T,O,R},\gamma)$3, after which the Top-1 modality is selected for simulation. Supervised training uses a loss composed of regression terms, specified as ADE/FDE, together with a classification term given by cross-entropy.

The functional role of the PWM is explicitly to bridge the gap between static datasets and dynamic closed-loop environments. Without such a model, offline replay cannot express reactive traffic behavior under counterfactual ego actions. With the PWM, neighboring agents become conditional on the ego plan inside the pseudo-simulation. This suggests that PerlAD treats trajectory prediction not as an auxiliary forecasting task, but as an operational mechanism for environment reconstruction.

4. Hierarchical decoupled planner

The planner is hierarchically decoupled into lateral and longitudinal subproblems. For lateral path planning, PerlAD uses imitation learning. The planner is initialized with queries $(\mathcal{X,S,A,T,O,R},\gamma)$4 and path anchors $(\mathcal{X,S,A,T,O,R},\gamma)$5, and forms a multi-modal query

$(\mathcal{X,S,A,T,O,R},\gamma)$6

A two-head MLP is then used, with a regression head for waypoint coordinates and a classification head for mode scores. The lateral imitation objective is

$(\mathcal{X,S,A,T,O,R},\gamma)$7

where $(\mathcal{X,S,A,T,O,R},\gamma)$8 is smooth-$(\mathcal{X,S,A,T,O,R},\gamma)$9 or $\mathcal{X}$0 between predicted and ground-truth waypoints, and $\mathcal{X}$1 is cross-entropy over the correct mode.

Longitudinal speed control is optimized with reinforcement learning. The RL state comprises features from camera encoders together with the selected lateral path $\mathcal{X}$2. The action is the discrete target speed $\mathcal{X}$3. The policy is written as $\mathcal{X}$4 and is parameterized by a classification head over an augmented query $\mathcal{X}$5. The objective is REINFORCE with standardized advantage: $\mathcal{X}$6

$\mathcal{X}$7

with $\mathcal{X}$8 sampled speeds per starting state and $\mathcal{X}$9.

The significance of this decomposition is stated directly in the method design: imitation learning is used where stable lateral trajectory generation is desired, whereas reinforcement learning is reserved for interactive speed control. A plausible implication is that the decomposition reduces the RL search space and transfers geometric path priors from logged data into the closed-loop planner.

5. Offline optimization procedure

Training proceeds in two stages. Stage 1 lasts 12 epochs with learning rate $N_{\rm cam}=6$0 and trains only the sparse perception backbone, specified as ResNet-50 plus sparse queries, with detection loss $N_{\rm cam}=6$1 and map segmentation loss $N_{\rm cam}=6$2. Stage 2 lasts 18 epochs with learning rate $N_{\rm cam}=6$3, AdamW, and weight decay $N_{\rm cam}=6$4; in this stage the backbone is frozen while transformer blocks, DeP, and PWM are jointly trained with

$N_{\rm cam}=6$5

Within each RL batch, the procedure samples a mini-batch of logged observations. For each sample, it initializes $N_{\rm cam}=6$6 from offline motion states, rolls out the pseudo-simulation by observing $N_{\rm cam}=6$7, predicting actions $N_{\rm cam}=6$8, generating reactive agent trajectories $N_{\rm cam}=6$9 via PWM, and simulating $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$0. It then accumulates $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$1, computes $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$2, and updates $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$3.

The reported optimization configuration uses AdamW, batch size $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$4, and $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$5 NVIDIA H20 GPUs. Additional dimensions are query size $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$6, $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$7, $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$8, $\mathcal{A}=\{a_{\rm lat}, a_{\rm lon}\}$9, and $\mathcal{S}$0. The entropy weight is $\mathcal{S}$1.

This training recipe characterizes PerlAD as a fully offline method: it uses offline datasets not merely for supervised pretraining, but also as the source of initial states for trial-and-error optimization inside pseudo-simulation.

6. Benchmarks and reported performance

PerlAD is evaluated on the Bench2Drive and DOS benchmarks, with closed-loop evaluation emphasized throughout (Gao et al., 16 Mar 2026). On Bench2Drive, the reported closed-loop metrics are Driving Score (DS), Success Rate (SR), Efficiency, and Comfortness, together with multi-ability SR metrics for Merging, Overtaking, Emergency Brake, Give Way, and Traffic Sign. The principal numerical result is DS $\mathcal{S}$2 versus $\mathcal{S}$3 for the prior RL method Raw2Drive, corresponding to $\mathcal{S}$4 DS, and SR $\mathcal{S}$5 versus $\mathcal{S}$6. On the Dev10 ablation suite, full PerlAD yields DS $\mathcal{S}$7, SR $\mathcal{S}$8, and CR $\mathcal{S}$9 collisions/100 m.

On the DOS benchmark, which is organized around safety-critical occlusion scenarios, the four scenarios are DOS 01 Parked Cars, DOS 02 Sudden Brake, DOS 03 Left Turn, and DOS 04 Red Light Infraction. The metric is Driving Score per scenario, averaged. PerlAD achieves average DS $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$0, reported as substantially above IL baselines, including UniAD $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$1, and RL baselines.

Benchmark	Metrics	Reported results
Bench2Drive	DS, SR, Efficiency, Comfortness; multi-ability SR	DS $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$2, SR $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$3; prior RL method Raw2Drive: DS $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$4, SR $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$5
Dev10 ablation suite	DS, SR, CR	DS $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$6, SR $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$7, CR $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$8 collisions/100 m
DOS	Average DS across four occlusion scenarios	Average DS $T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$9; UniAD $R$0

In methodological terms, these evaluations are used to support two distinct claims: first, that rendering-free offline RL can exceed a prior end-to-end RL baseline on a standard closed-loop driving benchmark; second, that the same framework retains reliability in occlusion-heavy safety-critical scenarios.

7. Limitations and prospective extensions

The reported strengths are threefold. The first is that rendering-free pseudo-simulation in vector space greatly accelerates RL for end-to-end driving without domain gap. The second is that the PWM provides reactive, conditional agent behavior and thereby closes the open-loop versus closed-loop gap. The third is that the hierarchical decoupled planner leverages imitation learning for stable lateral trajectories and reinforcement learning for interactive speed control (Gao et al., 16 Mar 2026).

The limitations are also stated explicitly. Offline data coverage limits generalization, and performance gains saturate when the reactive sample proportion is approximately greater than or equal to $R$1. Reward design still relies partially on a distance-to-ground-truth term, so fully data-driven reward modeling, including reward learning via human preferences, remains future work. In addition, the decoupled planning paradigm is specialized for urban, low-speed regimes; coupled planning for high-speed or non-urban scenarios is identified as unexplored.

The proposed extensions follow directly from these limitations. One is the introduction of a latent-space world model in the sensory feature domain to extrapolate beyond offline data. Another is replacement of the hand-crafted distance reward with learned preference models. A third is extension of the decoupled planner to a unified coupled planning policy that can cover a broader range of driving conditions. Collectively, these directions suggest that PerlAD is not presented as a closed endpoint, but as a particular instantiation of offline closed-loop end-to-end driving in which pseudo-simulation, conditional world modeling, and structured policy decomposition are jointly optimized.