- The paper defines post-training as refinement beyond imitation learning and organizes methods into distillation, preference alignment, reinforcement learning, and test-time refinement to improve closed-loop driving behavior.
- The paper shows that effective reinforcement learning depends on informative rollouts with behavioral diversity and meaningful reward variation, while distillation, preference data, world models, and recovery examples help address safety, interaction, and distribution-shift challenges.
- The paper finds that open-loop metrics and saturated benchmarks can hide compounding errors and interaction failures, motivating harder closed-loop evaluation, component-level analysis, real-world latency reporting, and continuous feedback from takeovers and near-misses.
Motivation and scope
End-to-end autonomous driving models—spanning trajectory-generative planners and Vision-Language-Action (VLA) models—map multimodal sensor inputs directly to future trajectories or maneuvers. The dominant training signal for these models remains open-loop imitation learning on expert demonstrations, yet the authors argue this is structurally insufficient for driving. Two failure modes motivate their argument: in closed-loop execution, small execution errors compound over time (a minor lane-centering deviation can eventually push the vehicle out of the drivable area), and long-horizon qualities such as comfort, safety margins, and interaction appropriateness are invisible to pointwise displacement losses—two trajectories can have nearly identical ADE with respect to an expert, yet differ drastically in smoothness and risk. Recovery behaviors, moreover, are scarce in expert logs by construction.
The paper's contribution is a unified definition and taxonomy of post-training for driving: any refinement stage applied after an initial imitation-learned (or SFT) policy πθ0​​ has been obtained, using supervision beyond offline demonstrations. The survey organizes a large and previously fragmented literature into four families distinguished by the form of supervision: distillation, preference-based alignment, reinforcement learning (RL), and test-time refinement. A shared abstraction is proposed in which post-training minimizes a task-specific loss over carefully selected observations o~ and heterogeneous supervision signals, subject to staying within a neighborhood B(θ0​) of the imitation policy, typically via a KL-type regularizer toward a reference policy. Test-time refinement is treated as a separate family because it selects among fixed-policy outputs rather than updating parameters.
Distillation
Distillation refines the policy with dense targets from a stronger teacher—an expert planner, a larger VLM, or an exponential-moving-average copy of the policy itself. The loss is instantiated as a mismatch between student outputs and teacher signals (KL divergence for distribution matching, regression or cross-entropy for pointwise labels), plus optional regularization. Representative systems include CRAFT, which regularizes updates with a KL term toward an EMA reference during reward-driven fine-tuning; PaIR-Drive, which runs a parallel imitation branch alongside an RL branch; RAPID, which distills an RL teacher with online adaptation; VLM-AD, which distills VLM-generated reasoning and action annotations through auxiliary heads; CoPhy, which distills language-aligned reasoning into the action policy; and Found-RL, which combines foundation-model distillation with RL.
The survey emphasizes a data-design insight: teacher labels are most useful when applied to states visited by the student during its own rollouts, following the DAgger principle, rather than to a fixed expert dataset. This directly targets the distribution shift that causes error compounding. In most systems, however, the teacher signal does not drive the update alone; it stabilizes a reward-driven RL update, preserving competence acquired during SFT while allowing denser supervision than reward ranking alone.
Preference-based alignment
Preference-based supervision uses relative comparisons between candidate behaviors, avoiding explicit reward-model training. The canonical instantiation is the DPO loss, which increases the likelihood gap between a preferred and a rejected trajectory relative to a reference policy. The survey identifies pair construction as the central design choice, since preference pairs are simultaneously the data and the supervision. DriveDPO's Safety DPO combines imitation similarity with rule-based safety scores, preventing the model from preferring human-like but unsafe trajectories; VL-DPO uses a VLM to auto-generate preference pairs from rollouts, reducing human annotation; CSN applies Plackett–Luce DPO with NLL regularization over ranked candidate trajectories; CPO++ injects counterfactual perturbations to suppress spurious correlations; TakeAD builds pairs from expert takeover and disengagement events; and Drive My Way aligns policies with individual driving habits and natural-language intent.
The authors state a clear limitation: the effectiveness of this family depends critically on whether the compared pairs expose meaningful behavioral trade-offs. If candidate trajectories do not differ along dimensions such as comfort or safety, improvements may be moderate or negligible.
Reinforcement learning post-training
RL is the most developed family, and the survey structures it around three design axes: reward construction, rollout collection, and optimization algorithms. It is careful to note that driving is not a direct transfer of LLM post-training practice—driving quality is multi-dimensional, long-horizon, and context-dependent, involving trade-offs among collision avoidance, drivable-area compliance, comfort, progress, and rule compliance.
Reward design. Rule-based rewards convert observable rollout properties (collision, off-road, TTC, progress, smoothness) into scalars; benchmark scores such as NAVSIM's PDMS and EPDMS are frequently used directly. These are interpretable and reproducible but often miss interaction semantics, e.g., whether a lane change is natural or a yield is well timed. Learned rewards address qualities that resist fixed rules: VLM critics (DriveCritic, trained from pairwise human preferences), actor–critic critics (VDRive), and RLVR-style verifiable rewards (LaViPlan). World-model-coupled rewards evaluate actions by predicted future states (CoPhy, DreamerAD, FLARE, OmniDreams, MAPLE). Counterfactual rewards compare trajectories with nearby alternatives to densify supervision when failures are rare (CRAFT). Reasoning-quality rewards supervise the chain-of-thought itself: Alpamayo-R1 penalizes explanations inconsistent with the executed action; Drive-R1 combines trajectory accuracy, meta-action correctness, repetition, and format terms; OmniDrive-R1 adds CLIP-based ROI grounding rewards. A contrasting result is NoRD, which shows GRPO-style optimization can improve a VLA driving policy without explicit reasoning at all.
Rollout collection. The survey makes an important diagnostic point about group-relative methods: if all sampled candidates receive similar rewards, advantages collapse and the update carries no signal. SFT policies tend to stay close to demonstrations, so reranking cannot recover trajectories never sampled. Useful groups therefore require both behavioral diversity in the output space—achieved via intent-conditioned flow heads (DIAL), trajectory-anchor vocabularies (DriveAnchor), or discrete action codebooks (ReflectDrive)—and quality spread, enforced by tree-structured sampling (PaIR-Drive) or variance-gated updates (SCORP). A complementary line constructs negative and recovery data: SafeAlign-VLA builds positive/negative pairs from risky scenes, SpanVLA adds negative and recovery samples, TakeVLA mines human takeover data, and ELF-VLA converts all-zero long-tail rewards into failure-diagnostic feedback. World models supply imagined continuations under counterfactual actions, with MAPLE running multi-agent latent rollouts to create interaction cases absent from the data.
Optimization. GRPO-style group-relative objectives dominate, applied to tokenized action/trajectory policies (AutoVLA, NoRD, AlphaDrive), reasoning-oriented rewards (AutoDrive-P³, Drive-R1, OmniDrive-R1), and hybrid reasoning modes (DriveAgent-R1's Mode-Partitioned GRPO). Beyond autoregressive decoding, Flow-GRPO extends group-relative optimization to flow-matching generators, and WAM-Diff uses GSPO with a masked diffusion policy. Where a learned critic exists, PPO or actor–critic updates are used (IRL-VLA's Reward World Model, VDRive's diffusion head).
Test-time refinement
The final family keeps πθ0​​ fixed and selects among sampled candidates via a verifier score, or applies self-reflection to revise unsafe outputs. DriveCritic's human-aligned evaluator can serve as an inference-time reranker; ReflectDrive searches over a discrete action codebook to replace unsafe action tokens; AutoDrive-R² incorporates self-reflection into its reasoning policy; C-CoT introduces counterfactual chain-of-thought at inference time. The survey states a sharp boundary condition: Best-of-N only helps if the base policy samples a good trajectory; otherwise the reranker merely chooses among poor options and performance saturates quickly. Test-time refinement is thus a complement to, not a substitute for, parameter-updating post-training—a conclusion supported by Seong et al.'s combination of RL post-training with test-time scaling. A practical constraint specific to driving is strict on-vehicle latency: unlike LLM test-time scaling, inference cannot afford arbitrary extra computation, motivating efficiency-oriented designs such as Fast-dDrive's block-diffusion decoding.
Evaluation protocols and their limits
The survey's evaluation analysis is arguably as valuable as its taxonomy. Open-loop evaluation on nuScenes and WOD-E2E (ADE/FDE, collision rate, RFS) is cheap and reproducible but cannot capture accumulating errors, multi-agent interaction, or recovery—precisely the quantities post-training targets. The authors cite evidence that the open-loop-to-closed-loop gap stems from observational domain shift and objective mismatch, meaning standard open-loop protocols can hide closed-loop failures entirely.
On the closed-loop side, the survey is explicit that NAVSIM v1 is one-shot and non-reactive and is best viewed as open-loop evaluation with simulation-based metrics; NAVSIM v2's two-stage pseudo-simulation and Bench2Drive's fully closed-loop CARLA protocol with reactive traffic are better aligned with post-training goals. Reported results show strong headline numbers—EvaDrive at 94.9 PDMS, TakeVLA at 89.72 DS and 73.73 SR on Bench2Drive, AutoVLA at 78.84 DS—but the survey identifies benchmark saturation as a central problem: PDMS/EPDMS scores now cluster in a narrow band, making single headline scores poor at separating methodological progress from random variation, and recommending harder splits such as navhard with component-level analysis. Two further gaps are noted: real-vehicle validation exists in a handful of works (Alpamayo-R1 with 99 ms end-to-end latency, HDP, DriveAnchor with 2.06 ms inference) but is not comparable across papers due to differing platforms and metrics; and inference cost is chronically under-reported despite being measurable without a vehicle, with BPF demonstrating up to 27% higher decoding throughput on Jetson AGX Orin through RL-time pruning.
Open problems
The survey consolidates open problems that cut across families. First, reward and exploration are a coupled design problem: composite rewards are typically hand-tuned with unreported gating rules and can be gamed on easy components, while a reward can guide learning only if sampled rollouts differ in reward-relevant ways. Second, world-model signals must be behavior-relevant rather than merely visually realistic: the key requirement is fidelity to the factors that determine driving rewards—collisions, agent reactions, route progress, rule violations—not pixel-level fidelity. Third, current post-training is largely a one-off refinement over fixed datasets; what is missing is a systematic data-feedback loop in which deployed policies expose takeovers, near-misses, and rule violations that are converted into supervision and re-evaluated on the same hard cases. Fourth, the field's heavy reliance on GRPO-based optimization may be limiting, since driving offers more structured signals (temporal, safety-critical, interaction-localized) than a single scalar per rollout; the authors pose the question of localized feedback around lane changes, yielding, braking, and near-collision events. Finally, the survey explicitly declines to crown a winning algorithm family, framing the central question as which learning signal best matches each driving problem.
Conclusion
This survey provides a coherent organizing framework for post-training in end-to-end autonomous driving, defining it as a distinct refinement stage whose objective is closed-loop behavior rather than offline fit, and taxonomizing methods by supervision form into distillation, preference alignment, RL, and test-time refinement. Its most useful contributions for practitioners are the diagnosis of rollout informativeness as the binding constraint on group-relative RL, the saturation analysis of closed-loop benchmarks, and the argument that evaluation—not optimization—is becoming the bottleneck for the field.
