DriveDPO: Safety-Driven Autonomous Driving

• DriveDPO is a safety-driven, end-to-end autonomous driving framework that integrates human imitation with rule-based safety scores via unified policy distribution distillation.
• It employs a two-stage training process, starting with unified distillation to fuse imitation and safety metrics, and then refining policy preferences with iterative Safety DPO.
• The framework achieves a state-of-the-art PDMS of 90.0 on NAVSIM by addressing imitation learning safety pitfalls through simulation-based negative sampling.

DriveDPO is a Safety Direct Preference Optimization Policy Learning framework for end-to-end autonomous driving that addresses a central safety problem in policies trained purely by imitation learning: such policies can look human-like but be unsafe (Shang et al., 22 Sep 2025). In the formulation reported for NAVSIM, the model predicts future trajectories directly from raw perception inputs, bypassing traditional modular pipelines, but replaces purely geometric supervision with a two-stage procedure that first distills a unified policy distribution from human imitation similarity and rule-based safety scores, and then applies iterative trajectory-level preference alignment through Safety DPO (Shang et al., 22 Sep 2025). The reported result is a new state-of-the-art PDMS of 90.0 on NAVSIM, together with qualitative evidence of safer and more reliable driving behaviors (Shang et al., 22 Sep 2025).

1. Safety problem and conceptual motivation

DriveDPO is motivated by the observation that mainstream end-to-end driving methods trained via imitation learning suffer from critical safety limitations because they fail to distinguish between trajectories that appear human-like but are potentially unsafe (Shang et al., 22 Sep 2025). In the reported setting, imitation learning typically minimizes a symmetric geometric distance between predicted and expert trajectories. Small deviations that keep imitation losses low can still produce lane departures or collisions, and symmetric losses such as MSE penalize left/right or early/late deviations equally even when their safety implications differ (Shang et al., 22 Sep 2025).

The framework is also positioned against recent score-based approaches that regress multiple rule-driven scores, such as collision risk or off-road penalty, per trajectory anchor. According to the paper, these methods decouple supervision from direct policy distribution optimization: score heads are trained independently for each candidate, and the planner is not directly optimized to produce a safety-aligned action distribution (Shang et al., 22 Sep 2025). DriveDPO is introduced specifically to remove this indirectness by directly supervising and then preference-aligning the policy distribution itself.

The conceptual connection is to RLHF-style alignment, but adapted to autonomous driving. Instead of human preference labels, DriveDPO derives preferences from safety rules and simulator feedback, forming trajectory-level pairs that compare safety-aligned desirability (Shang et al., 22 Sep 2025). The paper argues that Direct Preference Optimization is well-suited because it optimizes the policy under a KL constraint directly from preferences, avoiding reward maximization and value estimation complications (Shang et al., 22 Sep 2025). A plausible implication is that the method is intended to preserve the stability of supervised policy learning while making pairwise safety comparisons operational at the trajectory level.

2. Task formulation, anchor vocabulary, and architecture

The task is defined over raw observation $O$, with the model predicting $T$ future steps per trajectory, each step $(x_t, y_t, \theta_t)$ for $t = 1,\ldots,T$ (Shang et al., 22 Sep 2025). DriveDPO uses VADv2’s anchor vocabulary to discretize the action space. A set of $N$ anchors is built via k-means over human trajectories:

$V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$

The policy distribution is categorical over anchors,

$$\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,$$

and the final planned trajectory is selected by

$a^* = \arg\max_{a_i \in V} \pi_\theta(a_i).$

The model input comprises raw perception from multi-view cameras and LiDAR together with ego state and navigation command (Shang et al., 22 Sep 2025). In the reported implementation, the inputs are three forward-facing cameras concatenated to a $1024\times 256$ image, a BEV LiDAR feature map of size $64\times 64$, ego speed, acceleration, and navigation command (Shang et al., 22 Sep 2025). The backbone follows a ResNet-34-based Transfuser fusion architecture for camera and LiDAR (Shang et al., 22 Sep 2025).

Anchors are tokenized and cross-attend to scene tokens for context-aware decision-making. The anchor representation uses NeRF-style Fourier positional encoding,

$T$0

followed by

$T$1

Scene tokens from the fusion backbone are denoted

$T$2

A cross-attention decoder produces context-conditioned anchor representations,

$T$3

and a policy head with softmax yields

$T$4

The reported training configuration uses $T$5 anchors, a future horizon of $T$6 waypoints over 4 seconds, and positional encoding with $T$7 frequency bands (Shang et al., 22 Sep 2025). This architecture makes the planner explicitly distributional over a predefined anchor vocabulary rather than directly regressing a single trajectory.

3. Unified policy distribution distillation

The first stage of DriveDPO is Unified Policy Distribution Distillation, which directly supervises the policy distribution by fusing an imitation similarity score with a rule-based safety score into a single target distribution over trajectory anchors (Shang et al., 22 Sep 2025). The objective is to make the learned policy match this distilled target distribution via KL divergence.

For imitation similarity, given a human reference trajectory $T$8, the method defines a normalized similarity across anchors through a softmax over negative Euclidean distances:

$T$9

This quantity acts as the imitation similarity score $(x_t, y_t, \theta_t)$0 evaluated at $(x_t, y_t, \theta_t)$1 (Shang et al., 22 Sep 2025).

For rule-based safety, each anchor is forward-simulated in the NAVSIM simulator and scored using rule-driven indicators: No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C) (Shang et al., 22 Sep 2025). These are aggregated into the PDM Score (PDMS),

$(x_t, y_t, \theta_t)$2

Each anchor receives a score by forward simulation,

$(x_t, y_t, \theta_t)$3

which acts as the safety score $(x_t, y_t, \theta_t)$4 evaluated at $(x_t, y_t, \theta_t)$5 (Shang et al., 22 Sep 2025).

The distilled target distribution log-transforms both Sim and PDMS to accentuate differences among unsafe anchors and fuses them with weights $(x_t, y_t, \theta_t)$6 and $(x_t, y_t, \theta_t)$7 through a softmax competition:

$(x_t, y_t, \theta_t)$8

Equivalently,

$(x_t, y_t, \theta_t)$9

Training minimizes

$t = 1,\ldots,T$0

The paper characterizes this stage as directly optimizing the policy distribution rather than regressing separate score heads, aligning anchors competitively under unified human-and-safety supervision (Shang et al., 22 Sep 2025). In the reported training details, the distillation weights are fixed to $t = 1,\ldots,T$1 for imitation and $t = 1,\ldots,T$2 for safety (Shang et al., 22 Sep 2025). This weighting reflects the method’s explicit prioritization of safety in the fused target distribution.

4. Safety DPO and trajectory-level preference alignment

The second stage is Safety DPO, formulated as iterative trajectory-level preference alignment (Shang et al., 22 Sep 2025). Its purpose is to resolve the multi-objective tension between imitation and safety and to sharpen safety responsiveness after the initial unified distillation stage.

Preference generation proceeds by sampling $t = 1,\ldots,T$3 trajectories from the current policy $t = 1,\ldots,T$4 for each input $t = 1,\ldots,T$5 and selecting the preferred trajectory $t = 1,\ldots,T$6 as the highest score under $t = 1,\ldots,T$7 (Shang et al., 22 Sep 2025). The dispreferred trajectory $t = 1,\ldots,T$8 is then selected using strategies that emphasize “unsafe yet human-like” negatives. Two strategies are specified:

• Imitation-based rejection:

$t = 1,\ldots,T$9

• Distance-based rejection:

$N$0

The default strategy is imitation-based rejection, and $N$1 is a safety threshold (Shang et al., 22 Sep 2025). In the reported implementation, $N$2 candidates are sampled per iteration and $N$3 (Shang et al., 22 Sep 2025).

With a fixed reference policy $N$4, defined as the distilled pretrain, the per-sample DPO loss is

$N$5

where $N$6 is sigmoid and $N$7 is a temperature (Shang et al., 22 Sep 2025). In expectation form over inputs and preference pairs, the paper writes

$N$8

To prevent distributional drift, DriveDPO adds explicit KL regularization and continues to apply the Stage-1 KL loss as auxiliary supervision:

$N$9

with $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$0 (Shang et al., 22 Sep 2025). The paper states that continuing the unified KL term stabilizes optimization while preserving safety alignment (Shang et al., 22 Sep 2025). In the reported configuration, $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$1, and regularization consists of the explicit KL term against $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$2 together with the auxiliary $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$3 term maintained during DPO (Shang et al., 22 Sep 2025).

The algorithmic summary reported in the paper is: anchor construction by k-means over human trajectories; perception encoding with a Transfuser backbone and Fourier-feature anchor tokenization; policy prediction over anchors; unified distillation through $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$4 and $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$5; and iterative Safety DPO using sampled anchors, imitation-based rejection under $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$6, and updates with $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$7 (Shang et al., 22 Sep 2025).

5. Evaluation, metrics, and empirical results

DriveDPO is evaluated on NAVSIM, which combines real sensor data with non-interactive simulation built on OpenScene/nuPlan and provides eight high-resolution camera views and fused point clouds at 2 Hz (Shang et al., 22 Sep 2025). Models take 1.5 seconds of history and predict 8 future waypoints over 4 seconds (Shang et al., 22 Sep 2025). The official evaluation metric is PDMS:

$V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$8

The constituent quantities are No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C) (Shang et al., 22 Sep 2025). The paper notes that PDMS balances safety and progress; the multiplicative $V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.$9 suppresses trajectories with collisions or off-road violations, while the weighted sum scales progress, TTC, and comfort (Shang et al., 22 Sep 2025).

DriveDPO achieves a reported state-of-the-art PDMS of 90.0 on NAVSIM (Shang et al., 22 Sep 2025). The full model metrics are NC = 98.5, DAC = 98.1, EP = 84.3, TTC = 94.8, and C = 99.9 (Shang et al., 22 Sep 2025). The paper compares these results with strong baselines: DiffusionDrive, an imitation-based approach, with PDMS = 88.1, TTC = 94.7, DAC = 96.2, and EP = 82.2; and WOTE, a score-based method, with PDMS = 88.0, NC = 98.4, DAC = 96.6, EP = 81.7, TTC = 94.5, and C = 99.9 (Shang et al., 22 Sep 2025). On these figures, DriveDPO improves PDMS by +1.9 over DiffusionDrive and +2.0 over WOTE, and also surpasses a privileged PDM-closed planner with PDMS = 89.1 that uses ground-truth perception (Shang et al., 22 Sep 2025).

The stage-wise results are also reported. The unified distillation pretrain alone yields PDMS = 88.8 and notable gains on EP and DAC versus prior methods, while Safety DPO adds further improvements in NC (+0.6), DAC (+0.8), and TTC (+1.2) (Shang et al., 22 Sep 2025). In the ablations, unified supervision over the policy distribution outperforms independent score regression: ID-1 reaches PDMS 88.8, compared with 87.3 for score regression (ID-2) (Shang et al., 22 Sep 2025). Single-source supervision performs worse: imitation-only (ID-3) yields PDMS 82.5 with poor DAC, while rule-only (ID-4) yields PDMS 87.2, showing aggressive behaviors harming NC and TTC (Shang et al., 22 Sep 2025).

The DPO preference-construction ablation isolates the effect of negative sampling. Without DPO, the model achieves PDMS 88.8; vanilla preference selection using highest versus lowest $$\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,$$0 yields 89.3; distance-based rejection reaches 89.7; and imitation-based rejection achieves the best PDMS 90.0 (Shang et al., 22 Sep 2025). The paper interprets this as indicating that targeting “human-like but unsafe” negatives is most effective (Shang et al., 22 Sep 2025).

DriveDPO is also evaluated in the closed-loop Bench2Drive benchmark, where it attains Driving Score 62.02, Success Rate 30.62%, Efficiency 166.80, and Comfortness 26.79, outperforming representative end-to-end baselines including TCP with Driving Score 59.90 and Success Rate 30.00%, and UniAD with Driving Score 45.81 and Success Rate 16.36% (Shang et al., 22 Sep 2025).

6. Training regime, positioning, and limitations

The reported training setup uses AdamW with learning rate $$\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,$$1, 30 epochs of unified distillation followed by 10 epochs of Safety DPO fine-tuning, batch size 16 per GPU, and 6 NVIDIA L20 GPUs (Shang et al., 22 Sep 2025). The paper identifies the anchor vocabulary size $$\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,$$2 and DPO sampling with $$\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,$$3 per input as computational drivers (Shang et al., 22 Sep 2025). It also notes that exact inference latency is not reported, although the architecture is described as consistent with real-time planners using compact backbones such as ResNet-34 (Shang et al., 22 Sep 2025).

The reproducibility details are partial. The work uses NAVSIM Navtrain with approximately 103k samples and Navtest with approximately 12k, under Transfuser-like settings, and Bench2Drive training on a base subset with evaluation on 220 routes (Shang et al., 22 Sep 2025). Seeds and code availability are not specified (Shang et al., 22 Sep 2025).

Within the broader literature on preference alignment for driving, DriveDPO is closely related to later DPO-based formulations that use different preference sources. VL-DPO, for example, employs a frozen vision-LLM as a zero-shot reasoner to generate pairwise trajectory preferences from rollouts and finetunes a motion forecasting model via DPO, improving RFS and ADE on WOD-E2E without requiring the VLM at inference time (Xu et al., 19 May 2026). By contrast, DriveDPO derives preferences from simulator-based safety scores and imitation proximity rather than VLM judgments (Shang et al., 22 Sep 2025, Xu et al., 19 May 2026). A different line of work applies DPO to diffusion-based traffic scenario generation, using guide scores to construct pairwise preferences for scenario samples and fine-tuning a multi-guided diffusion model with a frozen reference model (Yu et al., 14 Feb 2025). This suggests that DPO has become a general mechanism for aligning either driving policies or scenario generators to structured traffic preferences, but the objects being aligned differ across these works.

The limitations identified for DriveDPO are specific. First, the method relies on PDMS, which is described as a predefined weighted composite that may miss nuanced risks in complex interactions; the paper proposes that more expressive safety evaluators or learned risk models could enrich preference construction (Shang et al., 22 Sep 2025). Second, preferences come from high-fidelity simulation and rule design, which can be limited in coverage and availability; the paper suggests mining latent preferences from large corpora of historical trajectories to reduce simulator dependence and enable weakly or self-supervised safety alignment (Shang et al., 22 Sep 2025). Third, although NAVSIM mixes real-world sensors with simulation, domain gaps remain, and robustness to unseen layouts, weather, and sensor noise is identified as an ongoing challenge (Shang et al., 22 Sep 2025).

Taken together, DriveDPO defines a policy-learning framework in which the action distribution is first shaped by a fused human-and-safety target distribution and then refined by pairwise preference updates that explicitly penalize “human-like but unsafe” candidates (Shang et al., 22 Sep 2025). The empirical record reported for NAVSIM and Bench2Drive indicates that this combination of unified distillation and Safety DPO improves NC, DAC, and TTC while producing a state-of-the-art PDMS of 90.0 on NAVSIM (Shang et al., 22 Sep 2025).