---
title: 'DriveDPO: Safety-Driven Autonomous Driving'
url: https://www.emergentmind.com/topics/drivedpo
type: topic
---

# DriveDPO: Safety-Driven Autonomous Driving

DriveDPO is a Safety Direct Preference Optimization Policy Learning framework for end-to-end autonomous driving that addresses a central safety problem in policies trained purely by imitation learning: such policies can look human-like but be unsafe [2509.17940]. In the formulation reported for NAVSIM, the model predicts future trajectories directly from raw perception inputs, bypassing traditional modular pipelines, but replaces purely geometric supervision with a two-stage procedure that first distills a unified policy distribution from human imitation similarity and rule-based safety scores, and then applies iterative trajectory-level preference alignment through Safety DPO [2509.17940]. The reported result is a new state-of-the-art PDMS of 90.0 on NAVSIM, together with qualitative evidence of safer and more reliable driving behaviors [2509.17940].

## 1. Safety problem and conceptual motivation

DriveDPO is motivated by the observation that mainstream end-to-end driving methods trained via imitation learning suffer from critical safety limitations because they fail to distinguish between trajectories that appear human-like but are potentially unsafe [2509.17940]. In the reported setting, imitation learning typically minimizes a symmetric geometric distance between predicted and expert trajectories. Small deviations that keep imitation losses low can still produce lane departures or collisions, and symmetric losses such as MSE penalize left/right or early/late deviations equally even when their safety implications differ [2509.17940].

The framework is also positioned against recent score-based approaches that regress multiple rule-driven scores, such as collision risk or off-road penalty, per trajectory anchor. According to the paper, these methods decouple supervision from direct policy distribution optimization: score heads are trained independently for each candidate, and the planner is not directly optimized to produce a safety-aligned action distribution [2509.17940]. DriveDPO is introduced specifically to remove this indirectness by directly supervising and then preference-aligning the policy distribution itself.

The conceptual connection is to RLHF-style alignment, but adapted to autonomous driving. Instead of human preference labels, DriveDPO derives preferences from safety rules and simulator feedback, forming trajectory-level pairs that compare safety-aligned desirability [2509.17940]. The paper argues that Direct Preference Optimization is well-suited because it optimizes the policy under a KL constraint directly from preferences, avoiding reward maximization and value estimation complications [2509.17940]. A plausible implication is that the method is intended to preserve the stability of supervised policy learning while making pairwise safety comparisons operational at the trajectory level.

## 2. Task formulation, anchor vocabulary, and architecture

The task is defined over raw observation $O$, with the model predicting $T$ future steps per trajectory, each step $(x_t, y_t, \theta_t)$ for $t = 1,\ldots,T$ [2509.17940]. DriveDPO uses VADv2’s anchor vocabulary to discretize the action space. A set of $N$ anchors is built via k-means over human trajectories:

$$
V = \{a_i \in \mathbb{R}^{T\times 3}\}_{i=1}^N.
$$

The policy distribution is categorical over anchors,

$$
\pi_\theta(a_i) = p_i,\quad i=1,\ldots,N,\qquad \sum_{i=1}^N p_i = 1,
$$

and the final planned trajectory is selected by

$$
a^* = \arg\max_{a_i \in V} \pi_\theta(a_i).
$$

The model input comprises raw perception from multi-view cameras and LiDAR together with ego state and navigation command [2509.17940]. In the reported implementation, the inputs are three forward-facing cameras concatenated to a $1024\times 256$ image, a BEV LiDAR feature map of size $64\times 64$, ego speed, acceleration, and navigation command [2509.17940]. The backbone follows a ResNet-34-based Transfuser fusion architecture for camera and LiDAR [2509.17940].

Anchors are tokenized and cross-attend to scene tokens for context-aware decision-making. The anchor representation uses NeRF-style Fourier positional encoding,

$$
\Gamma = \gamma(V) = (\sin(2^0 \pi V), \cos(2^0 \pi V), \ldots, \sin(2^{L-1} \pi V), \cos(2^{L-1} \pi V)) \in \mathbb{R}^{N\times T\times 6L},
$$

followed by

$$
E_{\text{anchor}} = \text{MLP}(\Gamma) \in \mathbb{R}^{N\times T\times d}.
$$

Scene tokens from the fusion backbone are denoted

$$
E_{\text{scene}} \in \mathbb{R}^{M\times d}.
$$

A cross-attention decoder produces context-conditioned anchor representations,

$$
\tilde{E}_{\text{anchor}} = \text{TransformerDecoder}(Q = E_{\text{anchor}}, K = E_{\text{scene}}, V = E_{\text{scene}}) \in \mathbb{R}^{N\times d},
$$

and a policy head with softmax yields

$$
\pi_\theta(a_i) = \text{Softmax}(\text{MLP}(\tilde{E}_{\text{anchor}}))_i,\quad i=1,\ldots,N.
$$

The reported training configuration uses $N = 8192$ anchors, a future horizon of $T = 8$ waypoints over 4 seconds, and positional encoding with $L = 10$ frequency bands [2509.17940]. This architecture makes the planner explicitly distributional over a predefined anchor vocabulary rather than directly regressing a single trajectory.

## 3. Unified policy distribution distillation

The first stage of DriveDPO is Unified Policy Distribution Distillation, which directly supervises the policy distribution by fusing an imitation similarity score with a rule-based safety score into a single target distribution over trajectory anchors [2509.17940]. The objective is to make the learned policy match this distilled target distribution via KL divergence.

For imitation similarity, given a human reference trajectory $\hat{a} \in \mathbb{R}^{T\times 3}$, the method defines a normalized similarity across anchors through a softmax over negative Euclidean distances:

$$
\text{Sim}(a_i) = \text{Softmax}(-\|a_i - \hat{a}\|_2),\quad i=1,\ldots,N.
$$

This quantity acts as the imitation similarity score $s_{\text{im}}(y|x)$ evaluated at $y = a_i$ [2509.17940].

For rule-based safety, each anchor is forward-simulated in the NAVSIM simulator and scored using rule-driven indicators: No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C) [2509.17940]. These are aggregated into the PDM Score (PDMS),

$$
\text{PDMS} = NC \times DAC \times (5 \times EP + 5 \times TTC + 2 \times C)/12.
$$

Each anchor receives a score by forward simulation,

$$
\text{PDMS}(a_i) = \text{ForwardSimulation}(a_i),\quad i=1,\ldots,N,
$$

which acts as the safety score $s_{\text{safe}}(y|x)$ evaluated at $y = a_i$ [2509.17940].

The distilled target distribution log-transforms both Sim and PDMS to accentuate differences among unsafe anchors and fuses them with weights $w_1$ and $w_2$ through a softmax competition:

$$
p_{\text{unified}}(a_i) = \text{Softmax}(w_1 \cdot \log(\text{Sim}(a_i)) + w_2 \cdot \log(\text{PDMS}(a_i))),\quad i=1,\ldots,N.
$$

Equivalently,

$$
p_{\text{unified}}(a_i) \propto \text{Sim}(a_i)^{w_1}\text{PDMS}(a_i)^{w_2}.
$$

Training minimizes

$$
L_{\text{unified}} = KL(p_{\text{unified}} \,\|\, \pi_\theta).
$$

The paper characterizes this stage as directly optimizing the policy distribution rather than regressing separate score heads, aligning anchors competitively under unified human-and-safety supervision [2509.17940]. In the reported training details, the distillation weights are fixed to $w_1 = 0.1$ for imitation and $w_2 = 1.0$ for safety [2509.17940]. This weighting reflects the method’s explicit prioritization of safety in the fused target distribution.

## 4. Safety DPO and trajectory-level preference alignment

The second stage is Safety DPO, formulated as iterative trajectory-level preference alignment [2509.17940]. Its purpose is to resolve the multi-objective tension between imitation and safety and to sharpen safety responsiveness after the initial unified distillation stage.

Preference generation proceeds by sampling $K$ trajectories from the current policy $\pi_\theta$ for each input $x$ and selecting the preferred trajectory $a_w$ as the highest score under $p_{\text{unified}}$ [2509.17940]. The dispreferred trajectory $a_l$ is then selected using strategies that emphasize “unsafe yet human-like” negatives. Two strategies are specified:

- **Imitation-based rejection**:
  $$
  a_l = \arg\min_{a_i} \|a_i - \hat{a}\|_2 \quad \text{subject to } \text{PDMS}(a_i) < \tau,\quad i=1,\ldots,K.
  $$

- **Distance-based rejection**:
  $$
  a_l = \arg\min_{a_i} \|a_i - a_w\|_2 \quad \text{subject to } \text{PDMS}(a_i) < \tau,\quad i=1,\ldots,K.
  $$

The default strategy is imitation-based rejection, and $\tau$ is a safety threshold [2509.17940]. In the reported implementation, $K = 1024$ candidates are sampled per iteration and $\tau = 0.3$ [2509.17940].

With a fixed reference policy $\pi_{\text{Ref}}$, defined as the distilled pretrain, the per-sample DPO loss is

$$
L_{\text{DPO}}(\theta) = - \log \sigma\!\left( \beta \left( \log \frac{\pi_\theta(a_w)}{\pi_{\text{Ref}}(a_w)} - \log \frac{\pi_\theta(a_l)}{\pi_{\text{Ref}}(a_l)} \right) \right),
$$

where $\sigma(\cdot)$ is sigmoid and $\beta$ is a temperature [2509.17940]. In expectation form over inputs and preference pairs, the paper writes

$$
L_{\text{DPO}}(\theta) = - \mathbb{E}_{x,(a_w,a_l)} \left[ \log \sigma \left( \beta \left( (\log \pi_\theta(a_w|x) - \log \pi_{\text{Ref}}(a_w|x)) - (\log \pi_\theta(a_l|x) - \log \pi_{\text{Ref}}(a_l|x)) \right) \right) \right].
$$

To prevent distributional drift, DriveDPO adds explicit KL regularization and continues to apply the Stage-1 KL loss as auxiliary supervision:

$$
L_{\text{total}} = L_{\text{DPO}} + \lambda_{KL} \cdot KL(\pi_\theta \,\|\, \pi_{\text{Ref}}) + \eta \cdot KL(p_{\text{unified}} \,\|\, \pi_\theta),
$$

with $\lambda_{KL}, \eta > 0$ [2509.17940]. The paper states that continuing the unified KL term stabilizes optimization while preserving safety alignment [2509.17940]. In the reported configuration, $\beta = 0.1$, and regularization consists of the explicit KL term against $\pi_{\text{Ref}}$ together with the auxiliary $KL(p_{\text{unified}} \,\|\, \pi_\theta)$ term maintained during DPO [2509.17940].

The algorithmic summary reported in the paper is: anchor construction by k-means over human trajectories; perception encoding with a Transfuser backbone and Fourier-feature anchor tokenization; policy prediction over anchors; unified distillation through $p_{\text{unified}}$ and $L_{\text{unified}}$; and iterative Safety DPO using sampled anchors, imitation-based rejection under $\text{PDMS}<\tau$, and updates with $L_{\text{total}}$ [2509.17940].

## 5. Evaluation, metrics, and empirical results

DriveDPO is evaluated on NAVSIM, which combines real sensor data with non-interactive simulation built on OpenScene/nuPlan and provides eight high-resolution camera views and fused point clouds at 2 Hz [2509.17940]. Models take 1.5 seconds of history and predict 8 future waypoints over 4 seconds [2509.17940]. The official evaluation metric is PDMS:

$$
\text{PDMS} = NC \times DAC \times (5 \times EP + 5 \times TTC + 2 \times C)/12.
$$

The constituent quantities are No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C) [2509.17940]. The paper notes that PDMS balances safety and progress; the multiplicative $NC \times DAC$ suppresses trajectories with collisions or off-road violations, while the weighted sum scales progress, TTC, and comfort [2509.17940].

DriveDPO achieves a reported state-of-the-art PDMS of 90.0 on NAVSIM [2509.17940]. The full model metrics are NC = 98.5, DAC = 98.1, EP = 84.3, TTC = 94.8, and C = 99.9 [2509.17940]. The paper compares these results with strong baselines: DiffusionDrive, an imitation-based approach, with PDMS = 88.1, TTC = 94.7, DAC = 96.2, and EP = 82.2; and WOTE, a score-based method, with PDMS = 88.0, NC = 98.4, DAC = 96.6, EP = 81.7, TTC = 94.5, and C = 99.9 [2509.17940]. On these figures, DriveDPO improves PDMS by +1.9 over DiffusionDrive and +2.0 over WOTE, and also surpasses a privileged PDM-closed planner with PDMS = 89.1 that uses ground-truth perception [2509.17940].

The stage-wise results are also reported. The unified distillation pretrain alone yields PDMS = 88.8 and notable gains on EP and DAC versus prior methods, while Safety DPO adds further improvements in NC (+0.6), DAC (+0.8), and TTC (+1.2) [2509.17940]. In the ablations, unified supervision over the policy distribution outperforms independent score regression: ID-1 reaches PDMS 88.8, compared with 87.3 for score regression (ID-2) [2509.17940]. Single-source supervision performs worse: imitation-only (ID-3) yields PDMS 82.5 with poor DAC, while rule-only (ID-4) yields PDMS 87.2, showing aggressive behaviors harming NC and TTC [2509.17940].

The DPO preference-construction ablation isolates the effect of negative sampling. Without DPO, the model achieves PDMS 88.8; vanilla preference selection using highest versus lowest $p_{\text{unified}}$ yields 89.3; distance-based rejection reaches 89.7; and imitation-based rejection achieves the best PDMS 90.0 [2509.17940]. The paper interprets this as indicating that targeting “human-like but unsafe” negatives is most effective [2509.17940].

DriveDPO is also evaluated in the closed-loop Bench2Drive benchmark, where it attains Driving Score 62.02, Success Rate 30.62%, Efficiency 166.80, and Comfortness 26.79, outperforming representative end-to-end baselines including TCP with Driving Score 59.90 and Success Rate 30.00%, and UniAD with Driving Score 45.81 and Success Rate 16.36% [2509.17940].

## 6. Training regime, positioning, and limitations

The reported training setup uses AdamW with learning rate $1\times 10^{-4}$, 30 epochs of unified distillation followed by 10 epochs of Safety DPO fine-tuning, batch size 16 per GPU, and 6 NVIDIA L20 GPUs [2509.17940]. The paper identifies the anchor vocabulary size $N = 8192$ and DPO sampling with $K = 1024$ per input as computational drivers [2509.17940]. It also notes that exact inference latency is not reported, although the architecture is described as consistent with real-time planners using compact backbones such as ResNet-34 [2509.17940].

The reproducibility details are partial. The work uses NAVSIM Navtrain with approximately 103k samples and Navtest with approximately 12k, under Transfuser-like settings, and Bench2Drive training on a base subset with evaluation on 220 routes [2509.17940]. Seeds and code availability are not specified [2509.17940].

Within the broader literature on preference alignment for driving, DriveDPO is closely related to later DPO-based formulations that use different preference sources. VL-DPO, for example, employs a frozen vision-language model as a zero-shot reasoner to generate pairwise trajectory preferences from rollouts and finetunes a motion forecasting model via DPO, improving RFS and ADE on WOD-E2E without requiring the VLM at inference time [2605.20082]. By contrast, DriveDPO derives preferences from simulator-based safety scores and imitation proximity rather than VLM judgments [2509.17940; 2605.20082]. A different line of work applies DPO to diffusion-based traffic scenario generation, using guide scores to construct pairwise preferences for scenario samples and fine-tuning a multi-guided diffusion model with a frozen reference model [2502.12178]. This suggests that DPO has become a general mechanism for aligning either driving policies or scenario generators to structured traffic preferences, but the objects being aligned differ across these works.

The limitations identified for DriveDPO are specific. First, the method relies on PDMS, which is described as a predefined weighted composite that may miss nuanced risks in complex interactions; the paper proposes that more expressive safety evaluators or learned risk models could enrich preference construction [2509.17940]. Second, preferences come from high-fidelity simulation and rule design, which can be limited in coverage and availability; the paper suggests mining latent preferences from large corpora of historical trajectories to reduce simulator dependence and enable weakly or self-supervised safety alignment [2509.17940]. Third, although NAVSIM mixes real-world sensors with simulation, domain gaps remain, and robustness to unseen layouts, weather, and sensor noise is identified as an ongoing challenge [2509.17940].

Taken together, DriveDPO defines a policy-learning framework in which the action distribution is first shaped by a fused human-and-safety target distribution and then refined by pairwise preference updates that explicitly penalize “human-like but unsafe” candidates [2509.17940]. The empirical record reported for NAVSIM and Bench2Drive indicates that this combination of unified distillation and Safety DPO improves NC, DAC, and TTC while producing a state-of-the-art PDMS of 90.0 on NAVSIM [2509.17940].

Source: https://www.emergentmind.com/topics/drivedpo