DiffAdapt: Unified Diffusion for Autonomous Driving

• DiffAdapt is a diffusion-based end-to-end autonomous driving framework that unifies perception, prediction, and planning via conditional BEV image generation.
• It transforms heterogeneous driving targets into rasterized BEV canvases, enforcing scene-level consistency through a shared latent generative model.
• DiffAdapt achieves state-of-the-art closed-loop performance on Bench2Drive by addressing the limitations of traditional sequential multi-task architectures.

DiffAdapt, as used here, denotes DiffAD, a diffusion-based end-to-end autonomous driving system that reformulates perception, prediction, and planning as a single conditional bird’s-eye-view image generation problem rather than a conventional multi-head stack (Wang et al., 15 Mar 2025). The method treats heterogeneous driving targets as rasterized BEV canvases, compresses them into a latent representation, and models their joint conditional distribution with a latent diffusion model conditioned on multi-camera observations, high-level commands, and temporal context (Wang et al., 15 Mar 2025). In this formulation, planning is not an isolated regression head but an emergent component of a generated scene representation, from which the ego trajectory is subsequently extracted (Wang et al., 15 Mar 2025).

1. Motivation and conceptual reframing

DiffAD is situated within the recent end-to-end autonomous driving literature that already departs from classical modular pipelines, but still typically preserves a sequential multi-task structure in which perception feeds prediction and prediction feeds planning (Wang et al., 15 Mar 2025). The paper identifies three limitations in those systems. First, they use sequential multi-task heads, so planning is trained on intermediate outputs rather than directly on raw sensor evidence, which propagates upstream errors and decomposes optimization into loosely coupled subproblems (Wang et al., 15 Mar 2025). Second, they rely on heavy query-based architectures with large numbers of task-specific learnable queries for objects, map elements, and trajectories, which shifts substantial computation toward auxiliary tasks (Wang et al., 15 Mar 2025). Third, they suffer from coordination complexity because each head uses a separate target format and loss, making it difficult to harmonize detection, mapping, forecasting, and planning into a single behaviorally coherent objective (Wang et al., 15 Mar 2025).

DiffAD addresses these issues by redefining autonomous driving as conditional generation of a unified BEV image that simultaneously encodes scene understanding, other-agent futures, and ego intent (Wang et al., 15 Mar 2025). The central conceptual move is not merely architectural consolidation; it is a change in the prediction target itself. Instead of representing perception outputs as boxes, motion outputs as vectors, and planning outputs as waypoints, the system rasterizes all such targets into three RGB BEV canvases and models their joint latent distribution (Wang et al., 15 Mar 2025). This suggests a scene-level representation in which cross-task consistency is enforced implicitly through a shared generative space rather than explicitly through hand-balanced task losses.

A common misconception is that such a unification removes planning as a distinct component. In fact, DiffAD still extracts a trajectory for control, but it does so from a generated latent BEV via a dedicated trajectory extraction network rather than from a separate planning head trained in isolation (Wang et al., 15 Mar 2025). Another misconception is that “single-framework” means “weakly supervised.” The method remains fully supervised, using rasterized ground-truth perception, prediction, and planning targets plus a vector-form ego trajectory target for extraction (Wang et al., 15 Mar 2025).

2. Formal problem statement and latent diffusion model

The method takes as inputs multi-view RGB images $I$, a high-level command $x_{cmd}$, and the previous latent BEV image $z_{bev}^{t-1}$ for temporal consistency, and learns the conditional distribution

$$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$

in latent space (Wang et al., 15 Mar 2025). The outputs are three rasterized BEV images: perception $m_{perc}$, prediction $m_{pre}$, and planning $m_{plan}$, each in $\mathbb{R}^{3 \times H \times W}$ (Wang et al., 15 Mar 2025).

A pre-trained VAE encoder compresses the three BEV canvases into latent tensors: $$[z_{perc}, z_{pre}, z_{plan}] = \text{encoder}([m_{perc}, m_{pre}, m_{plan}]),$$ followed by

$$z_{bev} = \text{concat}([z_{perc}, z_{pre}, z_{plan}]) \in \mathbb{R}^{c' \times h' \times w'}.$$

The diffusion process is then defined on $x_{cmd}$0 rather than directly on image space, which is the method’s latent diffusion formulation (Wang et al., 15 Mar 2025).

The forward noising process is specified as

$x_{cmd}$1

with the equivalent marginal form

$x_{cmd}$2

where $x_{cmd}$3 and $x_{cmd}$4 (Wang et al., 15 Mar 2025). Reparameterization yields

$x_{cmd}$5

The reverse process learns

$x_{cmd}$6

where $x_{cmd}$7 denotes the conditioning variables (Wang et al., 15 Mar 2025). Training uses the standard noise-prediction loss

$x_{cmd}$8

This yields a single denoising objective over the shared latent target rather than a collection of task-specific objectives (Wang et al., 15 Mar 2025).

The paper also frames temporal decision modeling as

$x_{cmd}$9

implemented in practice by conditioning on the previous latent BEV $z_{bev}^{t-1}$0 (Wang et al., 15 Mar 2025). This suggests that temporal consistency is imposed through latent action-conditioned recurrence rather than through an explicit recurrent planner alone.

3. Unified BEV target representation

The unified target representation consists of three RGB BEV canvases in a common spatial frame (Wang et al., 15 Mar 2025). The perception canvas $z_{bev}^{t-1}$1 contains bounding boxes of vehicles and pedestrians, static map elements such as lanes, road boundaries, intersections, and crosswalks, and other traffic elements including traffic lights, encoded with different colors per semantic type (Wang et al., 15 Mar 2025). The prediction canvas $z_{bev}^{t-1}$2 contains multi-agent future trajectories over a horizon, rendered as colored polylines with RGB interpolation over time so that temporal ordering is represented visually (Wang et al., 15 Mar 2025). The planning canvas $z_{bev}^{t-1}$3 contains the ego future trajectory, also rendered as a temporally color-interpolated polyline (Wang et al., 15 Mar 2025).

This representation has two immediate consequences. First, all driving tasks are expressed in the same BEV coordinate system, removing the need to reconcile boxes, vectors, maps, and waypoints across incompatible semantic spaces (Wang et al., 15 Mar 2025). Second, scene-level consistency becomes a property of the generated image itself: the ego plan is visually and spatially coupled to road structure and predicted agent motion (Wang et al., 15 Mar 2025). A plausible implication is that certain coordination constraints that would otherwise need explicit architectural interfaces are instead absorbed into the generative prior over valid BEV scenes.

The paper emphasizes that DiffAD replaces heterogeneous vector targets with rasterized canvases, but this should not be read as a purely cosmetic reformulation (Wang et al., 15 Mar 2025). The BEV rasterization is the mechanism that makes a single latent diffusion model possible. Without a shared image-like target, the unification claim would collapse back into a standard multi-head architecture with shared features.

4. Architecture, conditioning, and trajectory extraction

DiffAD has three principal components: a latent diffusion model, a BEV feature generator, and a trajectory extraction network (Wang et al., 15 Mar 2025). The latent diffusion model is DiT-based, operating on tokens derived from $z_{bev}^{t-1}$4, with a Transformer backbone and Adaptive LayerNorm conditional modulation (Wang et al., 15 Mar 2025). The BEV feature generator converts multi-view images into BEV features using a CNN followed by BEVFormer, producing $z_{bev}^{t-1}$5 (Wang et al., 15 Mar 2025). The trajectory extraction network consists of a Transformer plus MLP that decodes an ego trajectory from the generated latent (Wang et al., 15 Mar 2025).

Conditioning combines timestep, BEV features, command embedding, and previous latent BEV: $z_{bev}^{t-1}$6 where $z_{bev}^{t-1}$7 denotes dropout on the previous latent tokens, with probability $z_{bev}^{t-1}$8 reported as $z_{bev}^{t-1}$9–$$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$0 in experiments (Wang et al., 15 Mar 2025). Conditioning is injected through AdaLN: $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$1 The backbone then predicts $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$2 (Wang et al., 15 Mar 2025).

The trajectory extraction stage maps the final latent into a future ego path through

$$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$3

with $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$4 in BEV coordinates (Wang et al., 15 Mar 2025). Low-level control is then handled by the PID controller provided by Bench2Drive, which converts waypoints into steering, throttle, and brake (Wang et al., 15 Mar 2025). This means the overall pipeline is end-to-end in the learning sense, but closed-loop execution still depends on a conventional downstream controller.

A further architectural point is computational concentration. The paper reports that 83% of runtime lies in the diffusion head, with the trajectory extraction network contributing only 1 ms in the TensorRT-FP16 configuration (Wang et al., 15 Mar 2025). This indicates that the system’s efficiency bottleneck is not BEV decoding or control but iterative denoising itself.

5. Training procedure and closed-loop inference

Training is conducted on the Bench2Drive dataset, specifically the base subset of 1,000 clips, with both training and evaluation performed in the CARLA v2 closed-loop setting (Wang et al., 15 Mar 2025). Ground-truth supervision includes 3D bounding boxes, map elements, traffic lights, other-agent future trajectories, and ego future trajectories, rasterized into the three BEV canvases; the ego trajectory is also retained in vector form $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$5 for the extraction network (Wang et al., 15 Mar 2025).

The training objective contains two terms: $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$6 with total loss

$$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$7

The paper identifies this as a central simplification: no detection-specific, segmentation-specific, or forecasting-specific loss is used (Wang et al., 15 Mar 2025).

Training follows a two-stage curriculum (Wang et al., 15 Mar 2025). In Stage 1, the model performs single-frame perception-only learning, using real $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$8 while zero-padding $$p_\theta(m_{perc}, m_{pre}, m_{plan} \mid I, x_{cmd}, z_{bev}^{t-1})$$9 and $m_{perc}$0 (Wang et al., 15 Mar 2025). In Stage 2, full temporal joint training is performed over perception, prediction, and planning, with historical BEV feature fusion by ConvLSTM and action-guided conditioning via the previous latent BEV (Wang et al., 15 Mar 2025). The VAE is initialized from Stable Diffusion with downsampling factor 8, and DiT hyperparameters are reused from the DiT paper (Wang et al., 15 Mar 2025).

Inference begins with multi-view images, a command embedding, and latent noise $m_{perc}$1, then performs iterative denoising in latent space (Wang et al., 15 Mar 2025). The reported deployment choice is DDIM sampling with 10 steps, selected after comparison of 3, 5, 10, and 20 denoising steps (Wang et al., 15 Mar 2025). On the ablation reported in the paper, 10 steps yields approximately FID $m_{perc}$2, Driving Score $m_{perc}$3, and Success Rate $m_{perc}$4, with diminishing returns beyond that point (Wang et al., 15 Mar 2025). The generated latent can optionally be decoded back into BEV images for debugging and interpretability (Wang et al., 15 Mar 2025).

6. Empirical performance, efficiency, and failure modes

On Bench2Drive closed-loop evaluation, DiffAD reports the following overall performance (Wang et al., 15 Mar 2025):

Method	Driving Score	Success Rate
AD-MLP	18.05	0.00
UniAD-Tiny	40.73	13.18
UniAD-Base	45.81	16.36
VAD	42.35	15.00
TCP-traj*	59.90	30.00
ThinkTwice*	62.44	31.23
DriveAdapter*	64.22	33.08
DiffAD	67.92	38.64

The paper states that DiffAD outperforms UniAD and VAD by a large margin and also exceeds strong distillation-based methods such as ThinkTwice and DriveAdapter despite not using expert feature distillation (Wang et al., 15 Mar 2025). It is reported as particularly strong in interactive abilities such as merging, overtaking, and emergency braking, and it achieves the highest mean multi-ability score at 38.79% (Wang et al., 15 Mar 2025).

The joint-training ablation clarifies the role of auxiliary scene generation in planning quality. Planning alone yields DS 30.11 / SR 5.9, adding detection raises this to DS 59.10 / SR 29.09, and adding detection plus motion reaches DS 66.96 / SR 37.27 (Wang et al., 15 Mar 2025). This indicates that unified scene supervision materially improves planning rather than merely serving interpretability.

In efficiency terms, the model is larger than its principal E2E competitors: UniAD-base 84.2M, VAD-base 58.1M, and DiffAD-10steps 545.6M parameters (Wang et al., 15 Mar 2025). On an RTX 4090 in FP32, reported latency is 355 ms for UniAD, 140 ms for VAD, and 258 ms for DiffAD-10steps, corresponding to 2.8 FPS, 7.1 FPS, and 3.9 FPS respectively (Wang et al., 15 Mar 2025). Under TensorRT-FP16, DiffAD is reported at 42 ms total and 23.8 FPS, broken down as 1.6 ms for BEV features, 40 ms for diffusion, and 1 ms for trajectory extraction (Wang et al., 15 Mar 2025). The paper notes that 83% of inference time lies in the diffusion head, making it amenable to distillation or reduced NFE strategies (Wang et al., 15 Mar 2025).

The reported limitations are substantial. Despite improvement, the Success Rate of 38.64% remains far from complete task success (Wang et al., 15 Mar 2025). Failure analysis attributes roughly 44.5% of route failures to collisions with agents, 7.3% to red-light violations, and 7.3% to timeouts or failure to resume motion (Wang et al., 15 Mar 2025). The paper also notes ambiguity and low quality in some CARLA v2 traffic lights, uncertainty about performance on automotive hardware despite 23.8 FPS under aggressive optimization, relatively small training data in the Bench2Drive base subset, and the absence of real-world deployment evidence (Wang et al., 15 Mar 2025). These points are not incidental; they qualify the method’s state-of-the-art result as a simulator-bound advance rather than a solved deployment pathway.

7. Position within the literature and naming scope

Within autonomous driving, DiffAD is presented as distinct from diffusion applications that address only isolated modules. The paper cites DiffBEV for BEV perception refinement, PolyDiffuse for polygonal map reconstruction, MotionDiffuser for multi-agent trajectory prediction, and DiffusionDrive for truncated diffusion-based multi-modal trajectory prediction, arguing that these works apply diffusion to single components rather than to the entire end-to-end stack (Wang et al., 15 Mar 2025). DiffAD’s novelty claim is therefore not simply “using diffusion in driving,” but using diffusion to model the joint distribution over perception, prediction, and planning in a unified latent BEV space (Wang et al., 15 Mar 2025).

Relative to E2E-AD baselines, the paper positions imitation-learning MLP planners as lacking explicit environmental modeling, query-based transformers such as UniAD, VAD, and SparseAD as retaining multi-head sequential structure, ParaDrive as still multi-head despite dense BEV optimization, and distillation-based planners such as TCP, ThinkTwice, and DriveAdapter as dependent on privileged teacher features (Wang et al., 15 Mar 2025). DiffAD’s contribution, in that framing, is to replace task decomposition with a generative scene target while remaining fully differentiable and closed-loop trainable (Wang et al., 15 Mar 2025).

There is also a nomenclatural point. “DiffAdapt” is not a unique term across arXiv. In the present context it refers to DiffAD in autonomous driving (Wang et al., 15 Mar 2025), whereas later literature also used “DiffAdapt” as the title of a difficulty-adaptive inference framework for reasoning LLMs (Liu et al., 22 Oct 2025). The autonomous-driving usage is therefore best understood as a query-specific alias for DiffAD rather than a universally standardized name.

Overall, DiffAD represents a generative reformulation of end-to-end autonomous driving in which perception, prediction, and planning are collapsed into conditional BEV image generation, optimized with a diffusion loss plus trajectory extraction loss, and evaluated in closed loop with state-of-the-art Bench2Drive results (Wang et al., 15 Mar 2025). Its significance lies less in the adoption of diffusion per se than in the claim that holistic driving behavior can be modeled as a single latent scene-generation problem.