CausalDrive: Real-time Causal World Models for Autonomous Driving

Abstract: World models have emerged as a promising paradigm for scaling autonomous driving (AD) data, yet existing video generative models fall short as interactive simulators. Layout-conditioned renderers rely on "oracle" future trajectories of all background agents, rendering them strictly non-reactive. Conversely, pure action-conditioned predictors lack semantic control over complex interactions and suffer from prohibitive diffusion latencies, hindering closed-loop policy learning. To bridge this gap, we present CausalDrive, a controllable, real-time foundation driving world renderer. CausalDrive operates solely on the initial front-view frame, the ego-vehicle's trajectory, and a macroscopic text prompt. By excluding future NPC layouts, we compel the model to intrinsically predict causal interactions, enabling text-driven control over Driving Sociology, allowing users to dynamically orchestrate diverse counterfactual reactions to identical ego-actions. To overcome the efficiency bottleneck and address the covariate shift in autoregressive generation, we propose a novel Context-Forced DMD architecture. This combines continuous flow-matching with a self-correcting distillation objective, achieving interactive speeds of 12 FPS. This breakthrough transforms the passive video generator into a playable neural simulator. We demonstrate its versatility across three downstream applications: (1) generative closed-loop evaluation with significantly mitigated collision artifacts, (2) large-scale Reinforcement Learning (RL) post-training driven by a Video2Reward module, and (3) real-time human-in-the-loop simulation. Extensive experiments validate that policies trained within CausalDrive's reactive scenarios exhibit superior interaction capabilities in the real world.

• The paper introduces CausalDrive, an autoregressive flow-matching world model that predicts reactive multi-agent behavior from an initial frame, ego trajectory, and sociology prompt without future NPC layouts.
• The model combines Block Causal Attention, KV caching, causal ODE distillation, and Context-Forced DMD to reduce sampling to four or fewer solver steps and reach 12.4 FPS on one A100.
• CausalDrive achieves an 82.0% yielding compliance rate, 18.0% false-collision rate, 0.7792 PDM-Closed score, and 90.7 Navsim PDMS after RL fine-tuning, while remaining limited by monocular input and residual ghosting.

Motivation and problem statement

CausalDrive addresses a structural deficiency in current driving world models: the dichotomy between layout-conditioned renderers and action-conditioned predictors. Layout-conditioned systems such as MagicDrive, Panacea, and UniScene condition generation on the complete future trajectories of all traffic participants, making them strictly non-reactive "oracles" — surrounding agents cannot respond to novel ego behavior. Action-conditioned predictors such as Vista and Drive-WM are conceptually closer to true world models but suffer from posterior collapse (ignoring control signals) and offer no macroscopic semantic control over multi-agent behavior, while diffusion inference latencies preclude online RL or human-in-the-loop use.

The authors frame four desiderata for a "Foundation Driving World Renderer": photorealism, strict action-controllability, data-driven reactive agents ("driving sociology"), and real-time inference (>10 FPS). CausalDrive is designed to satisfy all four simultaneously by conditioning solely on the initial front-view frame, the ego trajectory, and a macroscopic text prompt — deliberately excluding future NPC layouts so that causal reactions must be predicted intrinsically rather than hard-coded.

Methodology

The system is built on a flow-matching Diffusion Transformer initialized from Wan2.1-1.3B, converted into a chunk-wise autoregressive simulator with Block Causal Attention and KV caching for $O(1)$ streaming complexity. Conditioning is decoupled across two channels: ego camera poses are encoded as Plücker coordinates and injected via Adaptive Layer Normalization (enforcing geometric fidelity of ego motion), while a sociology prompt $\mathcal{B}$ modulates multi-agent reactions through cross-attention. The prompt acts as a global sociological prior (e.g., "Polite" raises yielding probability across agents) rather than an instance-level script.

Two training stages address the core technical obstacles:

Causal AR teacher via flow matching. Directly distilling a bidirectional teacher into an AR student violates PF-ODE injectivity — one noisy state maps to multiple valid futures, producing blurred outputs (conditional expectation collapse). The authors therefore first train a causal autoregressive teacher with continuous flow matching under strict teacher forcing on ground-truth history, restoring injectivity between teacher and student.

Context-Forced DMD. Standard DMD fails in long-horizon AR generation because of a context mismatch analogous to covariate shift in imitation learning: the teacher scores gradients against perfect ground-truth history while the student rolls out from its own flawed history. Context-Forced DMD forces the frozen teacher to condition on the student's self-generated $N$-chunk rollout context, explicitly teaching error recovery from exposure bias. An adversarial discriminator on the fake score network preserves high-frequency road textures. Together with a preceding causal ODE distillation stage, this compresses sampling from ~50 solver steps to $\le 4$, achieving 12.4 FPS on a single A100 with sub-second latency.

SocioDrive-Bench

To supervise causal interactions without oracle layouts, the paper introduces SocioDrive-Bench: 20K clips, 80% mined from nuPlan logs and 20% synthesized in CARLA to counteract survival bias in expert datasets. Interactions are formalized in three categories with explicit kinematic triggers: ego-initiated interactions (e.g., cut-ins where a rear vehicle within 15 m decelerates at $a_{\text{obj}} < -1.5\,\text{m/s}^2$), ego-reactive interactions (hazard responses requiring $a_{\text{ego}} < -3\,\text{m/s}^2$), and complex negotiation at unprotected intersections and narrow passages. Annotation is automated through a two-stage VLM/LLM pipeline: clip-level micro-analysis with feature disentanglement (road structure vs. VRUs vs. vehicles), followed by LLM-based sequence-level synthesis providing temporal smoothing, interaction aggregation, and structured queryability for curriculum mining.

Experimental results

Simulation reliability. On nuPlan, CausalDrive attains FVD 121.6 with ADE precision/recall of 0.45/0.52, comparable to or better than Vista (FVD 323.37), GEM, Orbis, and Cosmos, while running at 12.4 FPS versus roughly 0.3–1.2 FPS for baselines — an order-of-magnitude throughput advantage that crosses the interactivity threshold. Notably, a non-real-time variant (CausalDrive$^{+}$) achieves FVD 113.6, indicating some fidelity is sacrificed for speed, though the gap is modest.

Reactivity. Under aggressive ego maneuvers on SocioDrive-Bench, Vista yields only 12.5% of the time with an 87.5% false-collision rate, whereas log-replay exhibits 100% ghosting by construction. With a "Polite" prompt, CausalDrive reaches an 82.0% Yielding Compliance Rate and reduces false collisions to 18.0%. This is the paper's strongest evidence that semantic prompting genuinely controls counterfactual NPC behavior; it also concedes that ghost artifacts are mitigated rather than eliminated entirely.

Closed-loop evaluation. Evaluating UniAD under PDM-Closed, CausalDrive achieves PDMS 0.7792 versus 0.6901 (DriveArena) and 0.7281 (DrivingSphere), with correspondingly higher route completion and lower ADS.

RL post-training. Using a Video2Reward module ($r_k = w_1 r_{\text{col}} + w_2 r_{\text{lane}}$) inside the simulator, RL fine-tuning of DiffusionDrive reaches PDMS 90.7 on Navsim, surpassing UniAD (83.4), DiffusionDrive (88.1), DriveDPO (90.0), and AD-R1 (89.8), with perfect Comfort (100) and TTC 94.7. The authors attribute the safety gains to the policy's ability to experience counterfactual crashes safely within simulation — a claim consistent with the design but one whose causal attribution would benefit from ablation isolating the contribution of sociology-conditioned scenarios specifically.

Limitations and open questions

Several constraints are acknowledged or evident. The paradigm is monocular front-view only; extension to multi-camera surround-view generation remains future work. The sociology prompt operates as a global prior, not instance-level control, so fine-grained per-agent orchestration is unresolved. The residual 18% false-collision rate indicates that ghosting is reduced, not solved. Additionally, the evaluation of downstream policies relies on simulated metrics (Navsim PDMS), and while the abstract claims real-world superiority of interaction capabilities, the presented evidence for sim-to-real transfer of RL-trained policies is indirect. Whether VLM-derived sociology labels generalize beyond nuPlan-style urban logs, and whether the Context-Forced DMD objective scales to larger backbones, remain open questions.

Conclusion

CausalDrive unifies controllable video generation with closed-loop reactivity by removing oracle layout conditioning, introducing a VLM-curated benchmark for driving sociology, and resolving the architectural and exposure-bias bottlenecks of AR diffusion distillation through Context-Forced DMD. The result is a neural simulator operating at interactive frame rates that supports generative closed-loop evaluation, RL post-training with state-of-the-art Navsim performance, and human-in-the-loop driving, establishing a concrete template for world models that function as environments rather than data engines.