ADriver-I: Autonomous Driving World Model

• The paper introduces ADriver-I, a world model that unifies vision-action pairs through autoregressive sequence prediction to merge control and environmental simulation.
• It employs a multimodal large language model with a diffusion-based video generator to predict control signals and generate future scene frames, outperforming baseline methods.
• Empirical evaluations on nuScenes and proprietary datasets reveal reduced L1 errors in speed and steering, while also highlighting challenges under sudden control changes.

ADriver-I is a general world model for autonomous driving that represents the driving process as an interleaved sequence of visual observations and control signals, and couples a multimodal LLM with a diffusion-based video generator to predict both action and future scene evolution (Jia et al., 2023). In its explicit arXiv formulation, the system takes vision-action pairs as inputs, autoregressively predicts the control signal of the current frame, conditions future-frame generation on the predicted control together with historical vision-action pairs, and can repeat this loop “infinite times,” thereby performing autonomous driving “in the world created by itself” (Jia et al., 2023). In adjacent technical syntheses, the name also appears as a broader conceptual label for intelligent driver-support, information-aware longitudinal control, and cooperative driver-intent systems; this suggests that ADriver-I has both a specific meaning as a world-model architecture and a broader meaning as a family of driver-intelligence abstractions (Luo et al., 14 Jan 2025).

1. Definition and representational core

The explicit ADriver-I architecture is motivated by a critique of the conventional autonomous-driving pipeline—perception, prediction, planning, and control—as modular and interpretable but also redundant and complex (Jia et al., 2023). Its central representational object is the interleaved vision-action pair $(I_t, A_t)$, where $I_t$ is a front-view RGB camera frame and $A_t$ is the control signal at time $t$, comprising speed and steering angle (Jia et al., 2023). A sequence of such pairs is treated as a multimodal token stream rather than as separate perception and planning products.

Visual inputs are encoded by CLIP-ViT-Large and mapped into the language-model embedding space through a two-layer MLP visual adapter, while control signals are serialized as text-like numeric tokens with three decimal places and multiplied by 1000 so that values become integers (Jia et al., 2023). The tokenization scheme inserts each image as a special visual token and each action as a bounded numeric token sequence. The resulting formulation allows the driving process to be modeled as autoregressive multimodal sequence prediction rather than as explicit box-, map-, or occupancy-centric reasoning (Jia et al., 2023).

This representation is also the main reason ADriver-I is described as a general world model rather than merely an end-to-end policy. The same token stream supports action prediction and future-scene generation, so the architecture unifies control and environment evolution in one closed-loop formalism. A plausible implication is that the system is designed less as a replacement for any single module than as an alternative interface to the entire stack.

2. Multimodal control generation

The control-prediction component uses Vicuna-7B-1.5 as the language-model backbone, CLIP-ViT-Large as the visual encoder, and the visual adapter to align image features with the language embedding space (Jia et al., 2023). At time $t$, the model receives the current frame together with three historical vision-action pairs, namely $(I_{t-1}, A_{t-1})$, $(I_{t-2}, A_{t-2})$, and $(I_{t-3}, A_{t-3})$ (Jia et al., 2023). During training, it autoregressively predicts $\{A'_{t-2}, A'_{t-1}, A'_t\}$; during inference, it predicts the current control $A'_t$ in one step (Jia et al., 2023).

The training objective is cross-entropy over the output digit tokens rather than a direct continuous regression loss (Jia et al., 2023). This means that low-level control is cast as token prediction, with speed and steering angle treated as text-like outputs. The paper’s ablations report that direct absolute-number prediction outperforms alternatives such as Num2English, bin-based special tokens, and relative-difference encodings, and that two or three decimal places work best (Jia et al., 2023). Multi-round conversation supervision also improves speed prediction, with speed L1 decreasing from 0.078 to 0.072 under the reported setting (Jia et al., 2023).

On nuScenes, ADriver-I reports speed L1 error 0.072 and steering-angle L1 error 0.091, compared with 0.122 and 0.101 for MLP-Only, 0.106 and 0.095 for a CNN-based baseline, and 0.103 and 0.092 for a ViT-based baseline (Jia et al., 2023). On the large private highway-oriented dataset, the reported numbers are speed L1 0.035 and steering L1 0.015 (Jia et al., 2023). These results place the language-model component not as a generic captioning front-end but as the primary control policy.

3. Diffusion-based world dynamics and closed-loop rollout

The second major component is a video latent diffusion model built on Stable Diffusion 2.1 and extended to video following “Align Your Latents” (Jia et al., 2023). It is conditioned on historical frames and a motion description derived from the predicted control signal, with GPT-3.5 used to convert control values into motion prompts such as “steady speed,” “accelerating,” “decelerating,” “turning left,” or “turning right” (Jia et al., 2023). Given the recent frame history and the current control, the model predicts four future frames, $I_t$0 (Jia et al., 2023).

The operative loop is straightforward. First, the multimodal LLM predicts the current control from the current frame and historical vision-action context. Second, the diffusion model generates future frames conditioned on that control and the recent visual history. Third, the next generated frame is fed back as the new current frame for the next control prediction (Jia et al., 2023). Because the loop can be repeated, ADriver-I acts as an agent-policy and a learned environment simulator simultaneously.

This architecture is what gives the system its world-model character. The LLM functions as the driving policy; the diffusion model functions as the world dynamics generator. The paper’s qualitative long-horizon examples show a few real initial frames followed by generated frames, with control predicted at each step from the generated context (Jia et al., 2023). The formulation is still supervised rather than reward-driven: it resembles model-based reinforcement-learning world models structurally, but its training remains based on logged driving data rather than on reward optimization. That distinction is explicit in the reported design.

4. Training corpora, metrics, and empirical performance

ADriver-I is trained on two principal corpora: a private dataset containing about 1.4M vision-action pairs from mostly highway scenarios, and nuScenes, from which front-view frames and ego-vehicle control signals are extracted (Jia et al., 2023). For the diffusion component, the nuScenes preparation yields about 23K video samples, each eight frames long, with four historical and four future frames (Jia et al., 2023).

The MLLM pretraining stage freezes the LLM and trains the vision encoder and adapter on the private dataset; the subsequent supervised fine-tuning stage freezes the vision encoder and trains the LLM and adapter on nuScenes and the private dataset separately (Jia et al., 2023). The reported setup uses image size $I_t$1, batch size 16, AdamW, learning rate $I_t$2, and two epochs in each stage on 8 A100 GPUs (Jia et al., 2023). The VDM is pretrained for 40K steps on the private video corpus at resolution $I_t$3 and fine-tuned for 40K steps on nuScenes, with learning rates $I_t$4 and $I_t$5 respectively (Jia et al., 2023).

For control evaluation, the paper reports both L1 error and threshold-based accuracy metrics. For scene generation, it reports FID = 5.5 and FVD = 97.0 for generating four future frames conditioned on four past frames (Jia et al., 2023). The constructed comparisons report DriveGAN at FID 73.4 and FVD 502.3, and DriveDreamer at FID 52.6 and FVD 452.0 (Jia et al., 2023). The paper emphasizes that these results are obtained without using HD maps or 3D bounding boxes as generation priors (Jia et al., 2023).

The resulting empirical picture is asymmetric but informative. Control prediction is substantially better than the constructed baselines on speed and modestly better on steering on nuScenes, while generation quality is reported as markedly stronger than the two generative baselines under the chosen metrics (Jia et al., 2023). This suggests that ADriver-I’s unification is not only architectural; it also produces competitive results in both halves of the loop.

5. Broader uses of the ADriver-I label

Outside the explicit world-model paper, the supplied literature uses “ADriver-I” as a broader conceptual label for intelligent driver-support layers. In a high-speed railway advisory system, the term is associated with an intelligent, AI-based advisory layer that has rich domain knowledge, interacts with drivers via natural language, grounds its responses in validated manuals and rules, and supports situational decision-making without directly taking control; the concrete implementation there is an LLM-based framework combining ChatGLM3-6B, LoRA fine-tuning, and RAG over railway manuals (Luo et al., 14 Jan 2025). In that setting, the phrase denotes an advisory system rather than an autonomous world model.

In longitudinal road-vehicle control, adjacent syntheses use the label for an information-aware or dependability-aware driver model. One synthesis links an ADriver-I-like framework to a layered architecture consisting of a perception and environment model, an MPC-based longitudinal planner, and powertrain optimization, with off-board information such as SPaT, eHorizon, V2V, and cloud services incorporated under Digital Dependability Identity constraints (Armengaud et al., 2019). Another describes an “ADriver-I”-type Information-Aware Driver Model that fuses sensor and V2X data, defines a dynamic safe gap, uses tunable comfortable acceleration and deceleration functions, and is designed to maintain safety, comfort, and operational efficiency across traffic states (Rahman et al., 2020). A third uses “ADriver-I / AIDM” for an active-inference longitudinal driver that maintains probabilistic beliefs over traffic states and selects control actions by minimizing expected free energy in a car-following task (Wei et al., 2023).

The label is also attached to cooperative and end-to-end driving abstractions. In a cooperative lane-change Driver Messenger System, it denotes a point-to-point, V2X-based driver-intent layer that maintains a local object map, recognizes a target vehicle using path history, and sends Driver Intent Messages to enable cooperative gap creation (Shah et al., 2022). In vectorized end-to-end driving, InVDriver is described as operating in “the same design space as ADriver-I,” with the term used for query-based transformer-centric end-to-end stacks and InVDriver presented as an intra-instance–aware variant of that lineage (Zhang et al., 25 Feb 2025). This broader usage suggests that, across adjacent literature, ADriver-I has become a convenient shorthand for architectures that bind driver intent, environmental context, and decision support into one integrated layer, even when the underlying technical mechanisms differ substantially.

6. Limitations and future directions

The explicit ADriver-I world-model paper is direct about its current limits. The diffusion component can generate low-quality frames when control signals change rapidly, particularly during sharp turns or sudden braking, and those degraded frames can then confuse the control model in subsequent steps (Jia et al., 2023). The two major components are also trained separately, so the control generator is not optimized jointly with the video generator and the generator is not optimized to produce observations that are maximally useful for downstream control (Jia et al., 2023). The system also lacks navigation or route-level conditioning; the stated future direction is to introduce navigation maps for long-distance autonomous driving (Jia et al., 2023). The authors further state that performance remains far from satisfactory for real-world deployment and that future versions will be trained on much larger private datasets (Jia et al., 2023).

The broader ADriver-I-associated literature reveals parallel limitations of scope. The advisory, longitudinal, cooperative, and activity-recognition formulations each emphasize one slice of the driver-intelligence problem—manual fault handling in rail (Luo et al., 14 Jan 2025), off-board dependable longitudinal planning (Armengaud et al., 2019), active-inference car-following (Wei et al., 2023), V2X-based cooperative intent sharing (Shah et al., 2022), or in-cabin driver activity recognition (Peng et al., 2022, Park et al., 13 Dec 2025). This suggests that “ADriver-I” is not yet a standardized canonical stack across the field. Rather, the name gathers together a set of attempts to replace rigid modularity with tighter coupling among perception, intent, prediction, control, and human interaction.

A plausible implication is that the next stage of ADriver-I research will be integrative rather than merely larger. The world-model line points toward tighter joint optimization of action and scene generation; the longitudinal-control line points toward trust-aware off-board integration and executable dependability contracts; the advisory line points toward grounded natural-language interaction; and the cooperative line points toward explicit negotiation among vehicles. Taken together, these strands frame ADriver-I as a research program aimed at making driving systems simultaneously more unified, more predictive, and more explicitly coupled to the states of the world, the vehicle, and the human operator.