---
title: 'ImagiDrive: Recurrent Imagination for Driving'
url: https://www.emergentmind.com/topics/imagidrive
type: topic
---

# ImagiDrive: Recurrent Imagination for Driving

Searching arXiv for the primary ImagiDrive paper and closely related imagination-and-planning driving work.
ImagiDrive is a unified imagination-and-planning framework for autonomous driving that integrates a Vision-Language Model (VLM)-based driving agent with a Driving World Model (DWM)-based scene imaginer in a recurrent loop [2508.11428]. Its central premise is that planning should guide imagination, and imagined futures should in turn refine planning. In this formulation, the driving agent predicts an initial trajectory from multimodal inputs, the scene imaginer generates corresponding future scenarios, and these imagined scenarios are then used to iteratively refine the agent’s planning decisions [2508.11428]. The framework is presented as a response to the separation between action-level reasoning and pixel-level future prediction in prior systems, with the aim of improving driving behavior in dynamic and safety-critical situations such as turning, intersections, and collision-prone scenarios [2508.11428].

## 1. Conceptual framing and relation to imagination-driven driving

ImagiDrive is defined by two model classes that had largely developed separately: VLM-based driving agents and DWM-based future-scene generators [2508.11428]. The paper characterizes VLMs as strong at contextual understanding, reasoning, and interpretable decision making, while DWMs are described as strong at generating detailed and plausible future driving scenarios. The motivation for ImagiDrive is that autonomous driving requires both capabilities: a planner should not only infer what to do from current observations, but should also be able to anticipate what the world will look like under a proposed action sequence [2508.11428].

The framework is introduced in two variants. **ImagiDrive-A** is a standard VLM-based planning model that uses the driving agent alone. **ImagiDrive-S** is the full system, which adds a scene imaginer and runs a recurrent imagination-and-planning loop [2508.11428]. The distinction is important because the paper’s experimental analysis consistently attributes the additional gains of ImagiDrive-S to the imagination loop rather than to the VLM backbone alone [2508.11428].

This design places ImagiDrive within a broader class of imagination-driven driving systems, but with a specific emphasis on iterative coupling. A related development is Uni-World VLA, which argues that prior imagination-based pipelines suffer from “open-loop imagination drift” and proposes interleaving future prediction and action generation step by step rather than producing a full rollout first [2603.27287]. This suggests that ImagiDrive belongs to a larger transition from one-shot future hallucination toward closed-loop or recurrent imagination tightly coupled to control.

## 2. System architecture

ImagiDrive-S contains three parts: a **driving agent**, a **scene imaginer**, and a **trajectory buffer** [2508.11428]. The driving agent predicts an initial trajectory from multimodal inputs. The scene imaginer generates future image frames conditioned on observed history and the predicted trajectory. The trajectory buffer stores trajectories from each iteration, supports early stopping, and selects the final robust trajectory [2508.11428].

The driving agent is built on a general-purpose VLM, instantiated in experiments with **LLaVA-1.6-7B** and **InternVL2.5-4B** [2508.11428]. Its multimodal inputs include visual input, ego state, a language prompt, and learnable trajectory query tokens. The paper defines image embeddings \(e_i\), language embeddings \(e_l\), ego-state embedding \(e_{\text{ego}}\), and trajectory queries \(q \in \mathbb{R}^{N_q \times C}\), and writes the VLM output as
\[
o_l, \tilde{q} = VLM(e_i,e_{ego},e_l,q).
\]
Each contextualized trajectory query is then decoded by a small MLP into a 2D waypoint, yielding a predicted future trajectory
\[
W = \{w_{t_1}, w_{t_2}, \dots, w_{t_{N_q}}\}, \qquad w_{t_i} \in \mathbb{R}^2
\]
[2508.11428].

The scene imaginer is not newly trained inside the framework. Instead, ImagiDrive reuses existing world models: **Vista** on nuScenes and **Epona** on NAVSIM [2508.11428]. The imaginer takes observed image history \(\mathbf{I}_{obs}\) and the current predicted trajectory \(\hat{Y}_{curr}\), and generates a short future video
\[
\hat{\mathbf{V}}_{fut} = SI(\mathbf{I}_{obs}, \hat{Y}_{curr})
\]
[2508.11428]. On nuScenes, Vista is described as being built on an SVD-like architecture with about **2.5B parameters**, with **1.6B** in the UNet, action sequence embedded via **Fourier encoding with 128 channels**, **50** denoising steps, and generated short-term videos of **25 frames at 10 Hz** [2508.11428].

The trajectory buffer is the component that turns the recurrent loop into a practical planning system. It stores all iteration trajectories, applies an early stopping criterion when trajectories converge, and selects the final output using a directional-consistency rule [2508.11428].

## 3. Recurrent imagination-and-planning loop

The full operational pipeline begins with multimodal input encoding: current image frame and optionally image sequence, ego state, text prompt describing the planning task, and learnable trajectory query tokens [2508.11428]. The driving agent predicts an initial sequence of waypoints. The scene imaginer then generates a short future video conditioned on that trajectory. Two future frames are selected from the generated sequence, specifically at **0.5 s** and **1.0 s** into the future, and are fed back into the VLM together with the current frame so that the trajectory can be refined [2508.11428].

The paper presents this recurrent logic conceptually as:
\[
\hat{Y}^{(0)} = Agent(I_c, \text{ego}, \text{prompt})
\]
\[
\hat{\mathbf{V}}_{fut}^{(i)} = SI(\mathbf{I}_{obs}, \hat{Y}^{(i)})
\]
\[
\{I_{0.5}^{(i)}, I_{1.0}^{(i)}\} \subset \hat{\mathbf{V}}_{fut}^{(i)}
\]
\[
\hat{Y}^{(i+1)} = Agent(I_c, I_{0.5}^{(i)}, I_{1.0}^{(i)}, \text{ego}, \text{prompt})
\]
with all \(\hat{Y}^{(i)}\) stored in the trajectory buffer [2508.11428]. The maximum number of recurrent iterations is **5** [2508.11428].

This loop is the framework’s central mechanism for connecting action-level decisions to pixel-level future consequences. The VLM proposes a continuous waypoint plan; the world model visualizes what the world would look like if the ego followed that plan; and the VLM then updates the plan using those explicit visualized consequences [2508.11428]. A plausible implication is that ImagiDrive treats imagined future images not as end products, but as decision-relevant evidence in an iterative control process.

This recurrent design is closely related to later work that makes the same critique more explicit. Uni-World VLA argues that generating a long imagined rollout before planning produces a “frozen hallucination,” because updated ego decisions cannot influence later imagined states, and proposes strict stepwise frame-action alternation instead [2603.27287]. In that sense, ImagiDrive can be understood as an earlier unified loop, while later systems push further toward fine-grained closed-loop interleaving.

## 4. Training objectives and efficiency mechanisms

The driving agent is trained with a joint language-and-trajectory objective:
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{l} + \lambda \mathcal{L}_{\text{traj}}
\]
where \(\mathcal{L}_{l}\) is language modeling loss implemented as cross-entropy over output tokens, and \(\mathcal{L}_{\text{traj}}\) is trajectory prediction loss implemented using Smooth L1 on waypoints [2508.11428]. The implementation specifies \(N_q = 6\), with \(\lambda_1 = 0.1\) and \(\lambda_2 = 0.5\) set empirically, although the surrounding formatting in the paper is noted as imperfect [2508.11428]. The world model itself is not retrained with a new ImagiDrive-specific loss; the framework reuses pretrained Vista and Epona [2508.11428].

Because both VLM inference and DWM generation are expensive, ImagiDrive introduces two explicit efficiency-and-robustness mechanisms: **Early Stopping Strategy (ESS)** and **Trajectory Selection Strategy (TSS)** [2508.11428].

The early stopping metric is the **Trajectory Convergence Ratio (TCR)**:
\[
\text{TCR}(\hat{Y}^{(i)}, \hat{Y}^{(j)}) =
\frac{1}{N_q}\sum_{t=1}^{N_q}
\frac{\|\hat{Y}^{(i)}_t-\hat{Y}^{(j)}_t\|_2}
{\|\hat{Y}^{(j)}_t\|_2+\varepsilon}.
\]
If the TCR falls below a predefined threshold \(\theta\), the recurrent process stops; the threshold is set to
\[
\theta = 0.05
\]
[2508.11428]. This gives a scale-normalized measure of trajectory convergence across iterations.

The trajectory selection strategy computes, for each candidate trajectory, an average normalized motion direction:
\[
\hat{\mathbf{v}}^{(i)} = \frac{1}{n-1} \sum_{t=1}^{n-1}
\frac{\mathbf{p}^{(i)}_{t+1} - \mathbf{p}^{(i)}_t}
{\|\mathbf{p}^{(i)}_{t+1} - \mathbf{p}^{(i)}_t\| + \varepsilon}.
\]
It then computes the normalized mean direction across all buffered trajectories,
\[
\hat{\bar{\mathbf{v}}} = \frac{1}{N}\sum_{i=1}^{N}\hat{\mathbf{v}}^{(i)}, \qquad
\hat{\bar{\mathbf{v}}} \leftarrow
\frac{\hat{\bar{\mathbf{v}}}}{\|\hat{\bar{\mathbf{v}}}\|+\varepsilon},
\]
and selects the trajectory with smallest angular deviation from this consensus:
\[
\theta^{(i)} = \arccos\left(
\mathrm{clip}\left(\hat{\mathbf{v}}^{(i)} \cdot \hat{\bar{\mathbf{v}}}, -1, 1\right)
\right),
\qquad
i^* = \arg\min_i \theta^{(i)},
\qquad
\hat{Y}^* = \hat{Y}^{(i^*)}
\]
[2508.11428].

These mechanisms materially change performance. On Turning-nuScenes, without ESS or TSS the average collision rate is **0.39** with **5** average iterations; with ESS only it is **0.28** with **2.3** iterations; with TSS only it is **0.22** with **5** iterations; and with ESS + TSS it is **0.21** with **2.3** iterations [2508.11428]. The paper’s strongest confirmed interpretation is therefore that ESS mainly reduces runtime, while TSS more directly improves robustness [2508.11428].

## 5. Empirical evaluation

ImagiDrive is evaluated on **NeuroNCAP**, **Turning-nuScenes**, and **NAVSIM** [2508.11428]. NeuroNCAP is described as a photorealistic closed-loop safety benchmark built on nuScenes; Turning-nuScenes is a challenging turning subset of nuScenes with **17 scenes** and **680 samples**; and NAVSIM is a large-scale real-world benchmark for non-reactive simulation [2508.11428].

The paper uses standard open-loop trajectory error and collision metrics, the **NeuroNCAP score (NNS)** for closed-loop safety, and **PDMS** on NAVSIM [2508.11428]. The appendix gives the NNS definition:
\[
\text{NNS} =
\begin{cases}
5.0 & \text{if no collision},\\
4.0 \cdot \max(0, 1 - v_i / v_r) & \text{otherwise},
\end{cases}
\]
where \(v_i\) is impact speed and \(v_r\) is reference impact speed under a no-action baseline [2508.11428]. PDMS is defined as a product of penalty terms for **NC** and **DAC** and a weighted average over **EP**, **TTC**, and **Comfort** [2508.11428].

On closed-loop NeuroNCAP, **LLaVA-1.6 + ImagiDrive-A** achieves NeuroNCAP average **2.37** and collision average **66.79**, while **LLaVA-1.6 + ImagiDrive-S** improves to **2.99** and **59.45** [2508.11428]. With **InternVL2.5**, ImagiDrive-A reaches **3.11** and **48.57**, while ImagiDrive-S reaches **3.49** and **44.90** [2508.11428]. Compared with ImagiDrive-A, the imagination loop improves the LLaVA variant by **0.62** in score and reduces collision by **7.34%**, and improves the InternVL variant by **0.38** in score and reduces collision by **3.67%** [2508.11428].

On open-loop Turning-nuScenes, **InternVL2.5 + ImagiDrive-A** obtains average L2 **0.85** and average collision **0.39**, whereas **InternVL2.5 + ImagiDrive-S** reaches **0.75** and **0.22** [2508.11428]. This best variant outperforms **MomAD**, which reports average L2 **0.76** and average collision **0.32** [2508.11428].

On closed-loop NAVSIM, **InternVL2.5 + ImagiDrive-S** reaches **PDMS 87.4**, with **NC 98.6**, **DAC 96.2**, **TTC 94.5**, **Comfort 100.0**, and **EP 80.5** [2508.11428]. This exceeds the listed baselines, including **Epona 86.2**, **World4Drive 85.1**, **DRAMA 85.5**, and **DrivingGPT 82.4** [2508.11428]. Related work subsequently reports even higher PDMS in a different unified architecture: Uni-World VLA reaches **89.4 PDMS** on NAVSIM and argues that interleaving frame prediction and planning at 2 Hz is especially effective [2603.27287]. This suggests that the imagination-and-planning paradigm remains active and rapidly evolving.

## 6. Interpretation, limitations, and broader significance

ImagiDrive’s main contribution is to show that a recurrent coupling between a VLM planner and a DWM scene generator can improve both open-loop and closed-loop autonomous driving metrics [2508.11428]. Its gains are not attributed solely to better world generation or solely to stronger multimodal reasoning; rather, the paper’s ablations indicate that the VLM agent alone is already strong, but the imagination loop adds anticipatory safety and planning refinement [2508.11428].

At the same time, the framework is not jointly trained end-to-end in the strict sense. The VLM driving agent is trained for planning, while the scene imaginer uses existing pretrained world models such as Vista and Epona [2508.11428]. This modularity simplifies integration, but it also means that the interface between planned trajectories and generated future scenes is not globally optimized [2508.11428]. A plausible implication is that future systems may benefit from tighter co-training or more explicit shared latent interfaces between control and generation.

Several limitations are evident in the paper. First, both VLM inference and DWM generation are expensive, which is why ESS is necessary [2508.11428]. Second, generated future frames can contain artifacts such as soft edges, ghosting, and reduced texture fidelity; to address this, the authors augment future images during training with Gaussian blur, shadow overlays, random noise, JPEG compression, color jitter, center-focused blur, directional trailing blur, Gaussian noise, and random horizontal flipping, all applied with **50% probability** [2508.11428]. Third, the framework avoids heavy explicit temporal modeling inside the VLM and may therefore depend strongly on the quality of world-model rollouts [2508.11428]. Fourth, the system remains an imitation-and-planning architecture rather than a full interactive simulator: it does not explicitly model causal reactions of surrounding agents to changed ego behavior beyond what is implicit in the pretrained world model [2508.11428].

In relation to later work, ImagiDrive can be seen as an important step toward unified imagination-driven driving, but not the last word on the problem. Uni-World VLA argues that imagination and planning should be interleaved at every step rather than through recurrent refinement over sparse key frames [2603.27287]. DriveDreamer-Policy argues that future imagination should be explicitly geometry-grounded through depth and organized in a depth-to-video-to-action hierarchy [2604.01765]. These developments suggest that the central questions opened by ImagiDrive concern not whether imagination helps planning, but how imagined observations should be represented, how tightly they should be coupled to action generation, and how much geometric structure is required for them to remain decision-relevant.

In summary, ImagiDrive is best understood as a recurrent autonomous driving framework in which a VLM planner and a DWM scene generator are linked in a unified imagination-and-planning loop [2508.11428]. Its technical core is the iterative cycle of trajectory prediction, trajectory-conditioned future video generation, and planning refinement using selected imagined frames, made practical by trajectory convergence-based early stopping and directional-consistency-based trajectory selection [2508.11428]. Its empirical significance lies in demonstrating that imagined future visual evidence can measurably improve planning quality and collision avoidance in both open-loop and closed-loop evaluation. Its historical significance is that it helped consolidate imagination as a first-class component of autonomous driving, rather than a peripheral visualization tool.

Source: https://www.emergentmind.com/topics/imagidrive