ImagiDrive: Recurrent Imagination for Driving

• ImagiDrive is a unified imagination-and-planning framework that couples VLM-based driving agents with DWM scene imaginer to improve driving safety and performance.
• It employs a recurrent loop that refines trajectory predictions by iteratively generating future scene imagery, thereby reducing collision rates and planning errors.
• The system incorporates efficiency mechanisms like trajectory convergence ratio and directional-consistency to achieve robust, real-time planning in dynamic scenarios.

Searching arXiv for the primary ImagiDrive paper and closely related imagination-and-planning driving work. ImagiDrive is a unified imagination-and-planning framework for autonomous driving that integrates a Vision-LLM (VLM)-based driving agent with a Driving World Model (DWM)-based scene imaginer in a recurrent loop (Li et al., 15 Aug 2025). Its central premise is that planning should guide imagination, and imagined futures should in turn refine planning. In this formulation, the driving agent predicts an initial trajectory from multimodal inputs, the scene imaginer generates corresponding future scenarios, and these imagined scenarios are then used to iteratively refine the agent’s planning decisions (Li et al., 15 Aug 2025). The framework is presented as a response to the separation between action-level reasoning and pixel-level future prediction in prior systems, with the aim of improving driving behavior in dynamic and safety-critical situations such as turning, intersections, and collision-prone scenarios (Li et al., 15 Aug 2025).

1. Conceptual framing and relation to imagination-driven driving

ImagiDrive is defined by two model classes that had largely developed separately: VLM-based driving agents and DWM-based future-scene generators (Li et al., 15 Aug 2025). The paper characterizes VLMs as strong at contextual understanding, reasoning, and interpretable decision making, while DWMs are described as strong at generating detailed and plausible future driving scenarios. The motivation for ImagiDrive is that autonomous driving requires both capabilities: a planner should not only infer what to do from current observations, but should also be able to anticipate what the world will look like under a proposed action sequence (Li et al., 15 Aug 2025).

The framework is introduced in two variants. ImagiDrive-A is a standard VLM-based planning model that uses the driving agent alone. ImagiDrive-S is the full system, which adds a scene imaginer and runs a recurrent imagination-and-planning loop (Li et al., 15 Aug 2025). The distinction is important because the paper’s experimental analysis consistently attributes the additional gains of ImagiDrive-S to the imagination loop rather than to the VLM backbone alone (Li et al., 15 Aug 2025).

This design places ImagiDrive within a broader class of imagination-driven driving systems, but with a specific emphasis on iterative coupling. A related development is Uni-World VLA, which argues that prior imagination-based pipelines suffer from “open-loop imagination drift” and proposes interleaving future prediction and action generation step by step rather than producing a full rollout first (Liu et al., 28 Mar 2026). This suggests that ImagiDrive belongs to a larger transition from one-shot future hallucination toward closed-loop or recurrent imagination tightly coupled to control.

2. System architecture

ImagiDrive-S contains three parts: a driving agent, a scene imaginer, and a trajectory buffer (Li et al., 15 Aug 2025). The driving agent predicts an initial trajectory from multimodal inputs. The scene imaginer generates future image frames conditioned on observed history and the predicted trajectory. The trajectory buffer stores trajectories from each iteration, supports early stopping, and selects the final robust trajectory (Li et al., 15 Aug 2025).

The driving agent is built on a general-purpose VLM, instantiated in experiments with LLaVA-1.6-7B and InternVL2.5-4B (Li et al., 15 Aug 2025). Its multimodal inputs include visual input, ego state, a language prompt, and learnable trajectory query tokens. The paper defines image embeddings $e_i$, language embeddings $e_l$, ego-state embedding $e_{\text{ego}}$, and trajectory queries $q \in \mathbb{R}^{N_q \times C}$, and writes the VLM output as

$o_l, \tilde{q} = VLM(e_i,e_{ego},e_l,q).$

Each contextualized trajectory query is then decoded by a small MLP into a 2D waypoint, yielding a predicted future trajectory

$$W = \{w_{t_1}, w_{t_2}, \dots, w_{t_{N_q}}\}, \qquad w_{t_i} \in \mathbb{R}^2$$

(Li et al., 15 Aug 2025).

The scene imaginer is not newly trained inside the framework. Instead, ImagiDrive reuses existing world models: Vista on nuScenes and Epona on NAVSIM (Li et al., 15 Aug 2025). The imaginer takes observed image history $\mathbf{I}_{obs}$ and the current predicted trajectory $\hat{Y}_{curr}$, and generates a short future video

$$\hat{\mathbf{V}}_{fut} = SI(\mathbf{I}_{obs}, \hat{Y}_{curr})$$

(Li et al., 15 Aug 2025). On nuScenes, Vista is described as being built on an SVD-like architecture with about 2.5B parameters, with 1.6B in the UNet, action sequence embedded via Fourier encoding with 128 channels, 50 denoising steps, and generated short-term videos of 25 frames at 10 Hz (Li et al., 15 Aug 2025).

The trajectory buffer is the component that turns the recurrent loop into a practical planning system. It stores all iteration trajectories, applies an early stopping criterion when trajectories converge, and selects the final output using a directional-consistency rule (Li et al., 15 Aug 2025).

3. Recurrent imagination-and-planning loop

The full operational pipeline begins with multimodal input encoding: current image frame and optionally image sequence, ego state, text prompt describing the planning task, and learnable trajectory query tokens (Li et al., 15 Aug 2025). The driving agent predicts an initial sequence of waypoints. The scene imaginer then generates a short future video conditioned on that trajectory. Two future frames are selected from the generated sequence, specifically at 0.5 s and 1.0 s into the future, and are fed back into the VLM together with the current frame so that the trajectory can be refined (Li et al., 15 Aug 2025).

The paper presents this recurrent logic conceptually as: $$\hat{Y}^{(0)} = Agent(I_c, \text{ego}, \text{prompt})$$

$e_l$0

$e_l$1

$e_l$2

with all $e_l$3 stored in the trajectory buffer (Li et al., 15 Aug 2025). The maximum number of recurrent iterations is 5 (Li et al., 15 Aug 2025).

This loop is the framework’s central mechanism for connecting action-level decisions to pixel-level future consequences. The VLM proposes a continuous waypoint plan; the world model visualizes what the world would look like if the ego followed that plan; and the VLM then updates the plan using those explicit visualized consequences (Li et al., 15 Aug 2025). A plausible implication is that ImagiDrive treats imagined future images not as end products, but as decision-relevant evidence in an iterative control process.

This recurrent design is closely related to later work that makes the same critique more explicit. Uni-World VLA argues that generating a long imagined rollout before planning produces a “frozen hallucination,” because updated ego decisions cannot influence later imagined states, and proposes strict stepwise frame-action alternation instead (Liu et al., 28 Mar 2026). In that sense, ImagiDrive can be understood as an earlier unified loop, while later systems push further toward fine-grained closed-loop interleaving.

4. Training objectives and efficiency mechanisms

The driving agent is trained with a joint language-and-trajectory objective: $e_l$4 where $e_l$5 is language modeling loss implemented as cross-entropy over output tokens, and $e_l$6 is trajectory prediction loss implemented using Smooth L1 on waypoints (Li et al., 15 Aug 2025). The implementation specifies $e_l$7, with $e_l$8 and $e_l$9 set empirically, although the surrounding formatting in the paper is noted as imperfect (Li et al., 15 Aug 2025). The world model itself is not retrained with a new ImagiDrive-specific loss; the framework reuses pretrained Vista and Epona (Li et al., 15 Aug 2025).

Because both VLM inference and DWM generation are expensive, ImagiDrive introduces two explicit efficiency-and-robustness mechanisms: Early Stopping Strategy (ESS) and Trajectory Selection Strategy (TSS) (Li et al., 15 Aug 2025).

The early stopping metric is the Trajectory Convergence Ratio (TCR): $e_{\text{ego}}$0 If the TCR falls below a predefined threshold $e_{\text{ego}}$1, the recurrent process stops; the threshold is set to

$e_{\text{ego}}$2

(Li et al., 15 Aug 2025). This gives a scale-normalized measure of trajectory convergence across iterations.

The trajectory selection strategy computes, for each candidate trajectory, an average normalized motion direction: $e_{\text{ego}}$3 It then computes the normalized mean direction across all buffered trajectories,

$e_{\text{ego}}$4

and selects the trajectory with smallest angular deviation from this consensus: $e_{\text{ego}}$5 (Li et al., 15 Aug 2025).

These mechanisms materially change performance. On Turning-nuScenes, without ESS or TSS the average collision rate is 0.39 with 5 average iterations; with ESS only it is 0.28 with 2.3 iterations; with TSS only it is 0.22 with 5 iterations; and with ESS + TSS it is 0.21 with 2.3 iterations (Li et al., 15 Aug 2025). The paper’s strongest confirmed interpretation is therefore that ESS mainly reduces runtime, while TSS more directly improves robustness (Li et al., 15 Aug 2025).

5. Empirical evaluation

ImagiDrive is evaluated on NeuroNCAP, Turning-nuScenes, and NAVSIM (Li et al., 15 Aug 2025). NeuroNCAP is described as a photorealistic closed-loop safety benchmark built on nuScenes; Turning-nuScenes is a challenging turning subset of nuScenes with 17 scenes and 680 samples; and NAVSIM is a large-scale real-world benchmark for non-reactive simulation (Li et al., 15 Aug 2025).

The paper uses standard open-loop trajectory error and collision metrics, the NeuroNCAP score (NNS) for closed-loop safety, and PDMS on NAVSIM (Li et al., 15 Aug 2025). The appendix gives the NNS definition: $e_{\text{ego}}$6 where $e_{\text{ego}}$7 is impact speed and $e_{\text{ego}}$8 is reference impact speed under a no-action baseline (Li et al., 15 Aug 2025). PDMS is defined as a product of penalty terms for NC and DAC and a weighted average over EP, TTC, and Comfort (Li et al., 15 Aug 2025).

On closed-loop NeuroNCAP, LLaVA-1.6 + ImagiDrive-A achieves NeuroNCAP average 2.37 and collision average 66.79, while LLaVA-1.6 + ImagiDrive-S improves to 2.99 and 59.45 (Li et al., 15 Aug 2025). With InternVL2.5, ImagiDrive-A reaches 3.11 and 48.57, while ImagiDrive-S reaches 3.49 and 44.90 (Li et al., 15 Aug 2025). Compared with ImagiDrive-A, the imagination loop improves the LLaVA variant by 0.62 in score and reduces collision by 7.34%, and improves the InternVL variant by 0.38 in score and reduces collision by 3.67% (Li et al., 15 Aug 2025).

On open-loop Turning-nuScenes, InternVL2.5 + ImagiDrive-A obtains average L2 0.85 and average collision 0.39, whereas InternVL2.5 + ImagiDrive-S reaches 0.75 and 0.22 (Li et al., 15 Aug 2025). This best variant outperforms MomAD, which reports average L2 0.76 and average collision 0.32 (Li et al., 15 Aug 2025).

On closed-loop NAVSIM, InternVL2.5 + ImagiDrive-S reaches PDMS 87.4, with NC 98.6, DAC 96.2, TTC 94.5, Comfort 100.0, and EP 80.5 (Li et al., 15 Aug 2025). This exceeds the listed baselines, including Epona 86.2, World4Drive 85.1, DRAMA 85.5, and DrivingGPT 82.4 (Li et al., 15 Aug 2025). Related work subsequently reports even higher PDMS in a different unified architecture: Uni-World VLA reaches 89.4 PDMS on NAVSIM and argues that interleaving frame prediction and planning at 2 Hz is especially effective (Liu et al., 28 Mar 2026). This suggests that the imagination-and-planning paradigm remains active and rapidly evolving.

6. Interpretation, limitations, and broader significance

ImagiDrive’s main contribution is to show that a recurrent coupling between a VLM planner and a DWM scene generator can improve both open-loop and closed-loop autonomous driving metrics (Li et al., 15 Aug 2025). Its gains are not attributed solely to better world generation or solely to stronger multimodal reasoning; rather, the paper’s ablations indicate that the VLM agent alone is already strong, but the imagination loop adds anticipatory safety and planning refinement (Li et al., 15 Aug 2025).

At the same time, the framework is not jointly trained end-to-end in the strict sense. The VLM driving agent is trained for planning, while the scene imaginer uses existing pretrained world models such as Vista and Epona (Li et al., 15 Aug 2025). This modularity simplifies integration, but it also means that the interface between planned trajectories and generated future scenes is not globally optimized (Li et al., 15 Aug 2025). A plausible implication is that future systems may benefit from tighter co-training or more explicit shared latent interfaces between control and generation.

Several limitations are evident in the paper. First, both VLM inference and DWM generation are expensive, which is why ESS is necessary (Li et al., 15 Aug 2025). Second, generated future frames can contain artifacts such as soft edges, ghosting, and reduced texture fidelity; to address this, the authors augment future images during training with Gaussian blur, shadow overlays, random noise, JPEG compression, color jitter, center-focused blur, directional trailing blur, Gaussian noise, and random horizontal flipping, all applied with 50% probability (Li et al., 15 Aug 2025). Third, the framework avoids heavy explicit temporal modeling inside the VLM and may therefore depend strongly on the quality of world-model rollouts (Li et al., 15 Aug 2025). Fourth, the system remains an imitation-and-planning architecture rather than a full interactive simulator: it does not explicitly model causal reactions of surrounding agents to changed ego behavior beyond what is implicit in the pretrained world model (Li et al., 15 Aug 2025).

In relation to later work, ImagiDrive can be seen as an important step toward unified imagination-driven driving, but not the last word on the problem. Uni-World VLA argues that imagination and planning should be interleaved at every step rather than through recurrent refinement over sparse key frames (Liu et al., 28 Mar 2026). DriveDreamer-Policy argues that future imagination should be explicitly geometry-grounded through depth and organized in a depth-to-video-to-action hierarchy (Zhou et al., 2 Apr 2026). These developments suggest that the central questions opened by ImagiDrive concern not whether imagination helps planning, but how imagined observations should be represented, how tightly they should be coupled to action generation, and how much geometric structure is required for them to remain decision-relevant.

In summary, ImagiDrive is best understood as a recurrent autonomous driving framework in which a VLM planner and a DWM scene generator are linked in a unified imagination-and-planning loop (Li et al., 15 Aug 2025). Its technical core is the iterative cycle of trajectory prediction, trajectory-conditioned future video generation, and planning refinement using selected imagined frames, made practical by trajectory convergence-based early stopping and directional-consistency-based trajectory selection (Li et al., 15 Aug 2025). Its empirical significance lies in demonstrating that imagined future visual evidence can measurably improve planning quality and collision avoidance in both open-loop and closed-loop evaluation. Its historical significance is that it helped consolidate imagination as a first-class component of autonomous driving, rather than a peripheral visualization tool.