DriveMind: Autonomous Driving Causal Analysis

• DriveMind is a nuPlan-based VQA corpus that rigorously tests if natural-language Chain-of-Thought reasoning causally mediates driving trajectory planning.
• It combines real-world driving data with modular input ablations, structured semantic context, and plan-aligned supervision to isolate the effects of reasoning, visual input, and priors.
• Empirical analysis reveals a reasoning-planning disconnect, showing that planning performance remains largely unchanged despite the removal of generated reasoning, highlighting reliance on textual shortcuts.

DriveMind is a large-scale, nuPlan-based driving Visual Question Answering corpus designed specifically to study whether the “reasoning” produced by vision-language driving models actually causes their final planning output (Song et al., 6 Oct 2025). It couples real-world driving authenticity, rich structured semantic context, plan-aligned Chain-of-Thought supervision, and modular input ablations so that visual input, ego/history priors, and navigation priors can be removed cleanly during evaluation. In the cited study, DriveMind is not introduced primarily as a performance benchmark; it is introduced as a causal-analysis instrument for diagnosing whether planning in contemporary VLM driving agents is genuinely mediated by reasoning or instead dominated by textual shortcuts such as ego state, history, and navigation priors (Song et al., 6 Oct 2025).

1. Origin, scope, and research question

DriveMind was created in response to a specific methodological gap in VLM-based autonomous driving research: the common assumption that natural-language reasoning, especially Chain-of-Thought, causally drives the final trajectory plan had remained largely unverified (Song et al., 6 Oct 2025). The motivating paper argues that existing driving datasets were insufficient for this question. On one side, nuScenes-based VQA datasets provide real-world sensor data but lack the rich structured semantic annotations needed for fine-grained reasoning. On the other, simulation-based CoT datasets such as CARLA-based DriveCoT are controllable but suffer from sim-to-real mismatch and unrealistically idealized behavior (Song et al., 6 Oct 2025).

Against that background, DriveMind was designed with four explicit properties: it preserves real-world driving authenticity by building on nuPlan, supplies rich structured semantic context, includes plan-aligned CoT supervision, and is modular enough to support clean information ablations (Song et al., 6 Oct 2025). This modularity is central. The dataset is intended not merely to test whether a model can answer questions or predict plans, but to test whether planning survives when specific sources of information are removed.

A frequent misconception in VLM driving is that fluent textual reasoning implies causally grounded decision making. DriveMind was built precisely to challenge that assumption. Its core question is whether a model that first produces natural-language reasoning and then predicts a trajectory is actually planning from that reasoning, or whether the reasoning is only a plausible byproduct of training (Song et al., 6 Oct 2025).

2. Dataset construction and representational design

DriveMind contains approximately 50,000 samples spanning 61 driving scenarios, with the training set drawn from the nuPlan training split (Song et al., 6 Oct 2025). To address long-tail imbalance across scenarios, the authors use square-root weighted stratified sampling. Each sample is constructed through a three-stage pipeline.

In the first stage, nuPlan logs are parsed into structured multimodal inputs. Ego priors include current ego state, historical ego states, and the navigation goal. Visual input is formed by rescaling all camera images and stitching them into a three-row image grid corresponding to front, side, and rear views. Lane topology is encoded by identifying the ego lane and neighboring lanes, together with lane directions relative to vehicle heading. Traffic signal information includes current signal state and stop-line position for relevant lights. Dynamic objects within a 20-meter radius are extracted from nuPlan’s 3D annotations, with valid categories including vehicles, pedestrians, bicycles, and other obstacles such as cones or pillars. GPT-4.1 is then used to generate textual visual descriptors from cropped object images, such as “a white SUV” or “a pedestrian in a red jacket” (Song et al., 6 Oct 2025).

In the second stage, GPT-4.1 generates plan-aligned CoT. The prompt includes the preprocessed visual input, structured scene ground truth, and the expert future trajectory. The resulting CoT is organized into three parts: scene ground truth, causal analysis, and macro decision. The appendix further specifies a > block followed by an <answer> block, where the <think> block contains scene analysis and macro-level driving decision, and the <answer> block contains the final trajectory plan. A key constraint is that the macro decision is prompted to remain predictive rather than retrospective, so the generator is not allowed to “cheat” by treating the future trajectory as evidence (Song et al., 6 Oct 2025).

In the third stage, approximately 10% of samples, about 5,000, are manually reviewed by human experts, and the paper reports high logical correctness in the generated CoT (Song et al., 6 Oct 2025). The final sample contains visual inputs, textual priors comprising ego state, history, and navigation, and two targets: ground-truth CoT and ground-truth planning trajectory.

This design gives DriveMind an unusual combination of realism and intervention-friendliness. A plausible implication is that the dataset functions simultaneously as training data and as a controlled experimental substrate for causal diagnosis.

3. Task formulation and training regimes

DriveMind is formulated as a VQA-style instruction-following task for driving agents. The input is structured images plus a text prompt with priors, and the output is a natural-language CoT followed by a trajectory plan (Song et al., 6 Oct 2025). The model is trained to jointly produce the reasoning trace and the expert plan, which makes the benchmark suitable for testing whether reasoning truly mediates planning rather than merely co-occurs with it.

The paper evaluates representative VLM driving agents under both Supervised Fine-Tuning and Group Relative Policy Optimization. The representative models are Qwen2.5-VL as the main model, LLaVA-1.6 for scalability and generalization analysis, and Omnidrive as a state-of-the-art baseline with counterfactual reasoning (Song et al., 6 Oct 2025).

Under SFT, the authors define a family of ablation conditions: Base, CoT, Plan, Plan_NoV, CoT_NoHis, CoT_NoHis_Ego, CoT_NoPri, CoT_L, Plan_L_NoV, Omnidrive, and Omnidrive_NoPri (Song et al., 6 Oct 2025). These interventions progressively remove reasoning, visual input, history, ego state, or all priors. The training setup uses LoRA and cross-entropy, with LoRA rank $64$, LoRA alpha $64$, batch size $8$, warm-up ratio $0.05$, and a cosine learning-rate schedule, together with separate learning rates for the aligner, ViT, and LLM backbone (Song et al., 6 Oct 2025).

To test whether explicit policy alignment restores causal grounding, the study also applies GRPO to both the SFT-trained model and the base model directly. For this stage, 1,000 new VQA samples are drawn from nuPlan with the same scenario distribution as DriveMind. The rewards in the appendix are location reward, velocity reward, and format reward. For CoT_grpo, the weights are 0.45 for location, 0.45 for velocity, and 0.1 for format; for Base_grpo, the rewards are equally weighted. The GRPO configuration uses group size $8$, temperature $0.9$, and warm-up ratio $0.01$ (Song et al., 6 Oct 2025).

What matters encyclopedically is not only the training machinery but its purpose: the interventions are designed so that one can compare planning quality with and without explicit reasoning, with and without images, and with and without textual priors.

4. Empirical evidence for the reasoning-planning disconnect

The central empirical result associated with DriveMind is a consistent reasoning-planning disconnect (Song et al., 6 Oct 2025). The paper reports that CoT, Plan, and Omnidrive perform nearly identically on nuPlan metrics, which suggests that explicit reasoning does not materially improve planning and that planning can be carried out without the reasoning trace. An even more striking result is that Plan_NoV, which removes visual input as well as reasoning, performs almost as well as fully multimodal agents. The authors interpret this as strong evidence that the model can plan from textual priors alone (Song et al., 6 Oct 2025).

The converse intervention is much more destructive. When all priors are removed, performance drops sharply for both CoT_NoPri and Omnidrive_NoPri (Song et al., 6 Oct 2025). Representative open-loop trends reported in the paper are that CoT, Plan, Omnidrive, and Plan_NoV all score around $\sim 99$ open-loop at 1s, $\sim 97$ open-loop at 2s, and $\sim 92$ open-loop at 3s, whereas CoT_NoPri drops to about $64$0, and Base_grpo_NoPri drops further to about $64$1 (Song et al., 6 Oct 2025). The asymmetry is the point: removing CoT has only minor effects, while removing priors causes collapse.

The paper formalizes this finding as the Reasoning-Planning Decoupling Hypothesis, which states that current training paradigms are insufficient to forge a causal link between reasoning and planning, and that agents instead learn shortcuts from textual priors, with generated reasoning functioning as an ancillary byproduct rather than a causal mediator (Song et al., 6 Oct 2025).

This interpretation is narrower than a blanket rejection of reasoning in autonomous driving. It does not show that reasoning cannot matter; it shows that, under the examined training paradigms, high planning scores do not establish that reasoning is the cause of planning.

5. Diagnostic mechanisms: attention analysis and training-free probing

DriveMind is paired with a diagnostic methodology rather than only a dataset. One diagnostic component is sequence-level attention analysis. The paper reports that, for CoT_grpo, reasoning generation shows increasing image-token attention across layers, indicating that the reasoning sequence is at least visually grounded. However, when the model generates the planning sequence, attention to textual priors grows sharply, attention to generated reasoning rises somewhat and then fades, and image attention becomes negligible (Song et al., 6 Oct 2025). The reported planning-side values are image tokens $64$2, textual priors $64$3, and generated reasoning $64$4 (Song et al., 6 Oct 2025). A similar pattern is reported for Omnidrive.

The second diagnostic component is a training-free causal probe. The principle is that if planning is truly grounded in vision and reasoning, then small, semantically plausible perturbations to textual priors should not strongly alter the final plan. If the model is shortcut-reliant, small perturbations should induce large deviations (Song et al., 6 Oct 2025).

The first perturbation applies a small lateral velocity offset to ego priors, defined as $64$5 (Song et al., 6 Oct 2025). Robustness is quantified by the final lateral deviation between perturbed and original plans. The reported deviations are about $64$6 for CoT_grpo and about $64$7 for Omnidrive, far beyond a lane width (Song et al., 6 Oct 2025). The second perturbation inverts the lateral component of the historical trajectory, for example turning “merging from the left” into “merging from the right.” The paper reports direct contradictions in which the CoT remains visually consistent while the plan reverses direction because the priors were perturbed (Song et al., 6 Oct 2025).

These diagnostics matter because they convert an abstract concern about “shortcut learning” into measurable planning fragility. A plausible implication is that DriveMind shifts evaluation from output plausibility toward causal fidelity.

6. Position within the broader autonomous-driving literature

DriveMind sits within a broader movement toward multimodal, reasoning-centric autonomous driving, but it occupies a distinctive role within that landscape. DriVLMe is a video-language-model-based autonomous driving agent that combines perception, long-horizon navigation, dialogue, and route-planner tool use, and its contribution is to show how an LLM-based driving agent can talk, plan, and act in a simulator while exposing limitations such as unacceptable inference time, weak multi-turn interaction, and poor handling of unexpected situations (Huang et al., 2024). DriveMM, by contrast, is presented as an “all-in-one” large multimodal model for autonomous driving, trained through curriculum pre-training and large-scale unified fine-tuning to cover perception, prediction, planning, and driving-oriented QA across six public benchmarks (Huang et al., 2024).

Other subsequent systems explore more tightly coupled reasoning-and-planning architectures. MindDriver proposes a progressive multimodal reasoning chain of semantic understanding, semantic-to-physical space imagination, and physical-space trajectory planning, together with feedback-guided automatic annotation and progressive reinforcement fine-tuning (Zhang et al., 25 Feb 2026). Another MindDrive framework integrates a world-model-based Future-aware Trajectory Generator with a VLM-oriented Evaluator under a “what-if simulation – candidate generation – multi-objective trade-off” paradigm (Suna et al., 4 Dec 2025).

Within this literature, DriveMind differs in function. It is not primarily an end-to-end controller, nor primarily a planner, nor primarily a multimodal generalist model. It is a dataset and diagnostic framework whose purpose is to expose whether claimed reasoning in VLM driving agents is causally connected to planning (Song et al., 6 Oct 2025). This suggests that DriveMind belongs to the methodological infrastructure of the field: it provides the means to test whether future systems that generate CoT, future imagery, candidate trajectories, or interactive dialogue are actually using those intermediate artifacts as decision mediators rather than decorative outputs.

The broader significance is therefore epistemic as much as algorithmic. DriveMind argues that high planning scores are not enough, that CoT generation does not imply causal reasoning, and that future evaluations of VLM driving agents should include causal ablations and perturbation-based robustness tests alongside ordinary planning metrics (Song et al., 6 Oct 2025).