- Natural language information directs the AI-based driving vehicle's course through a robotic, decision-making framework, predispositioning, driveable conversational dialogues linked to external contextual models for multi-modal charting
- The AI effectively decreases driving errors are minimized by up to 88.6% during outlier event suppression
- Dynamic (item: driver-omic) instructions improve performance beyond static tags, as ' go straight ' might influence decision more than 'go forward. These annotation help in recurring loop mistakes
Overview and motivation
Instruction-grounded driving—where a passenger's natural-language utterance directly reshapes the vehicle's motion plan—remains largely untested on real-world data. Prior language-conditioned planners such as DriveMLM (Wang et al., 2023), GPT-Driver (Mao et al., 2023), and LMDrive (Sima et al., 2023) operate either in simulation (CARLA) or over fixed, templated command vocabularies, and none accept free-form, referential passenger directives tied to real sensor logs. This paper addresses that gap by adapting OpenEMMA (Xing et al., 2024), an open-source MLLM-based end-to-end driving framework derived from Waymo's EMMA, to consume doScenes [doScenes] instructions—free-form passenger-style annotations grounded to nuScenes (Caesar et al., 2019) scenes—and measuring whether such conditioning measurably alters predicted trajectories. The work is positioned as the first application of doScenes to a planning task and the first analysis of fine-grained instruction conditioning on a real-world trajectory predictor.
Methodology
The experimental design is deliberately minimal to isolate the effect of language. OpenEMMA's pipeline—scene description, object identification, and intent estimation via chain-of-thought reasoning over front-camera frames and ego state, followed by generation of a 10-step speed–curvature trajectory—is left architecturally unmodified. The single intervention is the injection of a doScenes instruction into the scene-description prompt template, framed as a passenger directive with a safety-priority clause: comply unless unsafe, in which case explain and choose the safest alternative. Because the instruction is the only controlled change, output variation is attributable to the language, modulo VLM stochasticity, which the authors acknowledge as a source of run-to-run variance.
Evaluation uses LLaVA-1.6-Mistral-7B, selected for native compatibility with OpenEMMA's prompt format and for reproducibility (fully open-source, locally runnable); stronger VLMs such as Qwen2.5-VL (Bai et al., 19 Feb 2025) are explicitly deferred. The evaluation set comprises 849 annotated scenes drawn from the actionable subset of doScenes (1,423 of 3,924 annotations with clear language–trajectory alignment; non-actionable instructions such as "wait at the red light" are excluded). Because multiple annotators provide distinct instructions per nuScenes clip, each instruction is treated as an independent evaluation, and ADE is computed at the instruction level. Inference is expensive: roughly 20 minutes per clip, with the full evaluation running over seven days on RTX 4090 and RTX 6000 Ada GPUs.
Results
The headline finding concerns outlier suppression. Across all 849 scenes, the no-instruction baseline produces extreme failures—out-of-bounds waypoints extending past the scene boundary, frequently in scenes where the ego vehicle should remain stationary (e.g., at a red light or behind a pedestrian). Instruction conditioning reduces mean ADE by 98.7% relative to this baseline, a figure driven almost entirely by eliminating these catastrophic failures. After a 97.5th-percentile filter removes roughly twenty outlier scenes, the aggregate improvement narrows to 5.1% (mean ADE 2.879 without instructions versus 2.929 averaged over all doScenes instructions; the best instruction per scene achieves 2.732). The authors are transparent that only the best-phrased instructions beat the baseline once outliers are removed, and they interpret the two observations jointly: the VLM is prone to unrealistic predictions, and prompt quality modulates trajectory accuracy.
| Condition |
Mean ADE (all scenes) |
Mean ADE (Q97.5 filtered) |
| No instruction |
6201.443 |
2.879 |
| doScenes, best instruction |
9.999 |
2.732 |
| doScenes, average instruction |
78.527 |
2.929 |
| doScenes, worst instruction |
151.420 |
3.110 |
Two prompt-level analyses characterize what makes an effective instruction. With respect to length, ultra-short (0–4 words) and short (5–8 words) instructions degrade performance relative to no instruction (e.g., 3.323 and 3.076 versus 3.001), while typical-length (9–12 words) and long (19+ words) instructions improve it, with the longest bucket performing best (2.784 versus 2.795). With respect to referentiality, instructions referencing dynamic agents yield the lowest ADE (2.764), outperforming static-only (3.027) and non-referential (3.397) prompts; the interpretation is that references to moving objects supply temporal and relational context that the planner can exploit, whereas generic directives such as "go straight" add little beyond the visual scene.
Qualitative cases underscore that small aggregate differences mask safety-critical behavioral changes: in one scene, the instruction "Stop at the curb on the right side of the road right before the crosswalk" prevents the model from generating waypoints through an active crosswalk; in another, "Go straight when the stoplight turns green" corrects an otherwise predicted left turn into oncoming traffic. These examples indicate that instruction conditioning functions as a safety mechanism even where its average ADE benefit is modest.
Limitations and open questions
The paper concedes several substantive constraints. First, doScenes annotations are a posteriori reconstructions of what a passenger might say, not in-the-moment utterances, creating a distributional gap that may reduce instruction compliance in real settings. Second, the prompt template imposes an "always-act" bias: the model produces a trajectory even for unsafe, contradictory, or infeasible instructions rather than recognizing that no action should be taken. Third, ADE alone cannot confirm instruction understanding—it only measures proximity to the human driver's chosen path, one of many reasonable trajectories—and the evaluation is open-loop, so closed-loop validation remains an open requirement. Fourth, restricting to the actionable subset substantially reduces the effective dataset size, and the authors frame the results as a feasibility demonstration rather than a definitive performance measurement. Finally, the reliance on a single, comparatively weak backbone (LLaVA-1.6-Mistral-7B) leaves open whether stronger MLLMs or alternative architectures—graph-structured planners such as DriveMLM or closed-loop systems such as LMDrive—would exhibit the same prompt-quality sensitivity.
Conclusion
This paper establishes a reproducible, instruction-conditioned baseline for real-world motion planning by injecting doScenes passenger directives into OpenEMMA, demonstrating that free-form natural language can both prevent catastrophic planner failures (a 98.7% mean ADE reduction driven by outlier suppression) and subtly steer trajectory accuracy when phrased well (up to 5.1% improvement after filtering). The prompt-level findings—that dynamic referentiality and moderate-to-long phrasing improve alignment—offer actionable guidance for instruction design in human-in-the-loop AV systems, while the acknowledged limitations of open-loop ADE evaluation, the always-act bias, and the single-model scope delineate the boundaries of the claim that language meaningfully conditions driving behavior.