Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles

Abstract: Most Human-Machine Interaction (HMI) research overlooks the maneuvering needs of passengers in autonomous driving (AD). Natural language offers an intuitive interface, yet translating passenger open-ended instructions into control signals, without sacrificing interpretability and traceability, remains a challenge. This study proposes an instruction-realization framework that leverages a LLM to interpret instructions, generates executable scripts that schedule multiple model predictive control (MPC)-based motion planners based on real-time feedback, and converts planned trajectories into control signals. This scheduling-centric design decouples semantic reasoning from vehicle control at different timescales, establishing a transparent, traceable decision-making chain from high-level instructions to low-level actions. Due to the absence of high-fidelity evaluation tools, this study introduces a benchmark for open-ended instruction realization in a closed-loop setting. Comprehensive experiments reveal that the framework significantly improves task-completion rates over instruction-realization baselines, reduces LLM query costs, achieves safety and compliance on par with specialized AD approaches, and exhibits considerable tolerance to LLM inference latency. For more qualitative illustrations and a clearer understanding.

• The paper introduces a scheduling-centric LLM framework that converts ambiguous natural language instructions into explicit, human-readable scheduling scripts for maneuver-level control.
• It demonstrates significant improvements in instruction realization rate (0.84) and safety by integrating LLM-driven scheduling with MPC-based planners for robust vehicle maneuvers.
• The work validates its approach using the POINT benchmark, providing strong empirical and qualitative evidence for transparent and ISO-compliant autonomous vehicle control.

Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles

Motivation and Problem Definition

Traditional human-machine interaction paradigms in autonomous driving predominantly target driver-centric interfaces, which assume direct driver intervention. As the field moves toward SAE L4-L5 autonomy, especially for applications such as Robotaxi, the primary vehicle users transition to passengers, requiring the HMI to support intuitively expressed natural language instructions with highly diverse phrasing and intent Figure 1.

Figure 1: The framework addresses passenger-driven open-ended instruction realization via natural human-like commands.

The central technical hurdle is bridging high-level, open-ended natural language instructions to actionable, safety-critical low-level control without sacrificing interpretability or traceability along the decision pipeline. Most prior work focuses on infotainment or standardized navigation, neglects maneuver-level driving instructions, and either fails to generalize beyond templated input or offers poor transparency. Furthermore, efficient closed-loop evaluation tools for such an open-ended interaction paradigm in realistic traffic scenarios are lacking.

Scheduling-Centric LLM Framework

The proposed framework decomposes instruction realization into three stages utilizing a scheduling-centric architecture Figure 2: natural language parsing, planner script scheduling, and low-level motion planning and control.

Figure 2: The LLM interprets open-ended instruction, generates a scheduling script, which orchestrates multiple MPC-based planners using real-time asynchronous triggers and feedback.

Passenger instructions are modeled as high-level, potentially ambiguous natural language utterances contextualized with a traffic scene description. A LLM serves as a high-level interpreter, mapping each instruction to an ordered sequence of atomic driving behaviors (lane keeping, lane change, acceleration, braking, etc.), with reference to context to mitigate ambiguity or hallucination.

The LLM output is an explicit, human-readable scheduling script, which orchestrates a modular suite of motion planners via asynchronous event triggers. This approach decouples long-horizon, low-frequency semantic reasoning (handled by the LLM) from real-time, high-frequency, safety-critical continuous control (handled by MPC-based planners and LQR controllers), ensuring both interpretability and robustness to LLM errors and latency Figure 3.

Figure 3: Mode-based categorization of LLM-driven driving: single-shot scheduling (Mode III, adopted here) enables contextual adaptability with minimal LLM overhead.

Benchmark: POINT

Due to the absence of dedicated evaluation frameworks, the paper introduces the POINT benchmark: a high-fidelity closed-loop testbed, which augments the nuPlan simulator with 1,050 rigorously validated open-ended passenger instruction–scenario pairs Figure 4. Instructions leverage wild-form conversational phrasings and capture high-risk maneuver tasks such as overtaking and pulling over. The benchmark measures intent recognition, instruction realization rates, safety, compliance, and trajectory efficiency, and provides strong “specialized” and “instruction-realization” baselines for ablation.

Figure 4: Distribution of instruction intent types reveals focus on high-risk maneuver tasks, supporting challenging closed-loop evaluation.

Experimental Results

Intent Recognition: Recognition accuracy demonstrates strong scaling with LLM size and context, with only the largest generative models (DeepSeekV3, Qwen-2.5-72B) exceeding 85% on open-ended, context-rich instructions Figure 5. This highlights the importance of advanced reasoning and extensive world knowledge for instruction understanding at open-domain scale.

Figure 5: Intent recognition accuracy of LLMs scales with model size and improved context handling.

Instruction Realization and Safety: Across all metrics, the proposed framework exhibits a realization rate of 0.84—surpassing the best LLM-based baseline by a considerable margin (by 64%-200%). Collision and compliance metrics are on par with specialized modular planners, confirming the effectiveness of hierarchical decoupling and script-based scheduling for ensuring safety even when faced with risky or non-standard instructions.

Ablation demonstrates the necessity of traffic context and script scheduling for robust interpretation and execution: omitting scene context or using single-atomic planners causes drastic realization drops (Table in main text).

Qualitative Analysis: Case studies show the system correctly infers latent maneuver intent, interprets ambiguous passenger requests, and safely executes complex instruction chains, as depicted in execution visualizations Figure 6.

Figure 6: Keyframes from instruction execution: adaptive multi-stage planner transitions in response to context and open-ended commands.

Latency Analysis: The framework’s decoupling strategy ensures strong robustness to LLM inference latency Figure 7: even under significant decision delays, safety remains uncompromised due to the real-time, feedback-driven MPC inner loop. The instruction-realization rate degrades gracefully, not catastrophically, with increasing LLM latency.

Figure 7: LLM latency sensitivity: Safety is preserved owing to fast, decoupled low-level control, with only modest impact on realization score.

Comparative Analysis

The paper’s taxonomy divides LLM-driven AD systems into three deployable modes Figure 3:

• Mode I: System parameter initialization only (inflexible for real-time context adaptation)
• Mode II: Continuous, high-frequency LLM-in-the-loop (robust but cost and latency prohibitive)
• Mode III: Single-shot script-based scheduling with asynchronous triggers (combining transparency, adaptability, and efficiency), as adopted by this work.

Compared to direct, end-to-end VLM-based policies, this framework prioritizes traceability, modular safety assurance, and script-level transparency demanded by industrial ISO standards. The empirical results support that decoupling LLM scheduling from control both enables open-ended instruction realization and preserves model-based safety/certifiability.

Practical and Theoretical Implications

Practically, this architecture enables the next generation of human-centric, passenger-driven autonomous systems to robustly realize real-world, open-form instructions without requiring brittle grammar engineering or large-scale retraining on in-domain dialog. The explicit, script-driven interface provides interpretability and facilitates formal verification/audit trails, critical for regulatory approval and debugging.

Theoretically, the explicit timescale and abstraction-level hierarchy operationalizes Control Theory principles (hierarchical decoupling, event-triggered scheduling) in the context of LLM-driven embodied systems. This allows integration with both classical verification techniques and statistical learning pipelines, expanding the task space combinatorially for new planners/triggers with low marginal integration cost.

Limitations and Future Directions

The framework’s expressivity is limited by the finite library of atomic planners and the richness of the asynchronous triggers. Scaling to more complex maneuvers necessitates expansion of both, which is straightforward within this compositional architecture. The current implementation evaluates only on text-based scene descriptions due to nuPlan’s lack of ego-view support; integrating vision-LLMs and 3D urban reconstruction will be vital for closed-loop VLA assessment.

Further, the responsiveness and generalization can be improved by advancing trigger expressiveness and exploiting adaptive learning for planner composition. Integrating world-model vision reasoning, e.g., via 3D Gaussian splatting and spatio-temporal CoT, will further tighten the loop between perceptual modalities and interpretable decision programs.

Conclusion

This work advances autonomous vehicle interaction via a scheduling-centric LLM framework, achieving high-rate, safe, and traceable realization of open-ended instructions. The POINT benchmark establishes standardized closed-loop evaluation for this challenging paradigm. The results demonstrate the viability of script-based LLM-MPC cooperation for ISO-compliant human-vehicle interaction in AD, opening practical pathways for transparent, flexible, and safe deployment of LLM-driven autonomy in real-world settings (2604.08031).