LangAuto Benchmark: Language-Guided Driving

• LangAuto is a closed-loop, language-conditioned benchmark that evaluates autonomous driving agents using natural language instructions and multimodal sensor data.
• It integrates multi-view cameras, LiDAR, and free-form navigation commands to assess route completion, safety infractions, and agent robustness under adversarial conditions.
• LangAuto serves as the primary protocol for testing systems like LMDrive, BEVDriver, GraphPilot, and TLS-Assist, driving research in vision-language decision-making.

The LangAuto benchmark is a closed-loop, language-conditioned evaluation suite for autonomous driving agents in the CARLA simulator. It emphasizes comprehensive assessment of language-guided driving, robust agent behavior under adversarial and ambiguous conditions, and the integration of vision, language, and action in a realistic, end-to-end setting. LangAuto provides the primary evaluation protocol for recent vision-language decision-making systems including LMDrive, BEVDriver, GraphPilot, and TLS-Assist, supporting systematic measurement of instruction adherence, route completion, and infraction-penalized driving performance (Shao et al., 2023, Winter et al., 5 Mar 2025, Schmidt et al., 18 Nov 2025, Schmidt et al., 14 Nov 2025).

1. Benchmark Definition and Motivation

LangAuto was introduced to overcome limitations of prior benchmarks that relied on high-level discrete commands or open-loop evaluation. In contrast, LangAuto tasks the agent with following rich, free-form natural language navigation instructions (e.g., “Turn right at the next intersection and proceed straight for 300 meters”) while responding in real time to the multimodal driving environment. This setting captures real-world linguistic diversity, instruction ambiguity, and driver-assistant interactions, advancing the paper of language-planning integration in closed-loop driving (Shao et al., 2023).

Key goals include:

• Language-grounded closed-loop evaluation: Agents interact with the environment based on natural instructions, not pre-defined command lists or open-loop samples.
• Robustness to adversarial events: Misleading or infeasible instructions are introduced (~5% of the time), requiring refusal or safe fallback.
• End-to-end sensor fusion and planning: Full integration of multi-view camera, LiDAR, and language input within a real-time feedback loop.

2. Dataset Composition and Driving Scenarios

LangAuto comprises a corpus of 64,000 instruction-following driving clips (2–20 s each), collected with 4 RGB cameras (front, left, right, rear), 1 LiDAR, and dense meta-data (vehicle state, object signals). It covers 8 official CARLA towns under 21 combinations of weather and time-of-day. Navigation instructions are diversified across 56 types, each with 8 phrase variants (via ChatGPT), spanning turns, following, lane changes, and abstract goals.

Route length and complexity define difficulty splits:

• LangAuto-Tiny: <150 m (short, simple)
• LangAuto-Short: 150–500 m (medium)
• LangAuto: >500 m (long, challenging)

Episodes may include:

• Notice instructions: Safety prompts such as “Watch for the pedestrian ahead,” triggered by adversarial simulator events (total 464,000).
• Misleading instructions: Commands impossible in context, simulating user errors or ambiguous communication.
• Sequential instructions: Multi-part tasks (10% of episodes), e.g., “Continue straight, then left, then right at the next light,” testing multi-instruction comprehension (Shao et al., 2023, Winter et al., 5 Mar 2025).

3. Evaluation Metrics and Protocol

LangAuto adopts and extends the CARLA Leaderboard metrics to the natural-language, closed-loop paradigm. The core metrics are:

• Route Completion (RC): Fraction of the reference route traveled by the agent.

$$RC = \frac{d_{\mathrm{travelled}}}{d_{\mathrm{total}}}$$

• Infraction Score (IS): A multiplicative penalty, initially 1.0, decayed at each infraction (e.g., collision, red-light, off-road) by predefined factors $\alpha_k$:

$IS = \prod_{k} (1 - \alpha_k)^{n_k}$

• Driving Score (DS): Primary ranking index, computed as:

$DS = RC \times IS$

Per-kilometer infraction counts for vehicle and pedestrian collisions, as well as open-loop prediction errors (Average Displacement Error, Final Displacement Error) are also tracked where relevant (Shao et al., 2023, Winter et al., 5 Mar 2025, Schmidt et al., 18 Nov 2025, Schmidt et al., 14 Nov 2025).

Agents are evaluated over all towns, weather, and route splits, with each run repeated 3× for robustness. No explicit held-out town split is standard; instead, results are averaged over the full test spectrum.

4. Baselines, Extensions, and Agent Integration

LangAuto serves as the evaluation bedrock for successive generations of language-based driving agents:

• LMDrive: First LLM-based closed-loop planner integrating multi-view vision, LiDAR, and natural-language navigation. Baseline DS ranges from ~36% (LLaVA-v1.5) to ~11% (random) depending on agent and route.
• BEVDriver: Incorporates BEV feature maps (projecting RGB/LiDAR into a top-down unified space) via a Q-Former, yielding up to +18.9% DS over previous methods on Short tracks.
• GraphPilot: Augments training (and optionally inference) with serialized traffic scene graphs, achieving up to +15.6% (LMDrive) and +17.7% (BEVDriver) DS gains through relational topology conditioning.
• TLS-Assist: A plug-and-play visual redundancy module for explicit traffic light/sign detection, reducing red-light and stop-sign infractions by 28–81% (LMDrive) and up to 56% (BEVDriver), with relative DS gains up to 14% (Winter et al., 5 Mar 2025, Schmidt et al., 14 Nov 2025, Schmidt et al., 18 Nov 2025).

Agent/Variant	DS (overall)	RC (%)	IS	Effect
LMDrive (baseline)	40.38	52.77	0.79	Base vision-lang agent
+ TLS-Assist	46.08	--	--	+14% DS, −28% RLI, −81% SSI
BEVDriver	44.70	49.70	0.90	BEV/top-down grounding
+ TLS-Assist	48.12	--	--	+7.7% DS, −56% RLI
+ GraphPilot (SG10)	52.61	64.10	0.87	+17.7% DS

RLI: red-light infractions; SSI: stop-sign infractions.

5. Distinctive Features and Practical Impact

LangAuto distinguishes itself among driving and language benchmarks by:

• Natural-language command coverage far exceeding prior CARLA Leaderboard or Town05 tasks, enabling the paper of semantic ambiguity, referent resolution, and instruction compositionality.
• Closed-loop, sensor-rich agent interaction, capturing cumulative error and temporal consistency, rather than open-loop snapshot prediction.
• Adversarial and multi-sentence evaluation paradigms, including deliberately impossible or unsafe instructions, for robust fail-safe reasoning analysis.
• Plug-and-play extension support: modular methods such as TLS-Assist and external relational priors (GraphPilot scene graphs) can be injected without architecture change, directly benchmarking value in context.

A plausible implication is that LangAuto serves as a strong baseline for future multimodal, language-centric control systems seeking sim-to-real transfer and robust, safety-aware navigation.

6. Limitations, Challenges, and Research Directions

Despite advances fueled by LangAuto, current models exhibit persistent challenges:

• Low long-route completion: Most agents exhibit RC < 50% on routes >1 km due to error accumulation.
• Handling of misaligned and sequential commands: Even top-performing models occasionally follow misleading instructions or lose context on 3-part tasks, reducing DS by 2–3%.
• Generalization to out-of-domain towns and instructions: No built-in sim-to-real transfer mechanism; models remain limited to CARLA’s visual and linguistic range (Shao et al., 2023, Winter et al., 5 Mar 2025).
• Safety-critical reasoning gaps: Explicit rule-enforcement modules (e.g., TLS-Assist) and relational priors mitigate but do not eliminate violation and collision penalties.

Open research questions involve multi-turn dialogue, multi-agent negotiation, guaranteeable constraint satisfaction under adversarial inputs, and true end-to-end co-adaptation of vision, language, and control. Extending LangAuto to human-in-the-loop and real-world deployment settings is an ongoing research direction.

7. Comparative Positioning and Benchmark Influence

Relative to preceding and contemporary evaluation suites, LangAuto offers:

• Higher linguistic diversity than command-driven CARLA tracks or static language-driven data.
• Comprehensive adversarial robustness through misleading command and notice tracks.
• Direct measurement of language-planning-vision synergy, catalyzing developments such as BEVDriver’s spatial fusion, GraphPilot’s graph-conditioning, and TLS-Assist’s natural-language scene augmentation (Winter et al., 5 Mar 2025, Schmidt et al., 14 Nov 2025, Schmidt et al., 18 Nov 2025).

This prominence has established LangAuto as the de facto benchmark for language-based, closed-loop planning research, providing critical insight into both the power and limits of current vision-language autonomous driving agents.