- The paper introduces a cached 7B vision-language backbone and 337M action expert that separate slow scene reasoning from fast per-tick waypoint prediction, sustaining 20 Hz control with 32.4 ms of model compute per tick.
- The paper shows that fresh control every tick raises route completion from 37.0% to 94.0% versus LMDrive, reduces route deviations and timeouts, and improves zero-shot transfer to unseen towns, although vehicle collisions increase.
- The paper uses staleness-augmented training to match deployment-time cache delays, achieving 0.031 m validation waypoint L1 error versus 0.037 m for synchronous-only training while exposing limitations in long-route hazard negotiation and simulation-only evaluation.
The latency problem in language-guided driving
Language-conditioned driving agents built on large vision-LLMs inherit instruction following and scene reasoning, but a 7B backbone that re-reads its full visual history at every control step cannot sustain the 20 Hz tick rate of a closed-loop simulator. The canonical open-source agent of this family, LMDrive, resolves the conflict by invoking its model on every second 50 ms tick and replaying the stale command in between, so half of all control outputs ignore the newest observations. This paper argues that the conflict is architectural rather than fundamental: instruction understanding and temporal scene aggregation change slowly, while mapping the current frame to a trajectory is computationally light but must happen quickly. The work builds on DriveVLA-W0's asymmetric coupling, in which the backbone never attends to expert tokens, and takes it to an asynchronous conclusion by making the backbone's per-layer key-value cache the interface between a slow deliberative system and a fast reactive one.
Architecture: cached context provider plus per-tick action expert
The system attaches a 337M-parameter, 32-layer transformer expert to a frozen LMDrive stack (frozen perception encoder, Q-Former compressing each frame into four visual tokens, frozen LLaMA-7B backbone). Each expert layer projects its tokens through rectangular 4096×512 projections into the backbone's attention geometry, concatenates the backbone's cached keys and values of the corresponding layer, and performs a single cross-attention operation. Because the backbone never attends back to expert tokens, the cache evolves identically whether or not the expert runs — the property that makes caching exact.
At deployment, the slow system appends one frame's four tokens to the cache every K=4 ticks (0.2 s) via an incremental forward pass whose cost is independent of history length; full cache rebuilds occur only on instruction changes, runtime notices, or episode boundaries. The fast system reads whatever cache exists at every tick, consuming ten tokens (a state token carrying previous waypoints, speed, and target point; the current frame's four visual tokens; five learnable waypoint queries) and regressing five waypoints in one forward pass. A golden equivalence test confirms numerical fidelity of incremental versus monolithic prefill, with waypoint differences below 4 mm at the fp16 noise floor.
Staleness-augmented training
Since the deployed cache lags the world by up to K−1 ticks, training draws δ∼U{0,…,δmax} per sample and masks the backbone prefix visible to each supervised frame so it ends δ frames in the past. All frames of a clip are supervised in parallel through per-block attention masks, reproducing deployment-time cache visibility without modifying the data pipeline. Notably, this augmentation regularizes even under synchronous evaluation: the randomized-δ expert leads its δ=0-only twin at every training epoch, converging to 0.031 m validation waypoint L1 against 0.037 m.
Closed-loop results
On the 32 LangAuto-Short town05 routes in CARLA 0.9.10, with identical simulator settings for all agents:
| Agent |
Control rate |
DS |
RC |
IS |
| LMDrive (replayed ticks) |
10 Hz |
28.8 ± 0.8 |
37.0 ± 0.4 |
0.80 ± 0.04 |
| FastSlow-LMDrive, frame-skipped |
10 Hz |
34.0 |
82.1 |
0.45 |
| FastSlow-LMDrive (ours) |
20 Hz |
32.9 ± 0.7 |
94.0 ± 2.6 |
0.37 ± 0.02 |
A frame-skip ablation running the identical trained expert at the baseline's 10 Hz cadence cleanly separates the two factors: the driving-score gain over the baseline traces to the expert itself, while per-tick freshness raises route completion from 82.1 to 94.0, cuts route deviations from 11.3 to 4.3 per kilometer, reduces timeouts from 1.3 to 0.08, and lowers red-light violations from 10.4 to 6.9. The authors are candid that the composite driving score difference between the two expert rows falls within run-to-run spread. They also note a trade-off: per-kilometer vehicle collisions rise from 3.2 to 11.2 as the agent completes routes through traffic it has not learned to negotiate safely, while inattention-related infractions (layout collisions, off-lane driving) drop sharply.
Latency
The legacy recompute path costs 88.6 ms of model compute per step at a 40-frame history and grows from 59 ms at 10 frames to 169 ms at 100 frames, exceeding the 50 ms budget at every history length — which is precisely why frame-skipping exists. The proposed system holds model compute flat at 32.4 ms per tick (7–9 ms expert, amortized ~5 ms cache maintenance), roughly 18 ms below budget regardless of history length, on a single RTX 3090 Ti. The measured end-to-end in-loop agent step is 58 ms median due to unoptimized sensor formatting and harness overhead; wall-clock rate is about 17 Hz, though CARLA's synchronous mode still delivers fresh control at every tick. An important structural point: without cache reuse, the 20 Hz row is not available as a design option at all.
Zero-shot transfer
Trained exclusively on short town05 clips, the expert transfers zero-shot to unseen towns, holding 84.3% completion on town01 and 94.4% on town02 against 94.0% in-domain, where the baseline reaches 40.5% and 30.7%. On town02 the system achieves 2.4× the baseline's driving score and 3.1× its completion, suggesting stale control hurts most where scenes are unfamiliar. Long-route transfer probes the boundary: on eight LangAuto-Long town03 routes, the system completes 85.4% of route but accumulates enough vehicle collisions and red-light violations to collapse its penalty factor to 0.04, yielding a 2.96 driving score. Completion transfers even to this tier; hazard negotiation over long horizons does not, which the authors attribute to the scarcity of dense signalized encounters in short training clips rather than to the architecture.
Limitations
The study is simulation-only, trained on short routes from a single town. Hazard negotiation does not transfer to long routes, vehicle collisions increase alongside completion, the PID controller inherits gains tuned for the slower baseline, and statistical support is thin: two runs per configuration for the main comparison, single runs for the ablation and transfer rows, with observed run-to-run spread up to 1.6 DS. Part of the open-loop waypoint advantage also reflects a privileged teacher-forced state token unavailable to the frozen baseline head, though closed-loop results confirm the gain without it. The authors explicitly state the results should not be read as evidence of road readiness.
Conclusion
This paper demonstrates that a language-guided driving agent can be decomposed along time rather than compressed: a frozen 7B backbone maintains an incrementally updated KV-cache representation of instruction and history, while a 337M expert reads that cache at every 50 ms tick, lifting route completion from 37.0 to 94.0 over the frame-skipping baseline and reducing open-loop waypoint error nearly fourfold relative to the backbone's own action head. Staleness-augmented training aligns the expert with the asynchronous cache it encounters at deployment. Open questions include whether long-route training data suffices to close the hazard-negotiation gap, and whether the cache interface can be enriched — for example via world-model pretraining or fast-slow disagreement as a runtime safety trigger.
