Variable-Rate Mixture-of-Transformers (MoT)

• Variable-Rate Mixture-of-Transformers is an asynchronous multi-expert architecture that decouples slow scene understanding from rapid action control in autonomous driving.
• It integrates a frozen, pretrained vision-language model for high-level reasoning with a trainable transformer for fast decision-making through joint attention over shared latent spaces.
• Empirical results show improved driving scores, reduced latency by 86.8%, and lower trajectory errors compared to synchronous transformer-based systems.

Variable-Rate Mixture-of-Transformers (MoT) refers to an asynchronous, multi-expert transformer architecture enabling efficient and effective unification of vision-language reasoning with action-level decision-making in end-to-end autonomous driving scenarios. The design, as operationalized in AutoMoT, leverages a split between high-level reasoning—performed at a low frequency by a pretrained vision-language transformer—and low-latency action generation—executed at a high frequency by a trainable transformer—fusing their capabilities via shared key-value memory and joint attention blocks. This architecture achieves competitive performance on both open-loop and closed-loop benchmarks for autonomous driving while sharply reducing inference latency, all without sacrificing the general reasoning capability of large pretrained vision-LLMs (Huang et al., 16 Mar 2026).

1. System Architecture and Expert Roles

The Variable-Rate Mixture-of-Transformers instantiates two distinct transformer “experts”:

• Understanding Expert (UE): A frozen, large-scale vision-LLM (Qwen3-VL-4B) that processes multi-view RGB frames and textual prompts, producing high-level reasoning tokens (scene representations) at a slow update rate.
• Action Expert (AE): A 1.6B-parameter transformer, trained from scratch, which ingests BEV features, the current RGB frame, and learned action queries, producing time-aligned control latents at a fast update rate.

Both experts operate within a unified latent space, interconnected by a persistent shared key-value (KV) cache mechanism. The UE generates keys and values $$\{K^l_{\text{scene}}(\tau), V^l_{\text{scene}}(\tau)\}_{l=1}^L$$, where $\tau$ indexes the most recent slow-cycle update. The AE at timestep $t$ computes $$\{Q^l_{\text{act}}(t), K^l_{\text{act}}(t), V^l_{\text{act}}(t)\}$$ for each layer $l$, subsequently decoding to meta-actions, temporal waypoints, and spatial route points.

2. Fast–Slow (Variable-Rate) Inference Paradigm

Variable-Rate MoT architectures explicitly decouple the update frequencies for reasoning and action:

• Slow Reasoning: UE operates at $f_{\text{slow}}\approx 2.6$ Hz, updating visual language scene representations every $0.38$ s.
• Fast Control: AE operates at $f_{\text{fast}}\approx 20$ Hz, enabling action outputs every $0.05$ s.

At every action frame $t$:

1. The most recent UE scene encoding at $\tau$0 is reused.
2. AE computes new action queries and latents.
3. Joint attention at each network layer fuses the slow (scene) and fast (action) key/value sets.

A formulaic blending of stale (slow) vs. fresh (fast) representations is possible, but in the AutoMoT system, simple concatenation and attention are employed, which achieves the desired asynchronous fusion effect.

3. Joint Attention and Latent Space Sharing

The “mixture” in Variable-Rate MoT is actualized through joint attention mechanisms:

At every AE transformer layer $\tau$1, the self-attention operation is replaced by joint attention over concatenated UE and AE KV:

$\tau$2

The attention update becomes:

$\tau$3

This ensures that the action pathway has direct access, via attention, to both pretrained reasoning outputs and contemporaneous action tokens. The use of a persistent KV cache for slow updates provides temporal stability for scene representations, while fast action tokens allow rapid control adaptation.

4. Vision–LLM Integration

In the AutoMoT implementation of Variable-Rate MoT:

• The UE (Qwen3-VL-4B) is strictly frozen and handles all general scene-understanding tasks, such as VQA, using semantic prompts as input. No autonomous driving (AD)-specific fine-tuning is performed on this expert.
• The AE is randomly initialized and trained on downstream AD tasks—meta-action decision modeling (NuSync, PDM-Meta) and spatiotemporal trajectory planning (nuScenes, PDM-Lite).
• Ablation studies confirm that freezing the UE preserves general VQA/external reasoning performance (e.g., LingoQA ≈ 67.0), while fine-tuning the UE on AD-specific tasks risks catastrophic forgetting and yields marginal plan-level benefits.

This modular strategy exploits the pretrained reasoning capacity of VLMs for scene understanding while reserving system plasticity for task-specific action modeling.

5. Empirical Performance and Latency Reduction

Extensive benchmarking on scene understanding, planning, and control tasks demonstrates significant advantages for the Variable-Rate MoT approach. Key findings:

Method	Driving Score ↑	Success Rate ↑
TransFuser++	84.21	67.27 %
SimLingo	85.07	67.27 %
AutoMoT (MoT)	87.34	70.00 %

Method	L2@1s↓	L2@2s↓	L2@3s↓	Coll. Avg ↓
DriveTransformer	0.16	0.30	0.55	0.07 %
OpenDrive-VLA	0.15	0.31	0.55	0.10 %
AutoMoT (MoT)	0.14	0.29	0.54	0.07 %

Setting	L2@1s↓	L2@2s↓	L2@3s↓	Latency (s)↓
Synchronous (UE+AE)	0.140	0.290	0.537	0.38
AutoMoT (Async/MoT)	0.141	0.293	0.544	0.05

Empirical results indicate:

• Decoupling reasoning from action control yields superior driving scores and lower open-loop trajectory errors than direct end-to-end or synchronous transformer baselines.
• The per-step inference latency is reduced by 86.8% under asynchronous, variable-rate scheduling relative to synchronous alternatives.
• Scene understanding metrics on diverse VQA datasets are also preserved or improved.

6. Functional Implications and Model Boundary

The juxtaposition of frozen large-scale vision-language modeling and trainable action modeling exposes the functional boundary of pretrained VLMs in the AD context:

• For scene-understanding tasks, semantic prompting of the frozen UE yields competitive multi-task results without AD-specific fine-tuning.
• For action-level tasks—decision making and trajectory generation—AE fine-tuning remains essential; pretrained reasoning alone is insufficient due to distributional misalignment and absence of task-specific knowledge.
• Fine-tuning the UE for AD tasks leads to catastrophic forgetting of general VQA capabilities, as evidenced by ablation results, highlighting the utility of the Variable-Rate MoT’s modular, heterogeneous update-frequency paradigm.

7. Significance and Extensions

Variable-Rate Mixture-of-Transformers presents a methodologically robust solution to the inference-latency and task-alignment challenges intrinsic to integrating VLMs into end-to-end driving frameworks (Huang et al., 16 Mar 2026). By fusing asynchronous expert transformers via joint attention over shared latent spaces, AutoMoT demonstrates that pretrained reasoning can be preserved for perception while enabling high-speed, responsive control, with empirical gains in both accuracy and computational efficiency. This suggests generalizability to other fast-slow sensorimotor domains requiring hierarchical reasoning and low-latency action.