---
title: 'DriveMoE: Autonomous Driving MoE Framework'
url: https://www.emergentmind.com/topics/drivemoe
type: topic
---

# DriveMoE: Autonomous Driving MoE Framework

Searching arXiv for the cited DriveMoE paper and closely related autonomous-driving MoE works.

DriveMoE is a Mixture-of-Experts-based end-to-end autonomous driving framework that augments the Vision-Language-Action baseline Drive-\(\pi_0\) with two conditional-computation mechanisms: a **Scene-Specialized Vision MoE** for dynamic camera-view selection and a **Skill-Specialized Action MoE** for behavior-specific planning. Built on a pretrained PaliGemma VLM backbone and a flow-matching trajectory module, it is designed to address two specific weaknesses of end-to-end driving systems: inefficient processing of surround-view, multi-camera, multi-timestep visual input, and mode averaging over diverse or rare driving behaviors such as aggressive turns, emergency braking, and yielding. In Bench2Drive closed-loop evaluation, the paper reports state-of-the-art performance, with substantial gains over its Drive-\(\pi_0\) baseline in Driving Score, Success Rate, and open-loop trajectory error [2505.16278].

## 1. Problem setting and Drive-\(\pi_0\) baseline

DriveMoE is situated in end-to-end autonomous driving, where the model maps raw sensory input directly to future trajectory prediction and downstream control. The paper motivates the method through two persistent difficulties. First, autonomous driving must process surround-view, multi-camera, multi-timestep visual input, and uniform encoding of all views causes token count and compute to grow rapidly. Second, driving behavior is multi-modal and long-tailed: frequent behaviors such as lane following dominate the data, while rare but safety-critical cases require qualitatively different responses. The paper argues that a single monolithic policy tends to average across such modes, which degrades rare maneuver handling [2505.16278].

DriveMoE inherits the overall Vision-Language-Action structure from Drive-\(\pi_0\), itself described as an adaptation of the embodied-AI model \(\pi_0\) to autonomous driving. The inherited components are a pretrained **PaliGemma VLM** backbone, a **fixed text prompt** such as “Please predict future trajectory,” structured **vehicle state**, and a **flow-matching-based action module** for future trajectory generation. The model output is a predicted future trajectory of **10 waypoints**, which is converted by a common **PID controller** into **throttle, brake, and steering** commands [2505.16278].

The concrete input configuration used by DriveMoE is narrower than an exhaustive surround-view encoder. In practice, the model uses **two sequential front-view images** plus **one dynamically selected camera view**. The front-view sequence is intended to expose temporal cues such as surrounding-agent velocity, while the dynamic view is selected by the Vision MoE according to scene context. Together with the fixed prompt and vehicle state—including current and historical position, velocity, acceleration, and heading angle—these inputs are fused by the VLM backbone before trajectory planning [2505.16278].

## 2. Dual-MoE architecture

DriveMoE adds two MoE mechanisms to Drive-\(\pi_0\): one before expensive visual processing and one inside the action decoder. The front half of the architecture performs conditional perception by selecting contextually relevant camera views. The back half performs conditional planning by activating behavior-specialized experts in the flow-matching decoder. This yields conditional computation at both perception and action levels [2505.16278].

| Component | Function | Concrete implementation |
|---|---|---|
| Scene-Specialized Vision MoE | Selects relevant camera views | Two front views plus one Top-1 dynamic view |
| Skill-Specialized Action MoE | Activates behavior-specific planning experts | One shared expert and six non-shared experts in implementation |
| Base VLA stack | Multimodal fusion and trajectory generation | PaliGemma backbone plus flow-matching action module |

The two MoE components solve different subproblems. The Vision MoE reduces redundancy in multi-view perception by selecting the most relevant camera according to the current scene and route intention. The Action MoE replaces dense FFNs in the trajectory transformer with skill-specialized experts so that behaviorally distinct cases, such as merging or emergency braking, are not forced into a single undifferentiated decoder. The paper explicitly frames this as an attempt to improve both **efficiency in multi-view perception** and **robustness to diverse, rare, and safety-critical driving behaviors** [2505.16278].

## 3. Scene-Specialized Vision MoE

The Vision MoE is driven by the claim that not all camera views are equally useful in every situation. Rather than encoding all views uniformly, DriveMoE uses a lightweight router \(\boldsymbol{R}_{\text{vision}}\) that takes the **front-view embedding** \(\boldsymbol{e}_t^{\text{front}}\) and the **future goal waypoint** \(\boldsymbol{g}_t\) from the route planner, and predicts a probability distribution over camera views:
\[
\boldsymbol{p}_t = \text{Softmax}\left(\boldsymbol{R}_{\text{vision}}\left(\boldsymbol{e}_t^{\text{front}}, \boldsymbol{g}_t\right)\right),
\]
where \(\boldsymbol{p}_t \in \mathbb{R}^{N}\) and \(p_t^v\) is the selection probability of view \(v\) at timestep \(t\) [2505.16278].

Operationally, the method uses **Top-1** dynamic view selection at test time. The “experts” in this part of the system are effectively the available camera streams or view pathways, not separate per-view neural FFN experts. Because the routing happens before the expensive visual backbone, non-selected views can be skipped entirely. The selected views are augmented with learnable **view-specific positional embeddings** so that camera identity and cross-view spatial structure are retained [2505.16278].

The router is trained with manually generated heuristic camera-view annotations derived from future trajectory, maps, and object context. The appendix-level rules described in the paper include the following mappings: for **intersection turning**, annotate the front-side camera facing the intended exit direction; for **lane change**, annotate a corresponding rear-side camera if the target lane has the same direction and a front-side camera if opposing-lane occupancy is required; for **highway merging/cut-in**, annotate the side camera facing the merging location; and for **yielding to emergency vehicles**, annotate the camera facing the approaching emergency vehicle. With binary camera-view labels \(\boldsymbol{y}_t \in \{0,1\}\), the vision router is trained by
\[
\mathcal{L}_{\text{Vision-Router}} = -\lambda_0 \sum_{v=1}^{N} \boldsymbol{y}_t^v \log(\boldsymbol{p}_t^v).
\]
This makes the visual routing explicitly supervised rather than latent [2505.16278].

## 4. Skill-Specialized Action MoE

The Action MoE is inserted into the **flow-matching trajectory transformer**. The paper’s core modification is to replace each dense FFN in the decoder with a mixture of expert FFNs. For decoder layer \(\ell\), with hidden input \(\mathbf{h}^{(\ell-1)} \in \mathbb{R}^d\), the router \(\boldsymbol{R}_{\text{action}}\) produces routing probabilities
\[
\boldsymbol{r}_k^{(\ell-1)} = \text{Softmax}(\boldsymbol{R}_{\text{action}}(\mathbf{h}^{(\ell-1)})), \quad k \in \{1,2,\dots,\boldsymbol{K}\},
\]
and each expert computes
\[
\boldsymbol{y}^{(\ell)} = E^{(\ell)}(\mathbf{h}^{(\ell-1)}).
\]
The layer output is written as
\[
\boldsymbol{h}^{(\ell)} = \sum_{k=1}^{K}\boldsymbol{r}_k^{(\ell-1)}\boldsymbol{y}_k^{(\ell-1)} + \sum_{m=1}^{M}\boldsymbol{y}_m^{(\ell-1)}.
\]
The paper notes that the notation is imperfect: the text uses \(K\) and \(M\) inconsistently, while the implementation description is more specific [2505.16278].

In implementation, the Action MoE uses **1 shared expert and 6 non-shared experts**. The non-shared experts are aligned with Bench2Drive skill categories: **Merging**, **Overtaking**, **Emergency Brake**, **Give Way**, and **Traffic Sign**, with an additional **ParkingExit** expert in the best ablation. The routing supervision comes from driving skill labels, trained via the paper’s shorthand cross-entropy expression
\[
\mathcal{L}_{\text{Action-Router}} = -\boldsymbol{y_k}\log(\boldsymbol{r}_k).
\]
The overall action-side loss is
\[
\mathcal{L}_{\text{Action}} = \lambda_1\mathcal{L}_{\text{FM}} + \lambda_2\mathcal{L}_{\text{Action-Router}}.
\]
The text also states that a **load-balancing regularization loss** \(\mathcal{L}_{\text{LB}}\) is used to avoid expert collapse and that router noise is injected following DeepSeekMoE, but the explicit load-balancing formula and coefficient are not provided in the paper [2505.16278].

The routing regime is described generically as sparse, with only the top-ranked experts activated. The narrative mentions Top-1 or Top-2 conceptually, but the concrete implementation states that the **top-3 experts selected by the action router are utilized during training and inference**. This is the most specific setting reported and therefore the operative one for the published results [2505.16278].

## 5. Training, inference, and control generation

DriveMoE is trained on the **Bench2Drive official training set**, using the **base set** of **1000 clips**, with **950 train** and **50 validation/test** for fair comparison. The supervision is entirely supervised or imitation-style: future waypoint learning through **flow matching**, heuristic camera-view labels for the vision router, and benchmark-defined skill labels for the action router. No reinforcement-learning stage is described [2505.16278].

Training proceeds in **two stages**. In **Stage 1**, for the first **10 epochs**, both Vision MoE and Action MoE use **ground-truth experts only**, while the routers are trained jointly. This is a teacher-forcing-style stabilization phase. In **Stage 2**, training continues for **5 additional epochs** from the Stage-1 checkpoint, but selected camera views and action experts are taken from the routers’ own predictions rather than ground-truth annotations. This reduces train-test mismatch and exposes the model to its own routing errors during optimization [2505.16278].

The reported optimization settings are: separate optimizers for the **VLA** and **Action MoE experts**, learning rate \(5 \times 10^{-5}\), warmup steps enabled, gradient clipping with max norm \(1.0\), and gradient accumulation to simulate batch size \(1024\). The loss weights are stage-dependent. In Stage 1, \(\lambda_0 = 0.05\) for the Vision Router loss, \(\lambda_2 = 0.03\) for the Action Router loss, and \(\lambda_1 = 1\) for the flow-matching loss. In Stage 2, \(\lambda_2 = 0.025\), while other hyperparameters remain the same [2505.16278].

At inference time, DriveMoE uses router predictions rather than annotations. The visual input is always the pair of sequential front-view images plus one **Top-1** dynamic camera view. The action module uses the router-selected experts, concretely the **top-3** experts in the implementation. The output is a trajectory of **10 future waypoints**. A fixed PID controller then computes control. The reported gains are
\[
K_{P}^{\text{turn}} = 1.25,\quad K_{I}^{\text{turn}} = 0.75,\quad K_{D}^{\text{turn}} = 0.3
\]
for steering and
\[
K_{P}^{\text{speed}} = 5.0,\quad K_{I}^{\text{speed}} = 0.5,\quad K_{D}^{\text{speed}} = 1.0
\]
for speed. The desired speed is computed from the **7th waypoint** and the steering angle from the **10th waypoint** [2505.16278].

## 6. Empirical results and ablations

The main benchmark is **Bench2Drive closed-loop evaluation** in **CARLA v0.9.15.1**, with **220 short routes** designed to stress challenging corner cases. The primary metrics are **Driving Score (DS)**, **Success Rate (SR)**, **Efficiency**, **Comfort**, and **Avg. L2**. Bench2Drive also reports ability categories: **Merging**, **Overtaking**, **Emergency Brake**, **Give Way**, and **Traffic Sign** [2505.16278].

The headline comparison against Drive-\(\pi_0\) is:

| Model | DS | SR | Efficiency | Comfort | Avg. L2 |
|---|---:|---:|---:|---:|---:|
| Drive-\(\pi_0\) | 60.45 | 30.00 | 168.41 | 14.88 | 0.56 |
| DriveMoE | 74.22 | 48.64 | 175.96 | 15.31 | 0.38 |

The paper states that, relative to Drive-\(\pi_0\), DriveMoE improves **Driving Score by 22.8%** and **Success Rate by 62.1%**. It also surpasses strong external baselines reported in the same table, including DiffAD, DriveTrans, and DriveAdapter, and is presented as state of the art on Bench2Drive [2505.16278].

Ability-specific results also indicate differential gains. DriveMoE reaches **34.67** on Merging, **40.00** on Overtaking, **65.45** on Emergency Brake, **40.00** on Give Way, and **59.44** on Traffic Sign, for a mean of **47.91**. The corresponding Drive-\(\pi_0\) scores are **29.35**, **36.58**, **48.83**, **40.00**, and **54.45**, with mean **41.84**. The largest gain occurs in **Emergency Brake**, rising from **48.83** to **65.45**, which is consistent with the paper’s emphasis on rare or safety-critical behaviors [2505.16278].

Ablations isolate the two MoE components. Removing the Vision MoE yields **DS 68.68** and **SR 42.45**; removing the Action MoE yields **DS 67.31** and **SR 40.56**; full DriveMoE reaches **DS 74.22** and **SR 48.64**. This suggests that the two mechanisms are complementary rather than redundant. The Vision MoE ablation is especially sharp: adding a fixed back, front-left, or front-right camera improves modestly over the Drive-\(\pi_0\) baseline, but a dynamic view without supervision rises to **DS 69.71**, and a **supervised** dynamic view reaches the full **74.22**. On the action side, the number of non-shared experts also matters: **6 non-shared experts** is best; **13** or **44** experts reduce performance, which the paper attributes to load imbalance and data fragmentation [2505.16278].

Open-loop router accuracy further clarifies the asymmetry between the two routing problems. On the validation set, the **Vision Router accuracy** is **88.85%**, while the **Action Router accuracy** is **65.40%**. A plausible implication is that view selection is a more directly supervised and geometrically constrained routing problem than behavior-skill selection, which is more abstract and multimodal. The paper does not provide latency, FLOPs, or throughput numbers for DriveMoE, so its efficiency claim remains architectural rather than numerically benchmarked [2505.16278].

## 7. Position within driving-oriented MoE research

DriveMoE occupies a specific point in the broader autonomous-driving MoE design space. Its distinctive feature is the combination of **view-level routing before vision encoding** and **skill-level expert routing inside a trajectory planner** within a VLA framework [2505.16278]. That differs from several adjacent lines of work.

In planning-oriented MoE systems, **TrajMoE** argues that the appropriate trajectory prior is scene-dependent and applies MoE inside a trajectory-scoring transformer over a predefined trajectory vocabulary, with reinforcement learning used only to refine scoring heads rather than to control the vehicle directly [2512.07135]. **EMoE-Planner** instead uses supervised **scene-level hard routing** and scene-specific query banks in a pure learning planner on NuPlan, emphasizing interpretable scenario experts and interaction-oriented loss shaping [2505.12311]. These systems are close in spirit to DriveMoE’s action-side specialization, but they are not VLA systems and do not combine that specialization with pre-backbone camera routing.

Other works place MoE in different parts of the stack. **MiniDrive** uses FE-MoE in the visual feature-engineering stage before the language model, prioritizing token-efficient multi-view question answering rather than trajectory generation [2409.07267]. **MoSE** implements hierarchical skill-oriented routing across perception, prediction, and planning inside a small VLM, but is evaluated primarily on reasoning and suggestion tasks rather than closed-loop control [2507.07818]. **D\(^3\)-MoE** and **DriveFine** represent later generative-planning variants: the former uses dual disentanglement with longitudinal and lateral expert pools in a diffusion planner [2606.04884], while the latter uses a block-MoE that separates generation and refinement experts in a masked diffusion VLA planner [2602.14577]. In perception, **MoE-RAM** applies statistic-augmented, decoupled routing and aggregation to semantic segmentation rather than control [2512.06664]. In driver monitoring, **VDMoE** uses spatial and temporal experts for drowsiness, cognitive load, HR, and RR estimation from RGB cabin video [2410.21086].

Taken together, this literature indicates that autonomous-driving MoE has developed along several axes: **perception-side routing**, **planning-side routing**, **reasoning-oriented routing**, **diffusion-generation routing**, and **driver-monitoring routing**. DriveMoE’s contribution is to combine two of those axes—scene-specialized perception and skill-specialized planning—inside a single VLA end-to-end driving system. Its strongest empirical evidence lies in closed-loop Bench2Drive gains; its least specified aspects are the exact flow-matching internals, the Action MoE load-balancing formulation, and the systems-level efficiency characteristics [2505.16278].

Source: https://www.emergentmind.com/topics/drivemoe