Decoupled Mixture-of-Experts (DMoE) Overview

• Decoupled Mixture-of-Experts (DMoE) is a modular neural architecture that decouples expert networks from routing mechanisms, enabling heterogeneous expert designs.
• It employs lightweight, dynamic routers for input-aware expert selection, resulting in efficient inference and reduced computational cost.
• DMoE has demonstrated superior performance in applications like autonomous driving, large language models, and collaborative multi-agent systems.

A Decoupled Mixture-of-Experts (DMoE) is a modular variant of the Mixture-of-Experts (MoE) architecture that explicitly separates or "decouples" expert networks and their associated routing mechanisms from core model components or from each other. This approach allows for highly heterogeneous expert design, input-aware dynamic expert selection, and functional-level modularity, resulting in increased adaptability, improved capacity, efficient inference, and greater ease of maintenance and extensibility. DMoE is characterized by the placement of experts at coarse architectural or semantic granularity—rather than distributing homogeneous experts at every layer—and by the use of lightweight, independent routers for dynamic activation. Recent work demonstrates DMoE’s advantages in diverse fields including autonomous driving, multimodal perception, collaborative multi-agent fusion, LLMs, and knowledge-injection systems.

1. Conceptual Foundations and Key Variants

The central principle of DMoE is the decoupling of expert subnetworks and gating mechanisms from fixed module boundaries or from each other, to allow independent specialization and dynamic selection. In contrast to monolithic or tightly coupled MoE architectures—which interleave experts at every layer and require homogeneous expert design—DMoE approaches often operate at functional-module granularity or across orthogonally decoupled axes such as modality, semantic task, or spatiotemporal structure.

Several major DMoE instantiations include:

• Functional-module DMoE: Experts are associated with high-level modules (e.g., backbone, projection, fusion) and can be individually selected (Xiang et al., 11 Aug 2025, Wang et al., 1 Jun 2026).
• Axis-decoupled DMoE: Distinct expert sets are assigned to orthogonal aspects of a task (e.g., lateral and longitudinal axes in trajectory planning) and routed independently (Feng et al., 3 Jun 2026).
• Block-decoupled, system-level MoE: Pre-gating routers decoupled from the expert backbone enable improved batching, memory reuse, and prefetching in LLM inference (Cai et al., 2024, Feng et al., 29 May 2026).
• Dynamic expert DMoE for multi-agent systems: Each agent has its own dynamically generated expert, allowing explicit modeling of observation heterogeneity (Kong et al., 21 Sep 2025).
• Knowledge-injection DMoE: Experts encode external knowledge and are plugged into fixed points in the base model, routed only when uncertainty triggers a knowledge deficit (Yue et al., 12 Jun 2026).

This architectural decoupling enables new training regimes, curriculum learning, independent expert update, and efficient adaptation to continuously evolving data.

2. Architectural Design and Routing Mechanisms

Architectural decoupling in DMoE is realized through careful partitioning of model modules and expert pools, as well as through the design of lightweight or functionally independent routers.

• Example: Hierarchically Decoupled MoE for BEV Perception The CBDES MoE framework (Xiang et al., 11 Aug 2025) decomposes the bird’s-eye-view perception pipeline into four functional modules. MoE is applied at the backbone level: a pool of K=4 structurally heterogeneous vision backbones (Swin, ResNet, ConvNeXt, PVT) are maintained, with a self-attention router dynamically selecting a single expert per input. This router, structured with conv→pool→self-attn→MLP, is itself hierarchically organized but remains decoupled from downstream projection, fusion, and task heads.
• Dynamic Expert Routing and Input-Aware Sparsity CBDES MoE uses a soft-gating mechanism during training for gradient flow, and top-1 hard gating at inference for maximal efficiency—only the most relevant expert is evaluated, resulting in a K-fold reduction in inference cost (Xiang et al., 11 Aug 2025).
• Image-Level, Task-Specific DMoE In the context of traffic sign recognition, DMoE organizes a pool of YOLO-based experts where each is specialized for clear, small-object, or adverse-weather traffic sign detection. A lightweight MobileNetV3-based gating network is trained via cross-entropy on domain labels to assign each input to its optimal expert, fully decoupled at the image level (Wang et al., 1 Jun 2026).
• Axis-Wise Decoupling and Diffusion MoE D$^3$-MoE for end-to-end driving planning splits trajectory prediction along lateral and longitudinal axes. Independent routers, trained with self-supervision using kinematic clustering, select axis-specific transformer-based experts, ensuring behavioral and physical factorization (Feng et al., 3 Jun 2026).
• Block-Level and System-Decoupled MoE in LLMs dMoE for diffusion LLMs aggregates per-token router outputs at the block level, chooses a common coreset of experts, and restricts all subsequent token routing in that block to this subset, drastically reducing memory-bound bottlenecks (Feng et al., 29 May 2026). Read-ME decouples routing from backbone modules, replacing layer-wise routers with a pre-gating network and carefully aligns system policies for prefetching, batching, and caching (Cai et al., 2024).

3. Mathematical Formulations and Loss Functions

DMoE frameworks are often defined by explicit mathematical formalism for routing, fusion, loss, and decoupling:

• SAR Gating in CBDES MoE: Given input $x$ of size $B\times C\times H\times W$, the self-attention router computes an embedding $G= (1/N)\sum_{i=1}^N T'_i$ (with $T$ the output of hierarchical conv/pool / flatten / self-attn), then outputs expert logits $S = \mathrm{MLP}(G)$ and routing probabilities $P_{b,k} = \mathrm{softmax}(S)_{b,k}$. Inference selects $k^{*}=\arg\max_k P_{b,k}$, ensuring only a single expert is activated per sample (Xiang et al., 11 Aug 2025).
• Block-Level Routing: For a block $B$ of $M$ tokens, dMoE aggregates token-level scores $x$0 to produce $x$1, normalizes $x$2, then selects a coreset $x$3 comprising the smallest set whose scores sum to at least a threshold $x$4 (Feng et al., 29 May 2026). Only experts in $x$5 are used for routing.
• Load-Balance and Diversity Regularization: To prevent expert collapse, many DMoE systems employ load-balancing losses (e.g., $x$6 in CBDES MoE) or metric-based expert diversity losses (e.g., DEML in collaborative perception (Kong et al., 21 Sep 2025)).
• Self-Supervised Routing Loss: In D$x$7-MoE, routers are trained via KL divergence between soft cluster labels $x$8 and the router’s softmax outputs $x$9 for each physical axis (Feng et al., 3 Jun 2026).

4. Empirical Performance, Efficiency, and Generalization

Empirical results consistently demonstrate that DMoE architectures provide measurable gains over both monolithic single-expert and standard per-layer MoE approaches:

Method / Task	Metric	Performance Gain	Efficiency Gain	Reference
CBDES MoE (BEV percep.)	mAP/NDS (3D Det)	+1.6 / +4.1	4× backbone FLOPs reduction	(Xiang et al., 11 Aug 2025)
CBDES MoE TSR (TSR)	mAP₅₀–₉₅	+2.3	–39.4% GFLOPs vs. YOLOv9c	(Wang et al., 1 Jun 2026)
D$B\times C\times H\times W$0-MoE (Driving Plan.)	PDMS/EPDMS	Best: 91.3/87.5	Parallel styles, dynamic routing	(Feng et al., 3 Jun 2026)
dMoE (LLM, LLaDA2.0-mini)	Experts / block	↓ 69.5 → 14.6	Memory ↓ 77%, Latency ↑1.66×	(Feng et al., 29 May 2026)
Read-ME (LLM, MMLU)	Acc. (5-shot)	+10.1 over dense	6.1% lower latency, 10% p95 cut	(Cai et al., 2024)
CoBEVMoE vs. CoBEVT	Vehicle IoU	+1.5	Dynamic kernels per agent	(Kong et al., 21 Sep 2025)
DMoE Knowl. Injection	F1/EM/ACC	Best vs. RAG, LoRA	3× less memory, 7× faster infer.	(Yue et al., 12 Jun 2026)

A consistent finding is that hierarchical or axis-wise decoupling allows for architectural and functional diversity among experts, input- or context-aware specialization, and improved robustness across domain shifts or task conditions. Additionally, DMoE variants consistently achieve these improvements without linear increases in runtime or memory cost often associated with vanilla MoE.

5. Practical Applications and Limitations

DMoE has been adopted widely in:

• Autonomous Driving: BEV perception (Xiang et al., 11 Aug 2025), traffic sign recognition (Wang et al., 1 Jun 2026), and trajectory planning (Feng et al., 3 Jun 2026), leveraging domain or behavioral specializations.
• Collaborative Perception: Explicit modeling of agent heterogeneity in multi-vehicle perception (Kong et al., 21 Sep 2025).
• LLM Acceleration and Knowledge Injection: System-level decoupled MoE for LLM inference (Cai et al., 2024, Feng et al., 29 May 2026), and dynamic, uncertainty-triggered knowledge experts (Yue et al., 12 Jun 2026).
• Multimodal and Visual Tracking: Dual Mixture-of-Experts for disentangling spatiotemporal and modality-specific cues (Hong et al., 5 May 2026).

Limitations include the potential for increased model storage (multiple large experts), the need for explicit domain or task labels during training or routing (particularly in hard-gated DMoEs), and possible router misrouting in hybrid or novel domains. Disk or memory requirements may be a constraint in very large expert pools, although most runtime techniques keep active GPU memory and inference time low (Yue et al., 12 Jun 2026).

6. Extensions and Future Research Directions

Potential avenues for extending DMoE frameworks include the integration of learned or hybrid routing (e.g., combining BM25 with neural encoders), hierarchical expert organizations (e.g., clusters of passages to mid-level experts in knowledge-injection), adaptation to multi-modal domains, and combining DMoEs with continual learning schemes for more robust knowledge management (Yue et al., 12 Jun 2026). Soft gating and dynamic sparsification schemes may also address under-utilization of partially relevant experts. Further, DMoEs may be adapted for new domains—such as real-time RL, language-to-vision grounding, or multi-agent planning—by exploiting their modular, update-friendly, and functionally specific structure.

7. Comparison to Other MoE Architectures

Decoupled Mixture-of-Experts contrasts with both traditional dense and conventional monolithic MoE approaches:

• Single-expert and monolithic MoEs: Feature a shared, static backbone or per-layer homogeneous experts, limiting architectural diversity and robustness under domain shift (Xiang et al., 11 Aug 2025).
• DMoE: Enables per-module or per-axis routing, supports arbitrary architectural diversity, and leverages sparse, input-aware expert selection for input-adaptive efficiency and increased domain generalization.
• System-level DMoE: Decoupling router from the backbone unlocks system-level optimizations (pre-computing, caching, batching), unattainable in classic per-layer MoEs (Cai et al., 2024).

These distinctions result in improved adaptability, capacity, inference efficiency, and maintainability across application domains.