Unified MoE Formulation

• Unified MoE is a framework that unifies conditional routing, dynamic expert activation, and parameter sharing to improve scalability and cross-modal performance.
• It employs diverse strategies like token-level, context-level, and modality-specific gating alongside top-p thresholding to optimize expert allocation and load balancing.
• The formulation facilitates robust training, efficient deployment, and cross-domain generalization in applications ranging from large language models to anomaly detection.

A Unified Mixture-of-Experts (MoE) formulation brings together diverse expert routing mechanisms, architectural strategies, and optimization approaches into a coherent framework applicable across modalities, tasks, and deployment scenarios. Modern unified MoE systems are characterized by conditional or dynamic expert activation, shared or hybrid expert pools, generalist backbone integration, and unified training objectives. These mechanisms enable sparsity, scalability, and specialization within a single model—mitigating negative interference, improving cross-domain generalization, and enabling production-scale deployment.

1. Canonical Unified MoE Formulations and Variants

Unified MoE systems extend the classic sparse MoE paradigm, which dispatches each token or representation to a small subset of experts, to the setting where expert activation and composition are conditioned on rich, unified routing criteria. The general unified MoE layer involves:

• Input $h \in \mathbb{R}^d$ (token/unit embedding),
• $N$ experts $\{E_i\}_{i=1}^N$, typically MLPs or specialized blocks,
• A gating network $$G(h) = \mathrm{softmax}(W_g h + b_g) \in \mathbb{R}^N$$,
• Top-$k$ or top-$p$ (cumulative-threshold) masking for sparsity,
• MoE output as $y = \sum_{i\in \mathcal{S}(h)} g_i(h)\,E_i(h)$ for active experts.

Several recent unified variants demonstrate substantial design diversity:

• Conditional MoE (Uni-Perceiver-MoE): gating driven by modalities, tasks, or token/context-derived attributes, enabling data-independent fusion at inference time for high efficiency (Zhu et al., 2022).
• Dynamic-Capacity MoE (UniMoE-2.0-Omni/UniMoE-Audio): top-$P$ gating for variable expert allocation per token; hybrid inclusion of routed (specialist), shared (generalist), and null (computation-null) experts to unify compute and adaptivity (Li et al., 16 Nov 2025, Liu et al., 15 Oct 2025).
• Unified MoE in Attention and FFN (UMoE): parameter sharing of expert pools across both attention and FFN sublayers, showing attention can be reformulated as specialist expert-mixing over contextualized token representations (Yang et al., 12 May 2025).
• MoE-in-MoE and Grouped Filtering (UniMMAD): hierarchical and groupwise expert composition to minimize parameter footprint while maintaining specialization across modalities and classes (Zhao et al., 30 Sep 2025).
• Physical-World Regularized MoE (UPRM): coarse-to-fine video understanding through structurally specialized experts and explicit regularization of the gating policy (Jin et al., 20 Feb 2026).

Unified MoEs are now instantiated in large multimodal LLMs, streaming code-switching ASR, anomaly detection, connectomic synthesis, fully omnimodal models, and unified large-scale training platforms.

2. Gating, Routing, and Dynamic Capacity Mechanisms

The unified MoE literature expands gating strategies beyond pure data-driven token-wise routing:

• Token-level: Each token is independently routed (deeply data-adaptive, high communication/parameter cost).
• Context-level: Gating input includes a global summary (improves gate stability in sequence models).
• Modality/task/attribute-level: Tokens with shared properties (e.g., modality, role, task) are routed collectively, enabling dense fusion during inference.
• Dynamic thresholding (Top-P): The number of experts per token varies, governed by cumulative gate probability, giving adaptive capacity per input (Liu et al., 15 Oct 2025, Li et al., 16 Nov 2025).
• Template-based and null experts: Inclusion of parameter-free experts to enable computation skipping for simple tokens (Li et al., 16 Nov 2025, Liu et al., 15 Oct 2025).

Regularization of expert usage—such as load-balancing penalties ensuring uniform or proportionate expert activation—is standard, e.g., using variance or KL terms on gate outputs (Li et al., 16 Nov 2025, Li et al., 2024, Zhu et al., 2022). In highly unified settings, experts are often allocated to modalities (text, vision, audio, video), and routing naturally aligns tokens with compatible expert subsets.

3. Architecture Integration and Parameter Sharing

Modern unified MoE models employ several integration and sharing approaches:

Model/Framework	Shared Across Layers	MoE Placement	Expert Pool Sharing
Uni-Perceiver-MoE (Zhu et al., 2022)	Yes (transformer backbone)	Both FFN, attn proj	Per-module, per-attribute
UMoE (Yang et al., 12 May 2025)	Yes	FFN, Attention	Joint expert pool (FFN, attn)
Uni-MoE(-2.0-Omni) (Li et al., 2024, Li et al., 16 Nov 2025)	Yes	FFN (Transformer backbone)	Experts assigned per modality
SC-MoE (Ye et al., 2024)	Yes	FFN (Conformer, Decoder)	Per-layer, per-direction
BrainCSD (Shen et al., 7 Nov 2025)	Hierarchical	ROI/Encoding/Refine	Module specific

Advanced architectures combine expert parameter re-use across different types of layers (attention/FFN: UMoE), across modalities, and across hierarchical levels. Generalist and omnimodal models (e.g., Uni-MoE-2.0-Omni) pretrain experts in a single-modality setting, then integrate them as routed experts in joint training with shared and null experts for unified downstream deployment.

Implementation-driven frameworks such as UniEP focus on unifying the deployment of diverse MoE topologies (top-$k$, top-$p$, overlapping expert groups) and communication patterns into scalable, auto-tunable mega-kernels for distributed high-throughput training (Zheng et al., 21 Apr 2026).

4. Training and Regularization Strategies

Unified MoE models are trained using multi-stage or curriculum-based strategies to control specialization, ensure stability, and foster cross-domain generalization:

• Stagewise expert pretraining: Each expert is initialized (often in a dense setting) on individual domain data (Liu et al., 15 Oct 2025, Li et al., 16 Nov 2025).
• Expert integration and gate warmup: The routing network and shared experts are trained (frozen domain experts) on balanced data to learn appropriate routing and general skills.
• Unified joint optimization: All parameters (experts, shared/backbones) are unfrozen for joint end-to-end multimodal or multitask training.
• Auxiliary losses: Load-balancing, variance reduction over expert usage (Zhu et al., 2022, Li et al., 16 Nov 2025, Li et al., 2024); specialist/generic regularizers.
• Instruction/LoRA adaptation: Low-rank adaptation applied to experts to enable modality/task-specific fine-tuning and efficient memory footprint (Jin et al., 20 Feb 2026, Li et al., 2024).
• RL-based joint preference optimization: GSPO and DPO for aligning omnimodal generative objectives (Li et al., 16 Nov 2025).

This staged design mitigates the risks of early negative transfer, specialist collapse, or expert starvation, achieving high downstream generalization and stable convergence even in highly imbalanced multimodal or multitask settings.

5. Applications Across Domains

Unified MoE formulations support an expansive range of application domains:

• LLMs/MLLMs: Omnimodal generation, cross-modal reasoning, image/text/audio/speech synthesis and understanding (Uni-MoE-2.0-Omni, Uni-MoE, UMoE) (Li et al., 16 Nov 2025, Li et al., 2024, Yang et al., 12 May 2025).
• Streaming and Non-streaming ASR: Real-time code-switching speech recognition with unified MoE routers supporting both modes (SC-MoE) (Ye et al., 2024).
• Anomaly Detection: Multi-modal, multi-class adaptive anomaly scoring using cross-gated MoE decompression (UniMMAD) (Zhao et al., 30 Sep 2025).
• Brain Analysis: Unified connectome synthesis and multitask brain trait prediction through hierarchical, neuroanatomically motivated MoEs (BrainCSD) (Shen et al., 7 Nov 2025).
• Security-oriented Video Understanding: UPRM integrates fine-grained and coarse-grained physical-world expert modeling for threat localization and explainability in videos (Jin et al., 20 Feb 2026).
• Distributed LLM Training: Expert-parallel MoE systems for production-scale, hardware-efficient training (UniEP) (Zheng et al., 21 Apr 2026).

Unified MoEs not only absorb imbalanced data distributions without catastrophic forgetting, but their sparse, conditional routing results in efficiency, improved cross-modal accuracy, and practical deployment (e.g., reduction from 772M to 120M parameters in automated educational scoring (Fang et al., 18 Nov 2025)).

6. Implementation, Optimization, and Scalability

Realization of unified MoE models at scale necessitates advances in expert parallelism, numerical stability, parameter search, and deterministic token mapping:

• Unified expert-parallel mega-kernels: UniEP’s mega-kernel abstractions fuse dispatch, computation (GEMM), and reduction phases, enabling ~1.1–1.4× speedup over previous distributed MoE backends on clusters (Zheng et al., 21 Apr 2026).
• Deterministic token ordering: Ensures bitwise equivalence with serial computation even under aggressive GPU overlap and communication schedules (Zheng et al., 21 Apr 2026).
• Auto-tuning abstractions: Systematic search over the expert-parallel configuration space, leveraging analytic performance models for optimal kernel schedules.
• Group conv/grouped filtering: Efficient parallel group-wise computation in decoders (UniMMAD) and hierarchical MoE-in-MoE structures enable parameter savings up to 75% while retaining specialization (Zhao et al., 30 Sep 2025).
• Universal gating and attribute-based fusion: Gate sharing across all layers (UMoE); attribute-level routing for fusion and cost reduction (Uni-Perceiver-MoE).

Multi-modal and multi-task unified MoEs routinely realize FLOP costs and inference latencies comparable to their dense counterparts while scaling parameterization and activations over orders of magnitude.

7. Significance and Outlook

Unified MoE formulations mark a critical convergence in the design of scalable, generalist foundation models. By abstracting expert routing, activation, and specialization under a sparse, conditional regime, these models:

• Avoid negative cross-task and cross-modality transfer that prevents generalist models from matching specialist alternatives (Zhu et al., 2022).
• Support efficient, robust, and incremental adaptation to new domains and tasks (rapid extension with minimal parameter updates) (Fang et al., 18 Nov 2025).
• Harmonize generalized reasoning, modality integration, and conditional computation without sacrificing efficiency.
• Enable deployment on heterogeneous or resource-constrained hardware by maximizing parameter and activation sparsity.

Unified MoE paradigms are now foundational in the design of next-generation LLMs, MLLMs, domain-flexible generative models, and distributed expert-parallel training infrastructures.

Key references: (Jin et al., 20 Feb 2026, Li et al., 2024, Yang et al., 12 May 2025, Li et al., 16 Nov 2025, Liu et al., 15 Oct 2025, Zhu et al., 2022, Ye et al., 2024, Zhao et al., 30 Sep 2025, Shen et al., 7 Nov 2025, Zheng et al., 21 Apr 2026, Fang et al., 18 Nov 2025).