Multi-gate MoE: Advanced Multi-task Framework

• Multi-gate MoE is a neural architecture that uses independent gating to selectively combine shared expert networks for efficient multi-task learning.
• The framework mitigates negative transfer by enabling nuanced expert selection and dynamic parameter sharing across various industrial and research scenarios.
• Extensions incorporate hierarchical gating, dynamic routing, and continual learning strategies to address challenges like expert collapse and task imbalance.

Multi-gate Mixture-of-Experts (MMoE) is an influential neural architecture paradigm for multi-task and multi-scenario learning, where multiple tasks can selectively leverage a set of shared or semi-shared expert networks via independent gating mechanisms. MMoE enables nuanced modeling of task relationships and supports efficient parameter sharing, mitigating negative transfer and accommodating complex industrial prediction requirements. The framework has been adopted and extended for large-scale recommendation, ranking, continual learning, soft sensing, and multimodal reasoning, with various innovations in gating, expert selection, and hierarchical architectures.

1. Fundamental Principles of MMoE

The core architectural principle of MMoE is to decouple expert selection from expert functionality for each task. An input $x \in \mathbb{R}^f$ is processed through a shared embedding module $f(x;\theta)$ to produce a shared latent representation $z$. This representation is simultaneously fed into $K$ parallel expert MLPs $\{\varphi_i\}_{i=1}^K$, each mapping $z$ to expert-specific outputs. For every task $t = 1, \ldots, N$, an independent gating network $g_t$ produces a $K$-categorical softmax distribution, yielding per-task soft assignments over experts.

The output for task $t$ is constructed as a convex mixture of expert outputs:

$$\psi_t(x) = \sum_{i=1}^K g_{t,i}(z) \cdot \varphi_i(z)$$

and is subsequently transformed by a task-specific "tower" network $\Phi_t$ to yield the prediction $\hat{y}_t(x) = \Phi_t(\psi_t(x))$ (Huang et al., 2023, Zou et al., 2022).

MMoE thus supports soft expert-sharing, where each task dynamically learns both which experts to consult and to what degree, as opposed to conventional MoEs with hard or shared gating.

2. Architectural Variants and Extensions

MMoE has inspired several architectural extensions for more refined multi-task or multi-scenario learning:

• Deep Hierarchical MMoE: AESM² introduces scenario-level and task-level MMoE modules stacked hierarchically, enabling separate modeling of shared versus scenario/task-specific factors. Gating is sparsified via negative KL-divergence–based top-K selection for both scenario- and task-specific/ shared experts, with automatic expert selection at each layer (Zou et al., 2022).
• Expert Hierarchies: HoME employs a two-layer hierarchy of experts, with level-1 "meta-experts" capturing shared and coarse-grained knowledge, and level-2 "fine-experts" operating at the task level. Tasks take weighted combinations over shared, category-specific, and task-specific experts, mitigating expert collapse and degradation, and enhancing representation granularity (Wang et al., 2024).
• Dynamic Routing and Selection: In AESM² and CL-MoE, gating includes stochastic exploration (via added Gaussian noise) and adaptive or dual routing mechanisms, where both instance-level and task-level routers are used in soft combination to modulate expert usage (Huai et al., 1 Mar 2025, Zou et al., 2022).
• Continual and Multimodal Learning: CL-MoE integrates MMoE with continual learning. It employs a dual-router system (instance- and task-level) for expert selection and a momentum-based parameter update mechanism (MMoE) that merges old and new expert parameters via adaptive momentum coefficients determined by expert usage across tasks (Huai et al., 1 Mar 2025).

3. Mathematical Formulation and Data Flow

The typical MMoE computation involves:

• Shared embedding: $z = f(x; \theta)$
• Expert outputs: $e_i(x) = \varphi_i(z)$
• Per-task gate: $g_{t,i}(x) = \mathrm{softmax}_i(W_t^g z + b_t^g)$
• Weighted mixture: $\psi_t(x) = \sum_{i=1}^K g_{t,i}(x) \cdot e_i(x)$
• Final prediction: $\hat{y}_t(x) = \Phi_t(\psi_t(x))$

Forward computation proceeds by embedding, parallel expert computation, gating per task, aggregation, and application of task-specific towers (Huang et al., 2023).

Extensions such as AESM² introduce additional sparsification, whereby experts are classified as task-/scenario-shared versus specific via KL divergence between gating distributions and prototype distributions (one-hot or uniform), enforcing structured expert assignments (Zou et al., 2022). In HoME, normalization (BatchNorm, Swish), feature privatization, and hierarchical gating decompose representations into coarse and fine components (Wang et al., 2024).

Dynamic MMoE, as in CL-MoE, employs soft combination of instance- and task-level gate vectors:

$$w_i(x; t) = \beta \cdot r_{\mathrm{inst}}(x)_i + (1-\beta) \cdot r_{\mathrm{task}}(t)_i$$

and updates expert parameters by fusing old and freshly fine-tuned parameters via per-expert momentum coefficients $\lambda_i$:

$$\theta_{t,i} = \lambda_i \theta_{t-1,i} + (1 - \lambda_i) \phi_{t,i}$$

where $\gamma > 0.5$ controls the trade-off between retention and plasticity (Huai et al., 1 Mar 2025).

4. Gate and Expert Network Design

Key implementation details for MMoE modules include:

• Expert Networks: Typically small MLPs (e.g., two hidden layers, $d_e=64$), individually parameterized, using smooth activation functions (Mish or Swish) to facilitate gradient flow. Dropout and $\ell_2$ weight decay are standard for regularization (Huang et al., 2023, Wang et al., 2024).
• Gate Networks: Each task's gating network is an independent MLP or linear transformation over the shared representation, outputting $K$ logits passed to softmax. Activations (Mish, Swish) and dropout ($p\approx0.2$) are used to prevent overfitting. Initializations ensure near-uniform distribution at the start of training (Huang et al., 2023, Wang et al., 2024).
• Normalization and Input Privatization: HoME applies batch normalization and Swish activations to expert outputs, aligning numerical scales and preventing expert collapse (e.g., >90% zeros with ReLU). Feature privatization via LoRA-style elementwise gating ensures experts receive distinct input subspaces, addressing underfitting for sparse tasks (Wang et al., 2024).
• Sparsification and Selection: AESM² uses negative KL divergence between gating distributions and ideal (specific/shared) prototypes to identify and select top-K scenario-/task-specific and shared experts, masking others out of the gating vector (Zou et al., 2022).

5. Comparative Experimental Performance

MMoE and its extensions consistently demonstrate superior empirical performance over hard or soft parameter sharing baselines, particularly by mitigating negative transfer and balancing task competition:

• Soft Sensor Modeling: Balanced MMoE (MMoE + GradNorm) outperforms hard-/soft-share baselines for sulfur recovery unit prediction, both by portraying task relationships through per-task expert selection and dynamically balancing gradients to avoid training domination by any one task. Empirically, 4–8 experts strike a good balance between separation and computational cost (Huang et al., 2023).
• Industrial and Recommendation Systems: AESM² yields significant AUC gains over PLE and vanilla MMoE (e.g., +0.23 to +1.01 points depending on scenario and task), notably in highly sparse or scenario-segregated settings. Online A/B testing confirms improvements in CTR, CVR, and gross merchandise volume (Zou et al., 2022).
• Short Video Platform Ranking: HoME achieves +0.52% average GAUC lift over vanilla MMoE, and statistically significant gains over state-of-the-art variants with equivalent expert blocks. Submodule ablations confirm cumulative benefits from normalization, hierarchy masking, and input privatization (Wang et al., 2024).
• Continual Visual Question Answering: CL-MoE attains an AP of 51.34% and AF (average forgetting) of −0.02% on 10 tasks, outperforming the previous best by over 7 AP points and ≈9 AF points, confirming the importance of dual routing and dynamic momentum update for stability and knowledge retention (Huai et al., 1 Mar 2025).

6. Limitations, Challenges, and Open Problems

MMoE architectures, though flexible, present several practical challenges:

• Expert Collapse and Degradation: Unnormalized or improperly activated experts may yield degenerate outputs, impairing effective gating. HoME demonstrates that batch normalization and activation choice are essential to prevent expert inactivity and to preserve the utility of shared experts (Wang et al., 2024).
• Balancing Task Gradients: Without explicit balancing, gradients from data-rich tasks may dominate, causing the "seesaw phenomenon" where improvements in one task degrade others. The integration of GradNorm task gradient balancing in BMoE and sparsification in AESM² mitigates such effects (Huang et al., 2023, Zou et al., 2022).
• Scalability of Gating and Selection: In large-scale multi-scenario or multi-task production systems, gate and expert structure may become prohibitively costly. Hierarchical and sparsified extensions (HoME, AESM²) address capacity and efficiency trade-offs (Wang et al., 2024, Zou et al., 2022).
• Catastrophic Forgetting in Continual Learning: For non-stationary settings, as addressed in CL-MoE, static expert-sharing is insufficient for continual knowledge accumulation or protection against forgetting. Dynamic momentum-based parameter updates and dual routers provide robust mechanisms for continual adaptation (Huai et al., 1 Mar 2025).

7. Relations to Other MoE Variants

MMoE distinguishes itself from older MoE paradigms:

• Single-Gate MoE: Employs a solitary gating function for all tasks; lacks the capacity to model task-specific expert preferences, leading to uniform expert usage across tasks (Huang et al., 2023, Zou et al., 2022).
• Hard MoE: Sparse expert selection via top-K gating is challenging to optimize (often non-differentiable), and using shared gates restricts expert specialization (Huang et al., 2023).
• PLE and CGC: Hierarchical MoE variants introduce additional shared- and task-specific experts in tree-like structures, but may still lack fine-grained scenario adaptation or dynamic, automatic expert selection as in AESM² (Zou et al., 2022).
• Advanced Hierarchies and Routers: Recent work layers multiple MoE modules or fuses global and local routers (as in CL-MoE and HoME), further improving expert utilization, knowledge sharing, and continual adaptation (Huai et al., 1 Mar 2025, Wang et al., 2024).

In summary, MMoE and its hierarchical, adaptive, and dynamic extensions constitute the state-of-the-art framework for flexible, robust, and scalable multi-task and multi-scenario learning across diverse industrial and research domains.