Multi-Gate Soft MoE Architecture

• Multi-Gate Soft MoE is a neural architecture that uses separate, task-specific gating networks to assign soft weights to expert models.
• It employs hierarchical gating and expert normalization techniques to optimize multi-task learning and mitigate issues like expert collapse.
• Empirical results demonstrate enhanced performance in applications like recommendations, sensor analytics, and multimodal fusion.

A Multi-Gate Soft Mixture-of-Experts (MoE) is a neural architecture that partitions input processing among multiple expert networks through task-dependent, learnable gating functions. In contrast to shared-gate or hard-choice MoEs, the multi-gate approach assigns each task or sub-task its own gate, allowing for soft, input-dependent weighting over a pool of experts. This paradigm underpins several state-of-the-art multi-task systems, hierarchical models, and scalable machine learning infrastructure for domains ranging from recommendation to multimodal fusion.

1. Definition and Core Architectural Principles

The multi-gate soft MoE framework entails the following essential components:

• Expert networks: A set of parallel sub-models (“experts”), often realized as MLPs, convolutional blocks, or other parametric functions. Each expert processes input features independently.
• Gating networks: For every output task or category, a separate small network (gate) computes a vector of soft assignment weights (typically via softmax or sigmoid) over experts, based on either the input or a shared representation.
• Soft combination: For each task, the output is a weighted sum of expert outputs, the weights being the output of the corresponding gate.

Let $z$ denote an embedding of the input $x$, $\phi_k(\cdot)$ expert networks, and $g^t(\cdot)$ the gating MLP for task $t$:

$\psi^t(x) = \sum_{k=1}^K g^t_k(z) \phi_k(z),$

where $$g^t_k(z) = \mathrm{Softmax}([V_k^t z + b_k^t]_{k=1}^K)$$. This structure generalizes to hierarchical and deeper (multi-layer) settings and supports complex routing via gates for domain, task, or modality (Huang et al., 2023, Wang et al., 2024, Nguyen et al., 2024).

2. Hierarchical and Multi-Level MoE Architectures

Hierarchical expansions yield architectures such as HoME (Wang et al., 2024) and HMoE (Nguyen et al., 2024), wherein experts and gates are organized into multiple levels:

• Layered expert organization: Experts are grouped into meta-experts (coarse categories) and task-experts (fine-grained), e.g., “global shared,” “category-shared,” and “task-specific” experts.
• Multi-stage gating: At each layer, gating functions select among candidate experts conditioned on meta-inputs (e.g., “interaction” vs. “watching-time” tasks receive different meta-gatings in HoME).
• Output fusion: The results from lower-level experts are recursively aggregated via higher-level gates, culminating in the task-specific prediction.

Table 1 presents canonical groupings from (Wang et al., 2024):

Level	Expert Group	Gate Input
Meta	shared_meta	Raw input
Meta	inter_meta/watch_meta	Raw input
Task	task-specific	[z^shared_meta; z^cat]

This hierarchy mitigates expert collapse and promotes regularization across tasks and sub-tasks.

3. Gating Mechanisms: Softmax, Laplace, and Beyond

The gating function profoundly influences MoE behavior and convergence rates. The default is a softmax over expert logits:

$$g_k(x) = \frac{\exp(\gamma_k^\top x + \gamma_{0,k})}{\sum_{j=1}^K \exp(\gamma_j^\top x + \gamma_{0,j})}$$

Alternative gating (notably Laplace gating), as analyzed in (Nguyen et al., 2024), utilizes

$$g_k(x) = \frac{\exp(-|f_k(x) - \mu| / b)}{\sum_j \exp(-|f_j(x) - \mu| / b)}$$

Replacing softmax with Laplace at one or both levels of hierarchical MoE provably breaks undesirable parameter interactions, thereby accelerating expert convergence and enhancing specialization. Laplace–Laplace (“LL”) gating at both levels yields an expert estimation rate of $\tilde O(n^{-1/4})$ even in heavily over-specified regimes, whereas softmax–softmax (“SS”) gating rates degrade polynomially with expert over-specification (Nguyen et al., 2024).

4. Remedies for Expert Collapse, Degradation, and Underfitting

Empirical deployments reveal recurrent pathologies:

• Expert collapse: Certain experts dominate gate allocations, starving others of gradient updates. Remedies include per-expert normalization (e.g., BatchNorm), replacement of ReLU by Swish activation for nonzero gradient propagation, and explicit load-balancing terms in the gate loss (Wang et al., 2024, Nguyen et al., 2024).
• Expert degradation: Shared experts may degenerate into task-specific roles. Inductive bias via hierarchical masks (i.e., meta-grouped experts and gates) prevents monopolization.
• Expert underfitting: Specific experts for sparse tasks may be ignored in favor of shared experts. Techniques include Fea-Gate privatization with LoRA-style input masking and self-gating connections (residual stacks of gated experts) to sustain gradients (Wang et al., 2024).

Table 2 (HoME ablation (Wang et al., 2024)) quantifies contributions:

Component	ΔGAUC (avg)
Feature-gate	+0.10–0.30
Self-gate	+0.10–0.30
Category mask	+0.10–0.30

5. Training, Optimization, and Theoretical Guarantees

Multi-gate soft MoE is trained with a composite loss, typically summing per-task (classification, regression) losses:

$\mathcal{L} = \sum_{t} w_t \mathcal{L}_t$

Adaptive weighting (as in GradNorm/TGB (Huang et al., 2023)) ensures balanced gradient flows:

• Compute per-task gradient norms $G_W^{(t)}$, moving $w_t$ to equalize convergence rates.
• Joint optimization of experts, gate MLPs, and, where present, feature-gates and auxiliary modules.
• Use of per-expert batch-normalization and non-saturating activation (e.g., Swish) stabilizes expert utilization and gradient propagation (Wang et al., 2024).

Theoretical advances (Nguyen et al., 2023, Nguyen et al., 2024, Liao et al., 8 Oct 2025) establish that, for soft-gated MoEs, density estimation converges at near-optimal parametric rates $\tilde O(n^{-1/2})$, but expert parameter estimation is bottlenecked by gating–expert interaction PDEs in the case of softmax and expert collapse. Modified gating (e.g., input transforms in the gate; Laplace gating) can restore independence, thus guaranteeing polynomial convergence in all regimes and robust feature learning in over-parameterized, multi-gate settings.

6. Empirical Results and Applications

Empirical validation across domains demonstrates the strengths of multi-gate soft MoEs:

• Short-video recommendation (Kuaishou): HoME achieves a global GAUC improvement ($+0.0062$ over MMoE baseline) and substantially higher online play-time per user (up to $+1.283\%$) (Wang et al., 2024).
• Industrial soft sensors: BMoE with multi-gate and GradNorm increases $R^2$ on key variables by 0.04–0.05 over one-gate MoE (Huang et al., 2023).
• Multimodal/vision: LL-gated HMoE improves AUROC and $F_1$ in MIMIC-IV tasks by up to $+2.5$ points over flat MoE; in computer vision, up to $+1.4\%$ top-1 on CIFAR-10 (Nguyen et al., 2024).
• Training dynamics: Soft-gated MoEs provably recover all teacher experts when over-parameterized and pruned, with phase transition in feature alignment during learning (Liao et al., 8 Oct 2025).

7. Best Practices and Design Guidelines

Based on accumulated theory and case studies:

• Employ separate gates (“multi-gate”) per task/category for all but trivial settings.
• Use Laplace gating (or softmax with transformed input) in deep/hierarchical MoEs to avoid expert-interaction slowdowns.
• Include per-expert normalization, non-zero-gradient activation, and feature/self gating to prevent expert collapse and under-utilization.
• Monitor and, if necessary, regularize gate utilization (load-balancing) to ensure active participation of all experts.
• In multi-task and sparse-label regimes, exploit category masks and input privatization to mitigate negative transfer and underfitting.

These practices underpin robust, scalable, and interpretable deployment of multi-gate soft MoEs in production and high-performance research systems (Huang et al., 2023, Wang et al., 2024, Nguyen et al., 2024, Nguyen et al., 2023, Liao et al., 8 Oct 2025).