Multi-gate Soft MoE Architecture

• The paper demonstrates a neural architecture that employs continuous, soft gating to dynamically route information across multiple expert networks.
• It leverages shared embeddings and hierarchical gating to scale deep networks while mitigating issues like expert collapse and underfitting.
• Empirical results reveal improved accuracy and reduced computational cost, validating its effectiveness in convolutional and multi-task settings.

A multi-gate soft Mixture-of-Experts (MoE) system is a neural architecture in which multiple gating networks coordinate how representations are selectively routed through several expert subnetworks, with decisions made in a continuous, differentiable manner. Unlike hard MoE approaches that use discrete routing or top-k selection, multi-gate soft MoE maintains soft, nonnegative gate vectors, supporting both expressivity and efficient training via standard gradient descent. Modern variants employ sophisticated hierarchies, gating mechanisms, and auxiliary strategies to address scaling challenges in deep learning and multi-task domains.

1. Architectural Principles of Multi-Gate Soft MoE

In canonical multi-gate soft MoE designs, layers of experts are dynamically activated through multiple gate heads, each modulated by shared or private embeddings of the input. In the DeepMoE architecture, every convolutional layer in a standard backbone (e.g., VGG, ResNet) is replaced with an MoE-convolutional layer, treating each input channel as an expert. All gating decisions are driven by a shared, shallow embedding network $M(x)$ that produces a low-dimensional latent vector $e \in \mathbb{R}^d$, which feeds into layer-specific gates $G^{(l)}(e)$.

For each layer $l$, the gate $g^{(l)} \in \mathbb{R}^{C_{in}^l}$ is obtained via:

$g^{(l)} = \mathrm{ReLU}(W_g^{(l)} e)$

and the corresponding output is the soft sum:

$$z^{(l)} = \sum_{i=1}^{C_{in}^l} g^{(l)}_i \cdot (K^{(l)}_i * x^{(l)}_i)$$

where $K^{(l)}_i$ denotes the filter bank for channel $i$ (Wang et al., 2018).

Similarly, the Balanced Mixture-of-Experts (BMoE) organizes a “shared bottom” embedding $f(x)$, $K$ shared experts $\phi_k(z)$, and $N$ task-specific gating networks $g_i(z)$:

$g_i^k(z) = \mathrm{softmax}(a_i(z))_k$

Soft mixing occurs via:

$\psi_i(z) = \sum_{k=1}^K g_i^k(z) \cdot \phi_k(z)$

Task predictions are computed as $\hat{y}_i = \Phi_i(\psi_i(z))$ (Huang et al., 2023).

Hierarchical variants such as HoME structure experts and gates into meta-categories (global, category-shared, task-specific), using feature-level gates and residual “self-gates,” further enhancing routing flexibility (Wang et al., 10 Aug 2024).

2. Mathematical Formulation of Soft Gating

Multi-gate soft MoE systems employ dense, nonlinear mappings from input features to gate values for each expert. The general gating process includes:

• Embedding network: $e = M(x)$ (for DeepMoE), or $z = f(x; \theta)$ (for MMoE/BMoE).
• Layer/task gates: $\mathrm{ReLU}$ or $\mathrm{softmax}$ transformations of gated linear projections.
• Expert mixing: Outputs are weighted sums of expert activations, scaled by gate outputs.

Sample formulation for gating in MMoE (Huang et al., 2023): $$\begin{align*} a_i(z) &= W^{g_i} z + b^{g_i} \in \mathbb{R}^K \ g_i^k(z) &= \frac{\exp(a_i^k(z))}{\sum_{\ell=1}^K \exp(a_i^\ell(z))} \end{align*}$$ A plausible implication is that employing $\mathrm{softmax}$-based gates enforces probability simplex constraints, facilitating equitable routing across experts.

3. Expressivity Via Layer-wise Multi-head Routing

Stacking $L$ layers of soft gating exponentially grows the number of distinct expert activation paths. In DeepMoE, the joint gating mask $(g^{(1)},\ldots,g^{(L)})$ over $L$ layers enables approximately $\prod_{l=1}^L \binom{C_{in}^l}{n_l}$ soft paths, with $n_l$ as average attended experts per layer. This results in a super-exponential, data-dependent expressivity, formally matching $n^{2^L}$ functional rank for width-$n$ networks, preserving broad feature-to-label mappings (Wang et al., 2018).

4. Joint Training Objectives and Regularization

Multi-gate soft MoE architectures employ joint training objectives incorporating:

• Base loss: Standard cross-entropy or regression losses applied to gated network outputs.
• Sparsity regularization: Encourage gate sparsity (e.g., $\ell_1$ penalty on $g^{(l)}$) to reduce FLOPs and encourage competitive expert selection (Wang et al., 2018).
• Embedding/classification loss: Auxiliary losses maintaining embedding informativeness, preventing gate collapse into degenerate modes.
• Task Gradient Balancing: In BMoE, the GradNorm module dynamically adapts per-task loss weights $w_i$ to equalize backpropagated gradient magnitudes, minimizing

$$\mathcal{L}_{\text{grad}} = \sum_{i=1}^N |G_W^{(i)}(t) - \hat{G}_W^{(i)}(t)|_1$$

where $\hat{G}$ targets are proportional to relative inverse training rates $r_i(t)$ (Huang et al., 2023).

HoME introduces stability primarily through architectural hierarchy and gating stratification, without explicit regularizers, relying on normalized expert outputs and strong gradient flows (Wang et al., 10 Aug 2024).

5. Hierarchical and Task-Structured Gating Schemes

Recent advances impose hierarchical gating to mitigate expert collapse, degradation, and underfitting. In HoME, tasks are grouped into meta-categories, with meta-gates assigning weights to global and category-shared experts. Task-gates activate relevant experts for individual tasks. Feature-Gate modules privatize per-expert input features using LoRA-style low-rank blocks, while Self-Gate units maintain gradient flow via residual mixing (Wang et al., 10 Aug 2024).

• Expert collapse is monitored using zero activation rate; BatchNorm+Swish activations mitigate dead experts.
• Expert degradation (shared experts used only by a single task) is countered by explicit task grouping and occupancy ratio metrics.
• Expert underfitting (data-sparse tasks not updating specific experts) is ameliorated by input privatization and residual shortcuts.

6. Empirical Results and Implementation Guidelines

Empirical benchmarks demonstrate measurable improvements:

• DeepMoE achieves higher accuracy with lower computational cost compared to standard convolutional models (Wang et al., 2018).
• BMoE and HoME demonstrate improved multi-task losses and mitigation of the negative transfer/seesaw effect, with offline AUC/GAUC uplifts up to +0.65% and online business metric increases in large-scale deployments (Huang et al., 2023, Wang et al., 10 Aug 2024).

Recommended hyperparameter settings include:

Model	# Experts	Gate Type	Embed Dim	Activation	Optimizer	Noted Improvements
DeepMoE	Per-channel	ReLU	$d$	ReLU	SGD	Lower FLOPs, higher accuracy
BMoE/MMoE	$K=4$	Softmax	$d_e=64$	Mish	Adam/SGD	Balanced task gradients
HoME	Hierarchical	Softmax, Sigmoid	$d\approx3000$	Swish+BatchNorm	Adam	GAUC +0.57%, robust deployment

Practical architectures include expert MLPs with BatchNorm+Swish, task-gates with softmax, LoRA-style feature privatizers, and full model sizes in the 225–300M parameter range for industrial-scale ranking with manageable inference overhead (Wang et al., 10 Aug 2024).

7. Research Directions and Open Issues

Multi-gate soft MoE frameworks pose ongoing challenges, including expert specialization, equitable gradient allocation, and robustness to data sparsity. Hierarchical and privatized gating have mitigated expert collapse, degradation, and underfitting in practice, yet fine-grained balancing under strict FLOP or latency constraints remains an active area. Further research may explore adaptive hierarchies, meta-gating strategies, and integration with emerging context-dependent expert selection methods.

Collectively, multi-gate soft MoE architectures have advanced scalable, expressive neural networks, particularly in convolutional and multi-task settings, by leveraging continuous multi-head gating for dynamic routing and robust training (Wang et al., 2018, Huang et al., 2023, Wang et al., 10 Aug 2024).