MoGe: Mixture of Group Experts

• MoGe is a neural architecture variant, known as Mixture of Group Experts, that applies group-sparse routing to ensure specialized and invariant expert activation.
• It employs a differentiable group penalty on gating inputs, promoting spatial locality and achieving improved accuracy and robustness on vision and language benchmarks.
• MoGe's grouped routing strategy efficiently balances device load and scales to large models, delivering significant throughput gains with minimal computational overhead.

MoGe

MoGe denotes either a specialized mixture-of-experts neural architecture or, contextually, a specific variant: "Mixture of Group Experts." The term arises prominently in advanced deep learning, particularly in the context of expert routing for efficient scaling of Transformer models, and as an architectural solution for both vision and LLM tasks. MoGe exploits group sparsity and group-structured routing, offering distinct theoretical and practical advantages over classical MoE (Mixture-of-Experts) designs. Alternative uses of "MoGe" as an acronym or abbreviation occur in adjacent literature, but this entry is restricted to the Mixture of Group Experts framework and its formal generalizations.

1. Theoretical Foundations of MoGE

MoGE is motivated by the classical sparse representation problem, where an input $y \in \mathbb{R}^m$ is approximated over a dictionary $\mathcal{D}\in\mathbb{R}^{m \times n}$ by a sparse code $w$:

$$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$

with $k \ll n$ giving representational efficiency. Vanilla MoE architectures parallel this principle, interpreting their gating weights as sparse codes, with the gating network $G(x)$ yielding activations $z \in \mathbb{R}^n$, top-$k$ sparsified to produce $w$. However, vanilla MoE exhibits severe limitations in expert specialization and scalability: for large $n$ and small $\mathcal{D}\in\mathbb{R}^{m \times n}$0, many experts receive non-disjoint assignments and fail to diversify (Kang et al., 12 Apr 2025).

MoGE addresses this deficiency by introducing structured group sparsity on the pre-sparsified gating inputs. Specifically, $\mathcal{D}\in\mathbb{R}^{m \times n}$1 is reshaped to $\mathcal{D}\in\mathbb{R}^{m \times n}$2 and regularized by a differentiable group-sparse penalty:

$\mathcal{D}\in\mathbb{R}^{m \times n}$3

where $\mathcal{D}\in\mathbb{R}^{m \times n}$4 and $\mathcal{D}\in\mathbb{R}^{m \times n}$5 is a Gaussian low-pass kernel of width $\mathcal{D}\in\mathbb{R}^{m \times n}$6. This approach induces spatial locality in gating activations, enforces clusterwise expert selection, and generates locally invariant representations under small input perturbations (Kang et al., 12 Apr 2025).

2. Algorithmic Implementation and Training Procedures

A canonical MoGE layer operates as follows:

1. Routing and Group Penalty: Compute gating activations $\mathcal{D}\in\mathbb{R}^{m \times n}$7, reshape $\mathcal{D}\in\mathbb{R}^{m \times n}$8 to $\mathcal{D}\in\mathbb{R}^{m \times n}$9, and evaluate $w$0 using the above group-sparsity mechanism (Algorithm 1 in (Kang et al., 12 Apr 2025)).
2. Top-$w$1 Routing: Apply a top-$w$2 operator to $w$3 to produce the sparse weighting $w$4. Only the top-scoring $w$5 gating activations contribute to the output.
3. Expert Aggregation: Model output is $w$6, where $w$7 are the expert sub-networks.
4. Objective: Training loss consists of the original task loss (e.g., cross-entropy or language modeling) plus $w$8, where $w$9 is a regularization hyperparameter.
5. Backward Pass: Gradients of $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$0 propagate through the softmax and gating net, shaping the spatial grouping behavior.

During inference, $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$1 is omitted but the routing network’s group structure persists.

3. Empirical Properties and Performance

MoGE demonstrates consistent performance gains across vision and language domains. In the vision domain, ViT-MoE and SwinMoE backbones equipped with MoGE regularization outperform their vanilla MoE counterparts on CIFAR-100, Tiny-ImageNet, and ImageNet-1K by $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$2 in top-1 accuracy for matched model capacity (Kang et al., 12 Apr 2025). For language modeling (WikiText-103), perplexity drops from 84.81 to 82.08 for SMoE-small and from 33.46 to 33.35 for MomentumSMoE-medium configurations. Invariance to minor transformations, quantified via IMED distances in the $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$3 space, improves by $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$4 for common input perturbations.

Scalability is a defining feature: as $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$5 increase, MoGE penalizes overlap and redundancy more effectively than vanilla MoE, leading to robustness in both accuracy and expert load. Overhead for the regularization and group-structured routing remains minor: runtime cost is $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$6 and memory increase is under 4 MB for practical models.

4. MoGE in Large Scale Systems: Grouped Routing and Hardware Coordination

The Mixture of Grouped Experts instantiation of MoGE, especially as realized in "Pangu Pro MoE" (Tang et al., 27 May 2025), further extends the group-structured principle by enforcing fixed token-to-expert assignments within explicitly designed expert groups. Here, the $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$7 experts are divided into $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$8 groups of $$\min_{w} \tfrac12\|y - \mathcal{D}w\|_2^2 \quad \text{s.t.} \;\|w\|_0 \le k,$$9, and for each input $k \ll n$0 experts are activated, with exactly $k \ll n$1 per group.

This enforces perfect device-level load balance in distributed inference/training: each device hosts one group and processes an identical number of tokens/expert activations per forward pass. In Pangu Pro MoE, a 72B-parameter model (of which 16B are active per token) achieves 1148–1528 tokens/s/card (with speculative decoding) on Ascend hardware, substantially outperforming dense baselines of equivalent parameter class. The MoGE routing protocol ensures zero straggling across devices, optimally pipelined communication, and empirical throughput gains of up to $k \ll n$2–$k \ll n$3 versus dense LLMs (Tang et al., 27 May 2025).

5. Connections to Sparse Coding, Invariant Representation, and MoE Generalizations

The MoGE framework is rooted in bridging sparse representation theory with deep expert routing. Its topographic grouping exploits overlapping $k \ll n$4-type constraints, reminiscent of both classical group Lasso and modern invariant representation learning (Kang et al., 12 Apr 2025). Empirically, the group regularization demonstrably reduces the sensitivity of expert assignments to nuisance variations, leading to more robust and semantically consistent expert divisions.

This suggests MoGE sits as an intermediate between vanilla top-$k \ll n$5 MoE and more elaborate structured-sparse or conditional computation networks. Furthermore, the group-structuring principle generalizes naturally to alternative grouping topologies (e.g., graph-induced, sequential, or hierarchical), though as yet these have not been extensively explored.

6. Limitations, Open Problems, and Future Directions

Limitations of existing MoGE designs include the assumption of uniform group sizes and static group structure—potentially suboptimal for highly non-uniform data or large expert counts. Dynamic or adaptive expert grouping, meta-learned group structures, and co-design of expert architecture and routing topology remain open areas. Grouped expert routing, while architecturally compatible with contemporary hardware, may require specific kernel and communication pathway optimization when porting outside the Ascend NPU ecosystem (Tang et al., 27 May 2025).

Empirical evidence is currently limited to moderate-scale transformer models in vision and language domains; operation at the scale of multi-billion parameter MoE LLMs beyond Pangu Pro MoE is yet to be rigorously characterized.

7. Summary Table: Variants and Key Features

MoGE Variant	Key Principle	Application Domain	Performance Impact
Group Sparse Regularization (Kang et al., 12 Apr 2025)	Group penalty on gating input	Vision (ViT/Swin), Language Modeling	↑ Acc./PPL, ↑ invariance, ≈0 cost
Grouped Routing Architecture (Tang et al., 27 May 2025)	Fixed group-expert activation	Large MoE LLMs (Pangu Pro MoE)	↑ Throughput, perfect load balance

In conclusion, the MoGE principle—imposing group structure in expert activation—addresses both statistical (specialization, invariance) and systems-level (load balancing, hardware parallelism) bottlenecks inherent in traditional sparsely activated MoE models. It achieves these with minimal architectural change, broad applicability, and negligible compute overhead, establishing itself as a key design in scalable efficient Transformer systems (Kang et al., 12 Apr 2025, Tang et al., 27 May 2025).