SIGMA-MoE: Sparse Mixture-of-Experts Models

Updated 5 July 2026

SIGMA-MoE is a family of sparse Mixture-of-Experts language models that employ selective expert activation via sigmoid gating and load-balancing to optimize resource efficiency.
Variants like Sigma-MoE-Tiny and the 200B Sigma-MoE demonstrate extreme sparsity with high total parameter counts but low activated compute, excelling in tasks like QA, math, and code.
Routing-aware training, including progressive sparsification and careful load balancing, is key to maintaining stability and specialization in large-scale distributed MoE systems.

SIGMA-MoE denotes a family of sparse Mixture-of-Experts language-model designs centered on high total parameter capacity, low active compute, and routing-aware training. In current arXiv usage, the term is most directly associated with three closely related strands: the architectural proposal of $\sigma$ -MoE for efficient Transformers (Csordás et al., 2023), the open-source extreme-sparsity model Sigma-MoE-Tiny (Hu et al., 18 Dec 2025), and the 200B Sigma-MoE model used to validate the SIGMA training stack on early-life AI accelerators (Qu et al., 15 Dec 2025). Taken together, these works suggest an evolving design space rather than a single canonical model.

1. Scope and nomenclature

The literature uses the name across related but non-identical contexts. The common denominator is a sparse MoE LLM in which only a small subset of experts is activated for each token, but the concrete routing policy, scale, and systems assumptions differ.

Variant	Defining characteristics	Source
$\sigma$ -MoE	sigmoid gating, global entropy regularization, expert dropout	(Csordás et al., 2023)
Sigma-MoE-Tiny	~20B total, ~0.5B activated, up to 96 experts per layer, top-1 routing	(Hu et al., 18 Dec 2025)
Sigma-MoE	200B total, 20B activated, 96 experts per layer, 8 activated per token	(Qu et al., 15 Dec 2025)

The 2023 $\sigma$ -MoE paper frames MoE and product-key memory as instances of a general framework for approximating two-layer feedforward networks, and argues for parameter-equal rather than compute-equal comparison when evaluating LLMs (Csordás et al., 2023). Sigma-MoE-Tiny then operationalizes an extreme-sparsity regime in an open-source decoder-only Transformer, while the later SIGMA training-stack work uses a much larger Sigma-MoE as a systems-scale stress test on 2,048 accelerators (Hu et al., 18 Dec 2025, Qu et al., 15 Dec 2025).

A persistent source of confusion is the overload of the acronym “SIGMA” on arXiv. “SIGMA: Refining LLM Reasoning via Sibling-Guided Monte Carlo Augmentation” explicitly states that it does not describe any “SIGMA‑MOE” or mixture-of-experts variant (Ren et al., 6 Jun 2025), and “SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization” likewise states that there is no Mixture-of-Experts component and that “SIGMA‑MOE” is not an architectural variant in that work (Zhuang et al., 27 May 2026). This suggests that SIGMA-MoE should be treated as a specific MoE lineage, not as a generic expansion of the acronym.

2. Architectural formulation

At the architectural level, SIGMA-MoE belongs to the standard sparse MoE family in which the Transformer feed-forward network is replaced by a mixture of expert FFNs. In the generic form described in the efficient-Transformer formulation, an MoE layer approximates the FFN as

$\hat{h}(x) = \sum_{e=1}^{N_E} g_e(x)\, f_e(x),$

where $f_e(x)$ is expert $e$ ’s FFN and $g(x)$ is a sparse gating vector (Csordás et al., 2023).

The distinctive contribution of $\sigma$ -MoE is its non-competitive gating. Rather than the softmax-normalized expert competition typical of Switch-style routing, it uses a sigmoid-based top- $k$ gate: $\tilde{g}_e(x) = \sigma(s_e(x)), \qquad g_e(x) = \begin{cases} \tilde{g}_e(x), & e \in \mathcal{I}_k(s(x)),\ 0, & \text{otherwise}. \end{cases}$ This is motivated by the observation that top- $\sigma$ 0 approximations of dense FFNs perform surprisingly well and do not require competitive normalization across hidden units (Csordás et al., 2023). The same paper couples this with global entropy regularization, expert dropout, and row-norm-balanced initialization of the selection matrix $\sigma$ 1, all aimed at preventing early expert collapse and improving utilization (Csordás et al., 2023).

A more classical probabilistic formulation appears in recent work on softmax-gated multinomial-logistic MoE, where the conditional distribution is

$\sigma$ 2

That work derives a batch MM algorithm with explicit quadratic minorization, coordinate-wise closed-form updates, monotone ascent of the objective, and global convergence to a stationary point in the standard MM sense (Tran et al., 8 Feb 2026). Although this is a statistical MoE rather than a large neural LLM, it provides a formal account of the same gate-plus-expert decomposition that underlies SIGMA-MoE-style systems.

The practical implication is that SIGMA-MoE combines two distinct but compatible ideas. One is sparse activation for resource efficiency; the other is explicit control of routing behavior, whether through sigmoid gating, load-balancing losses, or systems-aware locality constraints. This suggests that SIGMA-MoE is best understood as a routing-centric MoE program rather than merely a parameter-count scaling exercise.

3. Extreme sparsity in Sigma-MoE-Tiny

Sigma-MoE-Tiny is an open-source MoE LLM designed to push MoE sparsity to an extreme while retaining strong capability (Hu et al., 18 Dec 2025). Its core architecture is a decoder-only Transformer with MoE FFN layers in all transformer blocks. The model has $\sigma$ 3 transformer blocks, hidden size 1536, Group Query Attention with 16 query heads and 4 key/value heads, RMSNorm with pre-norm, and QK-Norm applied to both queries and keys before attention logits (Hu et al., 18 Dec 2025).

Each MoE layer contains up to 96 experts, each expert is a two-layer FFN with SwiGLU activation, and only 1 expert is activated per token (Hu et al., 18 Dec 2025). The resulting model has ~20B total parameters, ~0.5B activated per token, and a total-to-activated ratio of 40:1, described as the highest among open-source MoE LLMs at the time of writing (Hu et al., 18 Dec 2025). The paper explicitly frames this as “fine-grained expert segmentation”: many small experts rather than a few large ones, following the design trend of DeepSeek-V3 and Qwen3 (Hu et al., 18 Dec 2025).

The central technical obstacle is expert load balancing under extreme sparsity. Sigma-MoE-Tiny uses the conventional auxiliary load-balancing loss

$\sigma$ 4

with global-batch statistics, where $\sigma$ 5 is the fraction of tokens routed to expert $\sigma$ 6 and $\sigma$ 7 is the average gating probability assigned to expert $\sigma$ 8 (Hu et al., 18 Dec 2025). The paper shows that this loss becomes ineffective in the lower layers when there are 96 experts and top-1 routing: $\sigma$ 9 becomes nearly uniform while $\sigma$ 0 remains highly imbalanced, yet LBL is minimized in both cases (Hu et al., 18 Dec 2025). In layer 0, the min-loaded expert has relative deviation from uniform near $\sigma$ 1, while the max-loaded expert gets ~3× more tokens than uniform (Hu et al., 18 Dec 2025).

The main remedy is progressive sparsification. During the first 90% of training, the first 8 layers use a denser schedule—8, 8, 6, 6, 4, 4, 2, 2 activated experts respectively—while layers 8–55 already use top-1 routing (Hu et al., 18 Dec 2025). In the final 10% of training, all layers switch to 1 expert, i.e. the target sparsity (Hu et al., 18 Dec 2025). This improves load balance and stability early, while preserving the deployment-time efficiency of full top-1 routing. After 200B continued tokens, maintaining the initial sparsity gives 63.69% MMLU and converting to target sparsity gives 63.53% MMLU, so a 0.15B activated-parameter reduction yields almost no loss in performance (Hu et al., 18 Dec 2025).

The model also investigates a Top-1 LBL variant,

$\sigma$ 2

with a soft routing approximation, but reports that although Top-1 LBL significantly improves load balancing, models trained with it perform worse on MMLU than models with conventional LBL (Hu et al., 18 Dec 2025). This establishes a recurring SIGMA-MoE theme: perfect or near-perfect balancing can harm specialization.

Sigma-MoE-Tiny is pretrained on a mixture of public and proprietary data, including subsets of Nemotron-CC, deduplicated DataComp-LM, deduplicated FineWeb-Edu, and proprietary synthetic data (Hu et al., 18 Dec 2025). Post-training extends context length from 4K to 128K, increases RoPE base frequency from 10,000 to 1,000,000, and uses a four-stage curriculum with Long-CoT and Short-CoT data (Hu et al., 18 Dec 2025). The domain mixture in supervised fine-tuning is math : code : science : others $\sigma$ 3 (Hu et al., 18 Dec 2025).

4. Sigma-MoE in the SIGMA training stack

The large Sigma-MoE used in the SIGMA stack is a different artifact from Sigma-MoE-Tiny: it is a 200B Mixture-of-Experts LLM trained with Lucia Training Framework on top of Lucia Training Platform, on a 2,048-accelerator cluster (Qu et al., 15 Dec 2025). Its purpose is partly algorithmic and partly infrastructural. The model validates large-scale MoE training on early-life AI accelerators, where the main challenges are reliability, stability, and efficiency (Qu et al., 15 Dec 2025).

Architecturally, this Sigma-MoE is a decoder-only transformer with 63 transformer blocks. Each block contains a Group Query Attention layer and a Mixture-of-Experts feed-forward layer following DeepSeekMoE-style design (Qu et al., 15 Dec 2025). Each MoE layer contains 96 experts with 8 activated per token, yielding a 200-billion-parameter model with 20 activated billion parameters for each token (Qu et al., 15 Dec 2025). The paper does not spell out the exact routing formula, but it explicitly states that the only stability incident in the 75-day run was fixed by improving the MoE load-balancing loss function (Qu et al., 15 Dec 2025).

Training this model required a systems stack tailored to early-life hardware. Lucia Training Platform handles proactive node healthy validation, telemetry collection, anomaly detection, automated recovery, and node remediation; Lucia Training Framework, built on Megatron-LM, handles numerical validation, parallelism configuration, noise detection, and model-level stability analysis (Qu et al., 15 Dec 2025). Over five months, LTP achieved 94.45% effective cluster accelerator utilization, reduced average job recovery time from 2.52 hours to 0.16 hours, and increased the automation recovery ratio for job-hardware failures from 43.47% to 97.83% (Qu et al., 15 Dec 2025).

The optimization of Sigma-MoE on this stack emphasized both numerical correctness and throughput. Before large-scale training, the framework ran a numerical test pipeline comparing module outputs, gradients, and parameter updates on early-life hardware versus mature stacks; similarities below 0.99 were flagged as abnormal (Qu et al., 15 Dec 2025). This process uncovered a gradient bug in flash-attention’s vendor backend, diverged results in BLAS GEMM, grouped GEMM errors when expert parallelism was enabled, and accumulated gradient error in attention k-layernorm (Qu et al., 15 Dec 2025).

For model stability, the authors identify a diagnostic signal specific to large MoE training: in successfully trained large MoE models, average MoE parameter norm increases with depth, whereas a crashed model exhibits a “valley” in the layer-wise MoE parameter norm (Qu et al., 15 Dec 2025). In Sigma-MoE, a flawed MoE load-balancing loss began to manifest at 6K steps; traditional detection via loss spikes would have caught the collapse around 30K steps, but the parameter-norm detector flagged it at 6K, allowing the run to be stopped and resumed from a healthy checkpoint after fixing the load-balancing loss (Qu et al., 15 Dec 2025).

The final efficiency path raised MFU from 9.86% to 21.08% through fused attention with vendor-specific backend, global batch size tuning from 16M to 37M tokens, fused Bias-SwiGLU and grouped MLP GEMM, more balanced PP stages, fused MoE token permute/unpermute, VPP=2, and selective MoE layer recomputation (Qu et al., 15 Dec 2025). Removing two major stragglers—lazy OS page cache invalidation and Python garbage collection—improved average/peak throughput from 85.46% to 98.11% and reduced throughput standard deviation from 22.80 to 1.98 (Qu et al., 15 Dec 2025).

5. Empirical behavior and comparative performance

Sigma-MoE-Tiny and the larger Sigma-MoE occupy different operating points, but both are evaluated as high-capacity sparse LMs.

For Sigma-MoE-Tiny-Base, the reported pre-trained benchmarks include MMLU 64.81, BBH 63.23, ARC-C 80.29, WinoGrande 76.09, GSM8K 71.65, MATH 36.88, HumanEval 42.07, and MBPP 47.0 (Hu et al., 18 Dec 2025). The paper concludes that, with only 0.5B active parameters, the base model outperforms or matches dense models in the 0.6–4B range and an MoE with 2.4B activated parameters, especially on math and reasoning (Hu et al., 18 Dec 2025). After post-training, Sigma-MoE-Tiny reports MMLU-Redux 79.8, MMLU-Pro 63.7, GPQA-Diamond 46.4, MATH-500 94.6, AIME’24 65.4, AIME’25 48.8, HumanEval 79.9, and LiveCodeBench v6 42.2 (Hu et al., 18 Dec 2025). On GPQA-Diamond, it is described as achieving accuracy comparable to dense 7–10B models despite 0.5B activated parameters (Hu et al., 18 Dec 2025).

The 200B Sigma-MoE-Base is evaluated against DeepSeek V2-Base, Qwen2.5-72B-Base, and LLaMA-3.1-405B-Base. Reported results include BBH 82.3, MMLU 80.5, MMLU-Pro 59.8, GSM8K 84.1, MATH 54.6, and HumanEval 57.9 (Qu et al., 15 Dec 2025). The paper emphasizes that Sigma-MoE-Base achieves >80% on MMLU, 54.6% on MATH, 84.1% on GSM8K, and 57.9% on HumanEval, while using 20B activated parameters per token and being trained on 9T tokens (Qu et al., 15 Dec 2025). On MMLU-Pro and HumanEval, it exceeds all three cited baselines (Qu et al., 15 Dec 2025).

The efficient-Transformer $\sigma$ 4-MoE results add a complementary perspective. Under parameter-equal comparisons rather than compute-equal ones, $\sigma$ 5-MoE is reported as competitive with dense Transformer-XL on WikiText-103 and enwiki8 at two different scales, while being much more resource efficient (Csordás et al., 2023). This is important because much prior MoE literature claimed gains mainly by increasing total parameter count at fixed FLOPs; the parameter-equal setting isolates architectural effects.

Two objective tensions recur across these works. First, sparsity and expert specialization are beneficial, but extreme sparsity amplifies routing pathologies in lower layers (Hu et al., 18 Dec 2025). Second, more aggressive balancing is not unconditionally better: Top-1 LBL improves load balance but hurts MMLU in Sigma-MoE-Tiny (Hu et al., 18 Dec 2025). A plausible implication is that SIGMA-MoE is optimizing not for uniform expert use in the abstract, but for a controlled imbalance that preserves specialization without collapse.

Several neighboring works help situate SIGMA-MoE technically. LocMoE, developed in the context of PanGu- $\sigma$ 6, targets the distributed-training bottlenecks of sparse LLMs by combining load balance and locality in routing, reducing training time per epoch by 12.68% to 22.24% compared to classical routers such as hash router and switch router, without impacting the model accuracy (Li et al., 2024). It also derives a lower bound on expert capacity from the maximal angular deviation between the gating weights of the experts and the assigned tokens (Li et al., 2024). This reinforces a systems-level point already visible in the SIGMA stack: at scale, MoE efficiency depends as much on dispatch, communication, and expert placement as on nominal sparsity.

On the statistical side, recent work on softmax-gated multinomial-logistic MoE derives finite-sample rates for conditional density estimation and parameter recovery, and introduces a sweep-free selector of the number of experts based on dendrograms of mixing measures (Tran et al., 8 Feb 2026). That model is not a LLM, but it provides rigorous guarantees for stable maximum-likelihood training and principled model selection in the same gate-plus-expert formalism. This suggests that one unresolved frontier for neural SIGMA-MoE systems is the transfer of such guarantees to large, deep, minibatch-trained architectures.

The limitations documented in Sigma-MoE-Tiny are specific and consequential. The paper explicitly notes task scope restricted to QA, math, and code with clear, verifiable answers; non-learned relations in the sense that routing is configured through heuristic scoring and a fixed cosine-similarity evaluator rather than end-to-end learned signed graphs; scalability costs in agent-pool size for the associated coordination framework; and the absence of end-to-end MoE-style training in the signed-graph MAS analogy (Hu et al., 18 Dec 2025, He et al., 19 May 2026). For the MoE LLM itself, the open problems are better balancing without harming performance, improved lower-layer routing, and extension from 20B total parameters to much larger systems (Hu et al., 18 Dec 2025).

The SIGMA stack paper, by contrast, foregrounds hardware immaturity: frequent system disruptions, undefined failure modes, numerical bugs in low-level kernels, and unpredictable local noise (Qu et al., 15 Dec 2025). Its conclusion is therefore partly infrastructural. Sigma-MoE is not only a model family but also a benchmark for whether a platform can sustain long, stable sparse training runs. This suggests that the historical significance of Sigma-MoE may lie as much in co-design—routing, kernels, parallelism, telemetry, and recovery—as in any single gating formula.

In summary, SIGMA-MoE names a technically coherent but internally diverse MoE lineage. The 2023 $\sigma$ 7-MoE work contributes the non-competitive gating and parameter-equal evaluation perspective (Csordás et al., 2023); Sigma-MoE-Tiny demonstrates that extreme sparsity with up to 96 experts per layer and only 1 activated expert per token can be trained stably through progressive sparsification (Hu et al., 18 Dec 2025); and the 200B Sigma-MoE in the SIGMA stack shows that a 96-expert, top-8 model can be trained for 75 days on 2,048 early-life accelerators with 21.08% MFU and only one detected stability incident (Qu et al., 15 Dec 2025). Across these variants, the defining theme is not merely sparse activation, but the joint optimization of expert selection, load balance, numerical stability, and distributed systems efficiency.