Entropy-Regularized MoE Fusion

• The paper introduces an entropy-regularized Bayesian formulation for MoE fusion, optimizing expert routing via variational inference and top-k sparsity.
• It presents methods to enforce load balancing and prevent expert collapse through auxiliary losses and explicit entropy controls.
• Furthermore, the work connects information theory with routing ambiguity, offering a geometric perspective and empirical benchmarks in perplexity and expert utilization.

Entropy-regularized Mixture-of-Experts (MoE) fusion refers to a family of routing and expert selection mechanisms within sparse large-scale neural architectures, where the assignment of inputs (“tokens”) to experts is optimized under explicit entropy regularization. This approach is grounded in variational Bayesian inference, information theory, and, more recently, geometric and manifold-based probabilistic constructions. Entropy regularization—either explicit or implicit—shapes the trade-off between expert utilization balance (load balancing), output diversity, and routing sparsity, supplying a rigorous foundation for heuristic techniques such as Top-$k$ selection and auxiliary losses.

1. Variational Bayesian Formulation and Entropy in MoE Routing

The latent-variable model of Mixture-of-Experts introduces a discrete variable $z \in \{1, \ldots, E\}$ denoting expert assignment, with the output modeled by

$p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$

where $p(z)$ is usually uniform and $p(y \mid x, z)$ denotes the expert-conditional likelihood such as $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$ for some loss $\ell$ (Su et al., 7 Jan 2026).

Computing $p(y|x)$ is tractable only for modest $E$; in large-scale MoEs, practitioners employ a variational gating distribution $q(z|x)$, yielding the standard evidence lower bound (ELBO):

$z \in \{1, \ldots, E\}$0

where for uniform $z \in \{1, \ldots, E\}$1 and expanding $z \in \{1, \ldots, E\}$2,

$z \in \{1, \ldots, E\}$3

The entropy term $z \in \{1, \ldots, E\}$4 penalizes deterministic routing, encouraging high-entropy, uncertainty-aware, and more evenly distributed expert usage (Su et al., 7 Jan 2026).

2. Sparse Posterior Approximation: Top-$z \in \{1, \ldots, E\}$5 Routing and Load Balancing

MoE efficiency requires extreme sparsity; only $z \in \{1, \ldots, E\}$6 experts are permitted to process each input. Formalizing this via the $z \in \{1, \ldots, E\}$7-sparse simplex:

$z \in \{1, \ldots, E\}$8

the constrained optimization is

$z \in \{1, \ldots, E\}$9

A pivotal theorem states that if $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$0 yields gating logits $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$1, the entropy-regularized, $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$2-sparse solution is achieved by retaining only the $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$3 largest $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$4 and renormalizing—precisely the algorithmic structure of Top-$p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$5 routing. For a single forward pass, this involves Top-$p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$6 gating logits followed by softmax normalization on the activated subset (Su et al., 7 Jan 2026).

To enforce prior matching and resist expert collapse, an auxiliary load balancing loss is added:

$p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$7

where $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$8 is the assignment ratio, $p(y \mid x) = \sum_{z=1}^E p(y \mid x, z)\, p(z),$9 the average gating, and $p(z)$0 is the aggregated posterior. Minimizing $p(z)$1 maximizes the Rényi collision entropy $p(z)$2, pushing $p(z)$3 toward uniform distribution (Su et al., 7 Jan 2026).

3. Information-Theoretic Interpretation: Channel Capacity and Routing Ambiguity

Viewing the MoE router as a discrete channel $p(z)$4, mutual information $p(z)$5 quantifies the transmitted expert assignment information. Top-$p(z)$6 routing restricts conditional entropy:

$p(z)$7

thus lowering $p(z)$8, reducing noise, and enforcing sparsity-induced channel regularity (Su et al., 7 Jan 2026).

Input-dependent load balancing pushes marginal $p(z)$9, maximizing channel capacity. These mechanisms jointly maximize a lower bound on mutual information:

$p(y \mid x, z)$0

linking Top-$p(y \mid x, z)$1 and load balancing losses to information maximization, undergirding their effectiveness and necessity.

4. Geometric and Algorithmic Complexity: Coherence Barrier and Orthogonality

For input $p(y \mid x, z)$2, seeking the optimal $p(y \mid x, z)$3-sparse expert subset is equivalent to

$p(y \mid x, z)$4

where $p(y \mid x, z)$5 is the expert output matrix. This sparse subset selection is NP-hard (Su et al., 7 Jan 2026). Greedy Top-$p(y \mid x, z)$6 selection, commonly applied, can fail when expert representations (columns of $p(y \mid x, z)$7) have high mutual coherence,

$p(y \mid x, z)$8

A “Coherence Barrier” theorem states that if $p(y \mid x, z)$9, greedy selection is globally optimal; otherwise, routing ambiguity and suboptimality emerge. For perfectly orthogonal experts, greedy Top-$p(y|x,z) \propto \exp(-\ell(E_z(x), y))$0 recovers the global optimum in polynomial time, as the Gram matrix on any subset reduces to the identity (Su et al., 7 Jan 2026). Enforcing orthogonality via architectural or regularizer choices transforms the otherwise intractable routing into a “sort and select” operation.

5. Grassmannian and Concentration-Parametric Entropy Control

Grassmannian Mixture-of-Experts (GrMoE) introduces an alternative entropy-regularized fusion approach based on the Matrix Bingham distribution defined on the Grassmannian manifold $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$1. Each expert is parameterized by a concentration matrix $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$2 (or scalar $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$3) and a subspace projector $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$4. The routing probability is defined as

$p(y|x,z) \propto \exp(-\ell(E_z(x), y))$5

with global concentration scaling parameter $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$6 introduced for entropy/spasity modulation:

$p(y|x,z) \propto \exp(-\ell(E_z(x), y))$7

Explicit theoretical bounds are established connecting the concentration spectrum $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$8 to routing entropy $p(y|x,z) \propto \exp(-\ell(E_z(x), y))$9, top-$\ell$0 mass, and the probability of expert collapse, e.g.,

$\ell$1

and

$\ell$2

where $\ell$3, $\ell$4, and $\ell$5 are affinity statistics (Shihab et al., 19 Feb 2026).

The GrMoE mechanism allows continuous, monotonic control over routing entropy and effective expert sparsity by tuning $\ell$6 or the expert-specific $\ell$7, as opposed to discrete Top-$\ell$8 selection. Amortized variational inference further enables dynamic, uncertainty-aware gating (Shihab et al., 19 Feb 2026).

6. Empirical Performance and Applications

Empirical studies confirm that explicit entropy-regularized MoE fusion—either via Top-$\ell$9/auxiliary-loss approaches or Grassmannian gating—yields substantial improvements in routing accuracy, expert load balance, and collapse resistance compared to traditional methods. For instance, GrMoE demonstrates 0% routing collapse across all seeds at scales of 8, 16, or 32 experts, with LLM perplexity on par or improved relative to switch routing or softmax top-$p(y|x)$0 baselines (Shihab et al., 19 Feb 2026). Entropy regularization leads to interpretable expert concentration profiles and supports post-hoc sparsity tuning at inference without retraining.

Method	Perplexity (PPL)$p(y|x)$1	Collapse$p(y|x)$2	Routing Entropy $p(y|x)$3
Softmax Top-2	18.7	40%	1.12
GrMoE+Amort. (350M)	18.1	0%	1.29
GrMoE+Amort. (1.3B)	13.8	0%	1.42
GrMoE+Amort. (2.7B)	11.5	0%	1.38

In practical deployments, a single GrMoE model can be trained (e.g., $p(y|x)$4), then the $p(y|x)$5 sparsity dial used at inference to interpolate throughput and sparsity metrics. Expert-specific concentration parameters $p(y|x)$6 impart interpretability, reflecting specialization and relative sharpness across learned experts (Shihab et al., 19 Feb 2026).

7. Theoretical and Practical Implications

Entropy-regularized MoE fusion mechanisms provide a theoretically rigorous foundation for sparse expert selection in massive LLMs. The unified variational and information-theoretic framework demonstrates that canonical Top-$p(y|x)$7 routing and auxiliary load-balancing are not heuristics but the exact $p(y|x)$8-sparse entropy-regularized solution to the Bayesian posterior approximation problem under a uniform prior (Su et al., 7 Jan 2026). For generic (coherent) dictionaries, routing remains NP-hard, but geometric orthogonality regularization reduces the complexity to provably optimal greedy selection.

Recent advances employing Grassmannian geometry and concentration-parametric control inaugurate a new regime of interpretable, analytically quantifiable entropy-sparsity trade-off, obviating the need for ad-hoc balances or temperature annealing. This establishes connections between geometric structure, statistical mechanics, and practical token-expert fusion in large-scale distributed LLMs, with demonstrated empirical reliability and theoretical tractability (Shihab et al., 19 Feb 2026).