Path-Constrained Mixture-of-Experts

Abstract: Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling by activating only a subset of parameters for each input. However, conventional MoE routing selects each layer's experts independently, creating N^L possible expert paths -- for N experts across L layers. This far exceeds typical training set sizes, leading to statistical inefficiency as the model may not learn meaningful structure over such a vast path space. To constrain it, we propose \pathmoe, which shares router parameters across consecutive layers. Experiments on 0.9B and 16B parameter models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary load balancing losses. Analysis reveals that tokens following the same path naturally cluster by linguistic function, with \pathmoe{} producing more concentrated groups, better cross-layer consistency, and greater robustness to routing perturbations. These results offer a new perspective for understanding MoE architectures through the lens of expert paths.

• The paper introduces PathMoE, a novel MoE routing technique that constrains expert paths through block-wise parameter sharing to enhance sample efficiency and interpretability.
• It reduces routing entropy by about 1 bit, leading to higher accuracy and smoother training curves on multiple NLP benchmarks without auxiliary load balancing losses.
• Empirical evaluations demonstrate robust path specialization and cross-layer consistency, indicating that PathMoE effectively combines flexibility with controlled expert routing.

Path-Constrained Mixture-of-Experts: An Expert Path Perspective for MoE Routing

Introduction

Sparse Mixture-of-Experts (MoE) architectures afford computational scalability by selectively activating a subset of parameters per input, decoupling effective model capacity from inference/training cost. However, conventional MoE routing schemes apply independent, per-layer expert selection, generating an astronomical $N^L$ possible "expert paths" for $N$ experts over $L$ layers. Such a combinatorial path space vastly exceeds the coverage available from practical training corpora, leading to severe statistical inefficiency (Figure 1), as the majority of expert configurations receive negligible or no learning signal.

Figure 1: Statistical inefficiency of independent routing; the effective number of expert paths dwarfs dataset size.

Recent scaling work has largely overlooked the implications of the expert path combinatorics, assuming emergent structure from optimization is sufficient for reliable training. This paper makes two central contributions:

1. It reframes MoE routing through the lens of expert paths, showing that path structure is critical for efficient learning and interpretability.
2. It introduces PathMoE, a block-wise parameter sharing approach that constrains the routable path space, offering a principled trade-off between flexibility and statistical efficiency.

PathMoE and the Spectrum of Routing Constraints

Independent routing assigns a separate router to each layer (Figure 2a), maximizing path diversity at the cost of sampling only a vanishing fraction of the expert-path space. A trivial solution is to fully share routing parameters or routing decisions across layers (Figure 2c). While this reduces the path space to a manageable $N$, it underfits complex tasks by being overly restrictive.

PathMoE introduces an intermediate, block-wise parameter sharing design (Figure 2b): routers are shared across contiguous blocks of layers (with block size $B$), enforcing local path coherence while preserving global flexibility. Router weights $\mathbf{W}_b$ are tied within block $b$, so expert selection decisions over nearby layers become correlated—naturally encouraging smoother expert paths and reducing effective routing entropy.

Figure 2: The routing constraint spectrum for MoE. (a) Independent per-layer, (b) Block-wise parameter sharing (PathMoE), (c) Fully shared.

Theoretical Justification: Reducing Routing Entropy

For independent routing, layerwise router parameters (and thus routing probabilities) are initialized independently and, thanks to architectural residuals, token representations $(\mathbf{x}_l)$ change slowly across layers. However, without parameter sharing, path assignments are weakly correlated and mutual information between consecutive decisions is nearly zero.

PathMoE injects strong inter-layer correlations—by tying router weights within a block, the mutual information $I(E_l;E_{l-1})$ is maximized, decreasing the overall routing entropy $H(\mathbf{E})$ and collapsing the effective path space by about 50%. Empirically, with $N=16,L=24$, PathMoE yields 1 bit lower entropy (21.14 vs. 22.20), resulting in greater path reuse and higher average learning signal per path.

Empirical Evaluation

PathMoE and baselines were evaluated on a suite of 8 downstream NLP tasks (ARC-Easy, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, SocialIQA, WinoGrande) and on large-scale pretraining (FineWeb-100B, DCLM-Pro). Two architectural instantiations were explored: a 0.9B parameter model (16 experts, top-4) and a 16B parameter model (64 experts, top-6).

Key empirical findings:

• Performance: PathMoE surpasses strong baselines, achieving the highest average accuracy (49.62%) and best validation perplexity on the 0.9B model, and winning 10/12 categories on the 16B model.
• No auxiliary load balancing loss required: PathMoE obviates the need for expert balancing losses entirely, maintaining balanced expert usage and stable convergence even when auxiliary objectives are removed (Figure 3).

  Figure 3: Training curves: PathB4-MoE achieves higher and smoother accuracy without auxiliary load balancing loss, unlike Indep-MoE.

• Efficiency: Throughput and memory footprint are identical or slightly improved compared to independence, as block sharing reduces router parameter count without additional compute.

Emergent Behavior: Path Specialization and Consistency

PathMoE yields interpretable, robust expert path patterns:

• Path Frequency Concentration: PathMoE routes tokens through far fewer expert paths compared to independent routing, with the most frequent paths covering a much larger token fraction (Figure 4).

  Figure 4: Cumulative token coverage: PathMoE concentrates tokens into fewer dominant expert paths.

• Linguistic Specialization: Expert paths exhibit sharp linguistic domain clustering: tokens traversing a path often belong to coherent categories (person names, punctuation, discourse markers, etc.—Figure 5), reflecting emergent specialization without direct supervision.

  Figure 5: Token specialization: Each path processes tokens tied to distinct linguistic functions (punctuation, proper nouns, discourse connectives, etc.).

• Cross-Layer Consistency: PathMoE achieves 31% higher Jaccard path consistency and longer sustained token engagement with specific experts across consecutive layers (Figure 6).

  Figure 6: PathMoE networks show superior cross-layer path consistency and longer expert reuse runs relative to independent routing.

Robustness, Specialization, and Ablation Results

Despite fostering more specialized routing, PathMoE architecture is dramatically more robust to routing perturbations (random expert index permutations): perplexity degradation is 22.5$\times$ lower than with independent routing (Figure 7), contradicting the conventional view that tighter specialization leads to fragility.

Figure 7: (a) Lower routing entropy in PathMoE. (b) PathMoE maintains high performance under extreme routing perturbations.

Block size and top-$k$ routing studies confirm the architectural design space:

• Optimal block size is task- and domain-dependent but usually small ($B=4$).
• Top-2 routing is optimal, consistent with recent SOTA MoE.

  Figure 8: Ablation analysis reveals optimal model configurations for top-$k$ and block size.

Theoretical and Practical Implications

By constraining path diversity in a principled manner, PathMoE yields:

• Improved sample efficiency: More tokens per path, denser statistics for each expert-path pair.
• Enhanced interpretability: Clustering of paths by linguistic and functional domain, directly observable through unsupervised analysis.
• Elimination of auxiliary regularization: Architectural constraint replaces the need for hyperparameter-sensitive objectives.
• Robust specialization: PathMoE breaks the trade-off between specialization and robustness, suggesting new directions for designing MoE routing protocols.

The analysis also highlights that the choice of expert path structure is an independent axis of MoE expressivity and efficiency, orthogonal to improvements from representation normalization, recurrent routing, or expert-choice schemes.

Conclusion

This work reframes the Mixture-of-Experts routing problem from the perspective of compositional expert paths, demonstrating that careful architectural constraints can shrink the effective path space by orders of magnitude, yielding increased performance, robustness, and interpretability. PathMoE's block-wise parameter sharing is both a theoretically principled and empirically validated approach, suggesting promising new directions for MoE research, including learned path predictors, structural model compression, and extensions beyond token-choice routing paradigms.