PathMoE: Trajectory-Driven Expert Routing

• PathMoE is a framework that structures Mixture-of-Experts models by enforcing shared routing policies across contiguous layers to reduce effective path entropy.
• It employs blockwise router parameter sharing to concentrate training signals, improve semantic consistency, and optimize model performance.
• The approach enables trajectory-driven expert pruning and multimodal interaction modeling for enhanced interpretability and compression in deep learning.

PathMoE refers to a family of frameworks and algorithmic strategies that structure, constrain, or analyze routing in Mixture-of-Experts (MoE) architectures through the lens of expert activation paths. These approaches span from architectural modifications for improved learning efficiency and interpretability, to frameworks for global expert pruning and multimodal interaction modeling. Across instantiations, the unifying theme is a focus on the trajectory—a token’s sequence of expert selections across layers—as a principal unit of structure and meaning within MoEs.

1. Architectural Principle: Path-Constrained Routing

Conventional sparse MoE transformers independently route each token through one or more experts per layer, leading to a combinatorially large set of possible expert trajectories, or paths: with $N$ experts and $L$ layers, the path space is $N^L$. This vastness greatly exceeds realistic training set sizes, causing statistical inefficiency: most paths are rarely or never sampled, preventing robust specialization.

The PathMoE principle, as formulated in "Path-Constrained Mixture-of-Experts" (Gu et al., 18 Mar 2026), mitigates this by partitioning $L$ layers into contiguous blocks of size $B$. Each block shares its router parameters $W_b$, forcing all layers within a block to compute their gating distributions identically: $$p^l = \mathrm{softmax}(W_b x_l), \qquad \text{for}\ l\ \text{in block }b.$$ This block-sharing reduces the path space to $N^{L/B}$, where typical $B$ is 4–8, thus concentrating training signal and increasing the frequency with which any particular path is sampled.

Independent routing

• Path entropy $H(\vec{E})$ is high ($L$022.2 bits for $L$1), yielding $L$24.8 million effective paths. Block-shared PathMoE
• Path entropy falls ($L$321.14 bits, $L$42.3 million effective paths for $L$5), improving per-path sample efficiency.

The result is improved accuracy on language understanding benchmarks (+2–3 pp on several tasks), increased cross-layer consistency of expert assignments, and naturally balanced expert utilization, obviating the need for auxiliary load-balancing losses (Gu et al., 18 Mar 2026).

2. Path-Based Interpretability: From Experts to Trajectories

MoE models have traditionally focused interpretability and analysis at the level of the expert: e.g., probing for semantic specialization per expert or clustering tokens routed to the same expert. PathMoE generalizes this to the trajectory level.

"Polysemantic Experts, Monosemantic Paths" reframes the hidden state $L$6 at each MoE layer as comprising two orthogonal components:

• Control $L$7: the projection of $L$8 onto the router’s row-space, causally driving expert selection.
• Content $L$9: the router-invisible component, carrying almost all surface-level token information.

PathMoE analysis proceeds as follows:

• Each token $N^L$0 is assigned a discrete path

$N^L$1

where $N^L$2 is the top-1 expert allocated at layer $N^L$3.

• Cluster tokens by identical $N^L$4 and evaluate semantic diversity.

Findings:

• Individual experts are polysemantic (hosting diverse token types/functions).
• Paths, especially in the low-dimensional control subspace, are monosemantic: clusters of paths map closely to semantic function across surface forms and languages.
• For example, ":" tokens serving as a time separator, type annotation, or introductory punctuation each follow distinct expert trajectories, despite surface-form identity (Ye et al., 20 Apr 2026).

This suggests that trajectories—not experts—are a more faithful and robust unit of interpretability in deep MoEs.

3. PathMoE for Expert Pruning and Model Compression

The trajectory-based view naturally informs expert structural pruning. "MoE Pathfinder: Trajectory-driven Expert Pruning" (Yang et al., 20 Dec 2025) formulates the pruning problem as identifying the top-$N^L$5 highest-weight paths in the MoE's expert graph.

Key elements:

• The MoE is modeled as a layered, acyclic computation graph with expert-importance scores (nodes) and transition intensities (edges).
• Path weight for $N^L$6:

$N^L$7

incorporating upstream activation, downstream routing probabilities, and per-expert reconstruction fidelity.

• Top-$N^L$8 path selection is solved via dynamic programming.
• All experts appearing in any top path are retained; others are pruned.

Empirical results:

• At $N^L$9 expert sparsity, PathMoE pruning retains $L$080–90% of baseline task accuracy vs. random pruning’s $L$129%.
• Layerwise, the method yields non-uniform pruning, sometimes collapsing deep layers to a handful of “super-experts” (Yang et al., 20 Dec 2025).

Benefits include non-uniform, calibration-data-driven expert retention; no retraining or custom loss terms are needed.

4. Interaction-Aware Multimodal PathMoE

Outside of language modeling, PathMoE principles have been leveraged in multimodal architectures. "PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification" (Yu et al., 2 Mar 2026) integrates modality-expert and interaction-expert pathways in a gated MoE for interpretable medical prediction.

Core features:

• K=5 experts: image, text, graph (unimodal); redundancy and synergy (interaction).
• Input-dependent gating aggregates expert outputs via sample-specific softmax weights, allowing direct quantification of each modality’s (or their interaction’s) impact per decision.
• A perturbation-based interaction loss encourages specialization: redundancy and synergy experts are contrasted by randomizing individual modalities.

Performance outcomes:

• Adding cell-graph features moves macro-F1 from 0.668 to 0.709 (+0.041) on TCGA datasets.
• For ambiguous or rare subtypes, gating weights (e.g., graph gating $L$2) reveal the model’s reliance on tissue microarchitecture instead of texture or text (Yu et al., 2 Mar 2026).

This approach facilitates direct, case-by-case interpretability—critical for adoption in clinical AI applications.

5. PathMoE in Other Contexts and Extensions

Derivative works exploit trajectory-based MoE concepts in reinforcement learning and trajectory planning (e.g., TrajMoE (Xing et al., 8 Dec 2025)) and suggest several future directions:

• Online and adaptive path selection, updating top paths with new data for continual pruning or specialization (Yang et al., 20 Dec 2025).
• Secondary signals (e.g., Hessians) or task-mixed calibration for path weighting.
• Integration of token-level and path-level routing constraints for further efficiency gains.

A plausible implication is that path-based router sharing and trajectory-level pruning may generalize to sparse, modular neural architectures beyond MoEs, wherever combinatorial routing induces data sparsity.

6. Analytical and Empirical Insights

PathMoE instantiations agree on several empirical findings:

• Path constraint reduces effective path entropy and concentrates training signal.
• Shared routers across blocks induce strong mutual information between consecutive expert selections, increasing path consistency and cluster purity (Gu et al., 18 Mar 2026).
• Pruning or routing at the path level improves downstream robustness to routing perturbations (expert-index scrambling), compared to independent-layer MoEs.
• In the control subspace, cluster mutual information with semantic labels is significantly enhanced ($L$3 MI with language features in the content channel, MI $L$4 in control during mid network stages) (Ye et al., 20 Apr 2026).

These results support the view that expert paths furnish both a more interpretable and a more learnable granular structure than per-layer expert assignments in deep MoEs.

---

Summary Table: Core PathMoE Instantiations

Context	Primary Innovation	Key Reference
Path-constrained routing	Blockwise router parameter sharing, reduced path entropy	(Gu et al., 18 Mar 2026)
PathMoE interpretability	Control-content decomposition, path-based clustering	(Ye et al., 20 Apr 2026)
PathMoE pruning	Global trajectory-level expert selection via dynamic programming	(Yang et al., 20 Dec 2025)
Multimodal PathMoE	Modality- and interaction-aware gating, sample-wise weights	(Yu et al., 2 Mar 2026)

• Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs (Ye et al., 20 Apr 2026)
• Path-Constrained Mixture-of-Experts (Gu et al., 18 Mar 2026)
• MoE Pathfinder: Trajectory-driven Expert Pruning (Yang et al., 20 Dec 2025)
• PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification (Yu et al., 2 Mar 2026)