MoE Operator Selection

• MoE operator selection is a mechanism that assigns a token-specific subset of experts to achieve compute sparsity and adaptive specialization in neural networks.
• It utilizes gating strategies like Top-k, differentiable relaxations, and Bayesian techniques to balance efficiency, diversity, and training stability.
• Recent advances incorporate adaptive routing, operator-learning frameworks, and continuous expert indexing to enhance scalability and performance.

A mixture-of-expert (MoE) style operator selection mechanism designates, for each input (token, feature, or region), a subset of specialized components (“experts”) from a larger pool for computation, and defines how to combine their contributions to produce the operator output. While foundational MoE architectures used fixed or softmax-based gating, recent research has developed a rich taxonomy of selection strategies that balance expressivity, computational efficiency, diversity, and specialization. Key advances address the interplay of sparsity constraints, diversifying expert roles, optimization tractability, and the statistical structure of the gating mechanism.

1. Principles and Motivations Behind MoE Operator Selection

Mixture-of-experts models enable parameter and compute-efficient scaling of neural architectures by sparsely activating a small number of experts per input instance, thereby providing high effective capacity with sublinear resource usage. The primary objectives for MoE operator selection are:

• Compute sparsity: Only a few out of many experts are allocated per instance, reducing cost.
• Expert specialization: Each expert can develop complementary, possibly disjoint, functional behaviors.
• Avoiding redundancy: Simultaneous activation of experts with similar roles leads to wasted compute and reduced capacity utilization.
• Adaptivity: Selection can depend on data complexity, input domain, or other contextual cues.

Early approaches commonly used softmax or Top-$k$ gating, but this often resulted in expert collapse, redundancy, or poor utilization as evidenced in large-scale LLMs (Zheng et al., 15 Oct 2025, Chaudhari et al., 6 Mar 2026). Mechanisms that explicitly decouple expertise diversity, ensure differentiability, and enable adaptive expert use are crucial for optimal operator selection.

2. Gating Mechanisms: Top-$k$, Softmax, and Differentiable Relaxations

The MoE selection process generally involves a gating network that assigns weights or probabilities to each expert for a given input, followed by a sparsification or mixture step:

• Top-$k$ + Softmax: The classic MoE gating computes logits for all $n$ experts, selects the $k$ largest, and normalizes their weights:

$$\alpha_i(x) = \begin{cases} \frac{\exp(g_i(x))}{\sum_{j \in T(g(x))}\exp(g_j(x))} & \text{if } i\in T(g(x)) \ 0 & \text{otherwise} \end{cases}$$

where $T(g(x))$ collects the indices of the largest $k$ logits for input $x$. This design suffers from nondifferentiability in the selection step and limited control over selection structure (Vahidi et al., 9 Feb 2026).

• Differentiable gating: DSelect-k (Hazimeh et al., 2021) addresses nondifferentiability by formulating expert selection as a binary-encoded k-sparse selection, relaxed via smooth-step or sigmoid surrogates:
  • Each of $k$ parallel binary selectors encodes, via relaxed variables, an index in $k$0.
  • Final mixture weights arise from soft assignment, with an entropy penalty pushing codes towards true $k$1-hot vectors.
  • Guarantees on sparsity and gradient flow improve training stability and the ability to scale to large numbers of experts.
• Double-layered gating (DirMoE): Dirichlet-routed MoE (Vahidi et al., 9 Feb 2026) distinctly separates (i) which experts to activate (modeled by Bernoulli-Gumbel variables with explicit sparsity control) and (ii) how to weight them (Dirichlet-distributed mixture weights). The entire selection and routing block is differentiable, and hyperparameters independently modulate selection cardinality and mixture smoothness.

These approaches provide both explicit control and end-to-end trainability for operator selection mechanisms.

3. Diversity-Promoting and Adaptive Expert Selection

Uniform application of Top-$k$2 gating often leads to co-activation of functionally similar experts, which reduces capacity utilization. Addressing this, several diversity-promoting operator selection strategies have been proposed:

• Pairwise similarity suppression (GatePro): GatePro (Zheng et al., 15 Oct 2025) quantifies expert similarity via cosine similarity of router weight vectors,

$k$3

and, for each expert $k$4, finds its most similar peer $k$5. For each token, competition is enforced within each similar pair: the lower logit is penalized by $k$6 before Top-$k$7 is re-applied. This localized, parameter-free, token-dependent suppression prevents redundant co-activation, increases expert diversity (as measured by spectral entropy and angular separation), and yields empirically stronger performance across reasoning and factual benchmarks without introducing auxiliary losses or parameters.

• Dynamic gating by input difficulty: Rather than fixing $k$8, dynamic routing (Huang et al., 2024) adaptively sets the expert count per token according to the gate softmax confidence. The minimal set of experts whose normalized probabilities sum to a threshold $k$9 is activated, allocating more experts for challenging inputs (low confidence) and fewer for simple ones. This leads to cost-effective adaptive computation, demonstrably assigning more experts for hard tasks such as BBH and reducing average compute by $k$010% vs. Top-2 gating without accuracy loss.
• Hypernetwork and modality-aware routing: In multi-modal architectures, static linear routers may ignore token modality, leading to "router rigidity". EvoMoE (Jing et al., 28 May 2025) replaces the router with per-token, per-modality hypernetworks that generate routing weights conditioned on modality (e.g. visual vs. textual). This supports finer specialization and dynamic allocation suited to the token's intrinsic properties.
• RL-based stochastic selection (MoE-GRPO): MoE-GRPO (Ko et al., 26 Mar 2026) frames expert selection as a sequential MDP per token/layer and optimizes stochastic gating policies using group-relative policy optimization. Multi-rollout sampling and reward-based feedback encourage exploration and robust specialization, overcoming the expert collapse of deterministic Top-$k$1 policies. Modality-aware pruning further stabilizes training in vision-language settings.
• Adaptive Bayesian sparsity (HS-MoE): Horseshoe MoE (Polson et al., 14 Jan 2026) models the gate parameters with global-local shrinkage priors, enabling the posterior gating distribution to concentrate effective probability mass on only a few experts per input as dictated by data support. This approach yields theoretically sound, uncertainty-aware routes and offers practical recipes for variational and sequential inference in large-scale settings.

4. Operator Selection in Spatial and Operator-Learning Contexts

MoE selection extends naturally to operator learning and spatially structured problems:

• Partition-of-unity (PoU) gating for spatial operator learning: POU-MOR-Physics (Deighan et al., 6 Feb 2025) and PoU-MoE DeepONet (Sharma et al., 2024) employ spatial gating where, at each location $k$2, the output is a convex combination of spatially localized expert networks:

$k$3

with each $k$4, $k$5, typically modeled via softmax of coordinate MLPs. This gating learns to assign different "expert" operators to subdomains, automatically reflecting geometric or boundary-conditional specialization. Soft partitions enforce smooth transitions and model selection, with one or more "zero" experts enforcing (e.g.) Dirichlet boundaries. Empirical results on PDEs and high-Reynolds flow demonstrate near-perfect spatial decomposition and efficient specialization (Deighan et al., 6 Feb 2025, Sharma et al., 2024).

• Trunk ensembles and spatial locality: Ensemble DeepONet and PoU-MoE DeepONet (Sharma et al., 2024) build architectures where the trunk (evaluation point embedding) is itself an ensemble or spatial blend of specialized trunks, weighted by fixed or adaptive partitions-of-unity. These designs provide provable universal approximation, demonstrable improvements in error rates (up to 30× vs. vanilla DeepONet on some PDEs), and graceful integration of spatially heterogenous operator behaviors.

5. Infinite and Continuous Expert Selection

Traditional MoEs are limited by the number of discrete experts; recent work extends the paradigm to infinite or continuously indexed experts:

• $k$6-MoE: The $k$7-MoE (Takashiro et al., 25 Jan 2026) replaces the finite expert set with a continuous latent expert space $k$8. For each token, a router network predicts parameters of $k$9 (often a Gaussian), from which $n$0 samples are drawn, each defining a unique subnetwork within a shared large FFN via a continuous differentiable mask. The final output is the (typically Monte Carlo) average over the $n$1 sub-networks:

$n$2

This framework enables unbounded, token-adaptive specialization, allows flexible inference-time trade-offs between speed and accuracy via $n$3 and sparsity fraction, and, in experiments, matches or exceeds classical discrete-MoE efficiency and accuracy with far fewer trainable parameters.

• Generalizing to modules and blocks: The continuous-indexing concept can apply to operator selection beyond FFNs, enabling MoE-style routing across arbitrary blocks (attention heads, convolutional filters, etc.), with continuous mixture weights over a learned manifold (Takashiro et al., 25 Jan 2026).

6. Theoretical, Statistical, and Empirical Insights

• Identifiability and estimation rates: Theoretical analysis (Nguyen et al., 2024) demonstrates that selection of strongly identifiable nonlinear experts (e.g., those using sigmoid, tanh, or GELU activations) yields favorable parameter estimation rates ($n$4 for well-separated experts; $n$5 in over-specified settings). Conversely, polynomial or constant experts are "singular" in the softmax-gated MoE, leading to arbitrarily slow convergence and are thus discouraged.
• Expert utilization and inference optimization: Empirical analysis of large MoE LMs (Chaudhari et al., 6 Mar 2026) reveals strong load imbalance—only a handful of experts carry the majority of the routing (e.g., with $n$6, the top-3 experts often account for >50% of routing). The hidden-state similarity between using only the top-1 expert and the full top-$n$7 mixture is near 0.95 in many layers, and perplexity degrades only by about 5% when reducing $n$8 from six to one. This suggests substantive redundancy in large-scale deployments and justifies aggressive expert pruning, conditional inference, or dynamic rerouting.
• Specialization and monosemanticity: Methods explicitly promoting expert diversity or disentangling selection (e.g., GatePro (Zheng et al., 15 Oct 2025), DirMoE (Vahidi et al., 9 Feb 2026), MoE-GRPO (Ko et al., 26 Mar 2026)) yield higher spectral entropy, sharper topic/domain separation, and improved downstream accuracy compared to naive Top-$n$9 gating. These effects hold in both language and multimodal/vision-LLM settings.

7. Future Directions and Design Guidelines

• Decoupling selection and weighting: Disentangling which experts are selected from how their outputs are combined (cf. DirMoE) allows finer-grained control, hyperparameterization, and stability in both training and deployment.
• Structural and modular selection: Embedding modular abstractions (spatial, domain, modality, or continuous latent structure) in the gating and selection process supports specialized, interpretable, and efficient operator selection.
• Adaptive and learnable sparsity: Bayesian approaches (HS-MoE) and entropy/regularization-based schemes offer robust, data-driven, and uncertainty-calibrated control over sparsity, supporting both theoretical soundness and empirical efficiency.
• Operator-learning and scientific modeling: Partition-of-unity-based MoE gating (POU-MoE) is particularly effective in spatially heterogeneous settings such as PDE learning and boundary-conditional modeling, where it serves as a natural operator selector and enables automatic domain decomposition and uncertainty quantification.
• Scalable, differentiable selection: Relaxations like DSelect-k enable large-scale, gradient-based end-to-end optimization for massive expert pools with strict or soft sparsity constraints.

Fundamentally, effective MoE-style operator selection hinges on the interplay between architectural design, selection diversity, efficient and differentiable gating, and adaptation to the inherent structure—statistical, spatial, or semantic—of the target domain.

---

Key References:

• GatePro for parameter-free diversity-promoting expert selection (Zheng et al., 15 Oct 2025)
• DSelect-k for differentiable k-hot expert routing (Hazimeh et al., 2021)
• DirMoE for disentangled, variationally-trained selection and combination (Vahidi et al., 9 Feb 2026)
• MoE-GRPO for RL-based stochastic expert routing (Ko et al., 26 Mar 2026)
• EvoMoE and hypernetwork-based, modality-aware routers (Jing et al., 28 May 2025)
• Horseshoe MoE for Bayesian sparsity in the gating layer (Polson et al., 14 Jan 2026)
• Partition-of-unity MoE for spatial and operator selection (Deighan et al., 6 Feb 2025, Sharma et al., 2024)
• $k$0-MoE for continuous and infinite expert indexing (Takashiro et al., 25 Jan 2026)
• Empirical analysis of expert specialization and redundancy (Chaudhari et al., 6 Mar 2026)
• Identifiability and estimation rates for gating experts (Nguyen et al., 2024)
• Adaptive dynamic expert allocation (Huang et al., 2024)