Uncertainty-Guided Mixture of Experts

• The paper shows that using uncertainty estimates to guide gating and routing improves expert specialization and calibration across various domains.
• Uncertainty-guided MoEs incorporate both aleatoric and epistemic measures to refine predictions in tasks such as time-series forecasting, segmentation, and classification.
• By integrating Bayesian, evidential, and belief-based methods, these approaches balance computational efficiency with enhanced risk management and performance.

Uncertainty-guided mixture of experts denotes a family of Mixture-of-Experts (MoE) designs in which uncertainty is not merely reported after prediction, but participates directly in gating, routing, fusion, or expert invocation. In conventional MoEs, a separate gating network maps the input $x$ to weights $w_i(x)$ over experts. The uncertainty-guided literature modifies that template in several ways: by replacing input-based gates with functions of expert variance, by training experts on partitions of uncertain input densities, by fusing evidential or Dempster–Shafer uncertainty across experts, by placing Bayesian distributions over routing decisions, or by using uncertainty thresholds to trigger expert advice (Shavit et al., 8 Oct 2025, Luttner, 2023, Agbelese et al., 16 Sep 2025, Wang et al., 16 Mar 2026, Li et al., 10 Mar 2026). Across time-series forecasting, uncertain tabular learning, medical video analysis, segmentation, low-level vision, LLMs, and autonomous driving, the recurring objective is to align expert coordination with predictive confidence, calibration, or risk.

1. Conceptual scope and design space

The core departure from classical MoE is explicit in "MoGU: Mixture-of-Gaussians with Uncertainty-based Gating" (Shavit et al., 8 Oct 2025): input-based gating “ignores each expert’s own confidence in its prediction,” and “decoupling gating from uncertainty can lead to sub-optimal routing.” MoGU therefore defines gating weights from predicted variances rather than from an auxiliary gate network. By contrast, "Training of Neural Networks with Uncertain Data: A Mixture of Experts Approach" introduces "Uncertainty-aware Mixture of Experts" (uMoE), where uncertainty enters during training through sampling, thresholding, clustering into subspaces, and a gating unit that uses the global mode and subspace-mass vector $C_i$ (Luttner, 2023). "MEGAN: Mixture of Experts for Robust Uncertainty Estimation in Endoscopy Videos" uses a learned gating network that fuses each expert’s predictions and uncertainties (Agbelese et al., 16 Sep 2025). "Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers" instead confines Bayesian inference to the expert-selection stage, treating uncertainty as a property of routing itself (Li et al., 10 Mar 2026).

Taken together, these papers suggest several recurrent loci for uncertainty in MoE systems.

Representative model	Uncertainty-guided mechanism	Setting
MoGU	gating weights derived from each expert’s estimated variance	time series forecasting
uMoE	uncertain inputs partitioned into subspaces; gating uses global mode and $C_i$	tabular regression and classification
MEGAN	gating network fuses per-expert probabilities and scalar uncertainties	endoscopy videos
UME	DST certainty measures dynamically weigh expert opinions	hierarchical text classification
VMoER	Bayesian uncertainty over routing logits or routing temperature	MoE transformers

A common misconception is that uncertainty-guided MoE refers to a single routing rule. The literature shows otherwise. In some models uncertainty is an expert-output statistic, in others it is an input-distribution descriptor, a Dirichlet or DST quantity, a posterior over router logits, or a decision trigger for expert intervention. This diversity is structurally important because it changes what is being calibrated: forecast dispersion, expert reliability, routing stability, or safety-critical exploration.

2. Uncertainty as a gating signal in regression and forecasting

MoGU is the clearest instance of uncertainty-guided gating as a direct replacement for a conventional gate network. Each of $K$ experts models the conditional distribution of the target as a univariate Gaussian,

$y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$

The mean is produced by a standard forecasting head, while the variance is produced by a parallel uncertainty head. In the reported implementation, the variance head is “a shallow MLP with one hidden fully-connected layer of the same dimensionality as $g_i(x)$, followed by a Softplus activation for numerical stability,”

$$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$

with clamping to a small $\epsilon$ during loss computation to avoid division by zero (Shavit et al., 8 Oct 2025).

Its routing rule is precision-weighted:

$$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$

The predictive mean and variance are then

$w_i(x)$0

$w_i(x)$1

with an equivalent decomposition into aleatoric and epistemic terms. Training uses a weighted Gaussian negative log-likelihood,

$w_i(x)$2

and the paper reports that this weighted NLLG “yields better expert specialization” than the alternative mixture-NLL objective (Shavit et al., 8 Oct 2025).

Empirically, MoGU evaluates eight multivariate forecasting benchmarks: ETTh1, ETTh2, ETTm1, ETTm2, Electricity load, Weather, Exchange rates, and Influenza-like Illness. It experiments with $w_i(x)$3 and reports that $w_i(x)$4 trades off accuracy and overhead. Expert backbones are iTransformer, PatchTST, and DLinear. Across horizons $w_i(x)$5–$w_i(x)$6, MoGU “consistently outperforms single-expert models and conventional input-gated MoEs.” A concrete example is ETTh1 at horizon $w_i(x)$7 with three iTransformer experts: MoE yields MSE $w_i(x)$8 while MoGU attains $w_i(x)$9. For uncertainty quality, total uncertainty achieves $C_i$0–$C_i$1 and $C_i$2–$C_i$3, and “aleatoric uncertainty alone often correlates more strongly than epistemic” (Shavit et al., 8 Oct 2025).

uMoE addresses a different uncertainty source: uncertain inputs rather than uncertain outputs. The training set contains uncertain attributes represented by a full probability density $C_i$4 over the uncertain inputs, and the method applies a “Divide and Conquer” strategy. It draws $C_i$5 samples from each $C_i$6, keeps the top fraction $C_i$7, clusters the retained samples with $C_i$8-means into $C_i$9 subspaces, and constructs $C_i$0, the fraction of probability mass assigned to each subspace. Experts are then trained on local modes with weighted losses, and a gating unit uses the global mode and $C_i$1 to combine experts (Luttner, 2023).

This formulation leads to a distinct interpretation of uncertainty-guided MoE. In MoGU, the gate trusts experts that declare low variance. In uMoE, the system first restructures the uncertain input space, then learns experts on the resulting subspaces, and finally uses uncertainty-derived features in the gate. The reported robustness analysis varies $C_i$2 at uncertainty rates $C_i$3, observing best $C_i$4 for $C_i$5 uncertainty and best $C_i$6 for $C_i$7 uncertainty. Using Nested CV, uMoE outperformed four baselines on $C_i$8 settings, with “typical gains in MSE of 5–10 % or accuracy lifts of 2–5 pp” (Luttner, 2023). This suggests that uncertainty-guided expert coordination can be instantiated either as output-aware fusion or as input-aware decomposition.

3. Evidential and belief-based expert fusion

In classification, uncertainty-guided MoE often appears through evidential or belief-theoretic outputs rather than Gaussian regression heads. MEGAN is a two-stage framework in which $C_i$9 EDL-based models are first trained as “AI experts,” each with a different ground-truth channel or classifier architecture, and a lightweight gating network then fuses their predictions and uncertainties (Agbelese et al., 16 Sep 2025). For expert $K$0, the model produces evidence $K$1, Dirichlet parameters $K$2, class probabilities $K$3, and uncertainty $K$4. The gating network consumes the per-expert probability vectors, scalar uncertainties, and a shared feature embedding, and outputs fused probability $K$5 and fused uncertainty $K$6.

MEGAN’s fusion is explicitly uncertainty-shaped. Its gating loss is

$K$7

where $K$8 encourages high fused uncertainty on mistakes and low fused uncertainty on correct predictions, while $K$9 further sharpens that behavior. In a large-scale prospective UC clinical trial, MEGAN achieved “a 3.5% improvement in F1-score and a 30.5% reduction in Expected Calibration Error (ECE) compared to existing methods.” On QUASAR, MEGAN-Gated obtained F1 $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$0 and ECE $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$1. The same uncertainty estimate also supports sample triage: using class-specific thresholds $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$2, MEGAN retains $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$3 of videos as “confident” with F1 $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$4 and forwards $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$5 uncertain ones with F1 $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$6 for expert review (Agbelese et al., 16 Sep 2025).

"Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning" develops a different belief-based design, UME, around Dempster–Shafer Theory (DST) (Wang et al., 16 Mar 2026). Each expert is a LoRA adapter attached only to the last feed-forward layer of a frozen BERT backbone. The experts are ordered sequentially: early experts see all samples, while later experts are activated only for samples on which earlier experts remain uncertain. For expert $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$7, evidence $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$8 induces Dirichlet parameters $y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).$9, strength $g_i(x)$0, belief masses $g_i(x)$1, and uncertainty $g_i(x)$2. Pairwise conflict between consecutive experts is

$g_i(x)$3

Dynamic weights $g_i(x)$4 are then formed sequentially and the evidence vectors are fused by a temperature-scaled softmax over the weights. The reported effect is strongest on tail labels: UME wins $g_i(x)$5 out of $g_i(x)$6 tails on RCV1-V2, achieves $g_i(x)$7 on “e311” versus $g_i(x)$8 by HiTIN, and improves the average Macro-F1 over the strongest baseline by up to $g_i(x)$9 for grouped rare labels (Wang et al., 16 Mar 2026).

MoDEX extends the same general theme into a structured probabilistic model of uncertainty-aware classification. It models a latent advocate index $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$0 with plausibility weights $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$1, and conditioned on $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$2, the class-probability vector $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$3 is drawn from a Dirichlet distribution with concentration $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$4. The result is a mixture of Dirichlet opinions,

$$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$5

with concentrations decomposed into shared evidence and class-specific advocacy (Yoon et al., 25 May 2026). The paper states that MoDEX “strictly generalizes prior Dirichlet-based methods,” admits an interpretation as “a mixture of $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$6 EDL experts,” and decomposes epistemic uncertainty into intra-expert uncertainty and inter-expert disagreement. This suggests that uncertainty-guided MoE is converging with semantically interpretable distributional classification.

4. Multimodal aleatoric uncertainty in vision and dense prediction

In segmentation, uncertainty-guided MoE must represent distributions over high-dimensional outputs rather than scalar targets or class labels. "Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts" formulates the conditional distribution of a segmentation mask as

$$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$7

where the gating network predicts $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$8 and each expert $$\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),$$9 is stochastic through a latent Gaussian variable $\epsilon$0 (Gao et al., 2022). The uncertainty representation is explicitly two-level: the gating probabilities estimate how likely the image belongs to each mode, and each expert generates local shape variations by sampling $\epsilon$1. Learning uses a Wasserstein-like loss defined over an optimal-transport coupling between model samples and empirical annotations, with a relaxed transport constraint and a KL penalty on the gate weights.

This architecture targets aleatoric multimodality directly. On LIDC-IDRI, with sample count $\epsilon$2, the reported results are GED $\epsilon$3 versus best prior $\epsilon$4, matched IoU $\epsilon$5 versus $\epsilon$6, and ECE $\epsilon$7 versus $\epsilon$8. On modified multimodal Cityscapes at $\epsilon$9 samples, “Ours–compact” obtains GED $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$0, M-IoU $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$1, and ECE $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$2. Ablations show that replacing the OT loss with pairwise IoU or fixing $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$3 to uniform degrades GED to $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$4 or $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$5, while deterministic experts give $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$6 (Gao et al., 2022). The factual significance is that both stochastic experts and learned gating are treated as critical components of uncertainty modeling.

"Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation" tackles a different visual regime: efficient multi-task image restoration (Zhang et al., 2023). Instead of maintaining $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$7 independent FFN experts, it introduces a Feature Modulated Expert block with one shared FFN and $$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$8 lightweight modulation heads. The expert-specific modulated feature is

$$g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)} =\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.$$9

and the FME output is

$w_i(x)$00

Its Uncertainty-aware Router estimates router uncertainty by MC-dropout, computes the mean $w_i(x)$01 and covariance $w_i(x)$02 of router outputs, and whitens the logits,

$w_i(x)$03

before softmax and TopK selection (Zhang et al., 2023).

The reported benefits are both algorithmic and systems-level. On All-Weather with $w_i(x)$04, conventional MoE has $w_i(x)$05M parameters and $w_i(x)$06 s/image, whereas MoFME has $w_i(x)$07M parameters and $w_i(x)$08 s/image, corresponding to $w_i(x)$09 parameters and $w_i(x)$10 inference time. On RainCityscapes, MoFME reaches $w_i(x)$11 dB PSNR and $w_i(x)$12 SSIM against $w_i(x)$13 dB and $w_i(x)$14 for MoE-ViT, with $w_i(x)$15M parameters instead of $w_i(x)$16M and $w_i(x)$17 GMAC instead of $w_i(x)$18 (Zhang et al., 2023). In this line of work, uncertainty guidance is not only a calibration device; it is also part of efficient expert allocation.

5. Bayesian routing, online aggregation, and expert advice

A major recent direction shifts uncertainty from expert outputs to the routing mechanism itself. "Bayesian Mixture of Experts For LLMs" applies a structured Laplace approximation to the second linear layer of each expert in fine-tuned MoE LLMs, with an isotropic Gaussian prior over $w_i(x)$19 and a block-diagonal Kronecker-factored approximation to the Hessian (Dialameh et al., 12 Nov 2025). For a test input $w_i(x)$20, the predictive covariance of the logit vector is

$w_i(x)$21

The method is explicitly post-hoc and introduces no new parameters. On Qwen1.5-MoE, the reported excerpt gives ECE $w_i(x)$22, NLL $w_i(x)$23, and ACC $w_i(x)$24 from MAP to Bayesian-MoE; on DeepSeek-MoE, ECE improves from $w_i(x)$25 and NLL from $w_i(x)$26 (Dialameh et al., 12 Nov 2025).

The companion Bayesian-routing line places uncertainty directly over router weights, router logits, or the final selection distribution. "Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know" studies weight-space, logit-space, and selection-space routers on IBM-Granite-3.1 3B MoE (Li, 28 Sep 2025). The deterministic baseline has Jaccard similarity $w_i(x)$27 under Gaussian noise and ECE $w_i(x)$28 on OpenBookQA. The logit-space Full-Covariance Variational Router reports Jaccard $w_i(x)$29 and ECE $w_i(x)$30, while MFVR gives ECE $w_i(x)$31 and MCDR gives ECE $w_i(x)$32 (Li, 28 Sep 2025). "Variational Routing" refines this approach by freezing the original linear router, attaching an inference network in parallel, and optimizing either a Gaussian ELBO in logit-space,

$w_i(x)$33

or a temperature-based selection-space objective (Li et al., 10 Mar 2026). Across tested foundation models, VMoER reports routing-stability gains under noise of $w_i(x)$34, a $w_i(x)$35 reduction in calibration error, a $w_i(x)$36 increase in OoD AUROC, and less than $w_i(x)$37 additional FLOPs (Li et al., 10 Mar 2026).

The literature also extends uncertainty guidance beyond static neural routing. "Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making" formulates committee selection as a contextual-bandit problem in which experts vote and the learner must adaptively determine the optimal committee or weight vector (Liu et al., 19 Oct 2025). Its UCB-driven successive elimination algorithm uses confidence intervals

$w_i(x)$38

to prune suboptimal experts, yielding the regret bound

$w_i(x)$39

while the weighted-majority variant has

$w_i(x)$40

In simulations with Bernoulli experts, SEE reaches $w_i(x)$41 versus CombUCB’s $w_i(x)$42, and on GSM8K $w_i(x)$43-WMV reaches final accuracy $w_i(x)$44 versus $w_i(x)$45 for Zooming (Liu et al., 19 Oct 2025). Here uncertainty guidance takes the form of confidence intervals over expert competence.

A safety-critical variant appears in autonomous-driving reinforcement learning. "Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving" computes epistemic uncertainty from disagreement across IQN heads and aleatoric uncertainty from lower-tail quantiles of the risk-sensitive greedy action (Abouelazm et al., 28 May 2026). Expert advice is triggered when either uncertainty exceeds an adaptive percentile threshold,

$w_i(x)$46

and is then governed by a commitment-cooldown strategy and a stochastic early-stop heuristic based on $w_i(x)$47. In CARLA, the method “improving success by 5-7% and reducing failures” over the IQN baseline (Abouelazm et al., 28 May 2026). This is not a standard neural MoE layer, but it preserves the central principle that uncertainty should regulate when and how expert behavior is trusted.

6. Limitations, misconceptions, and open directions

A first limitation is computational form. MoGU states that its gating is dense—“all experts contribute”—and may therefore be “computationally heavier than sparse MoEs in large-scale LLMs.” The same paper notes that “variance heads can collapse if not carefully regularized, potentially underestimating uncertainty” (Shavit et al., 8 Oct 2025). MoFME addresses efficiency by redesigning the expert block itself, whereas Bayesian router methods target less than $w_i(x)$48 extra FLOPs by modifying only routing or selected expert layers (Zhang et al., 2023, Li et al., 10 Mar 2026). A plausible implication is that uncertainty-guided MoE divides into dense, calibration-oriented designs and sparse, systems-oriented designs.

A second misconception is that “uncertainty” is uniform across papers. MoGU decomposes total predictive variance into aleatoric and epistemic terms; MEGAN and MoDEX use Dirichlet evidence; UME uses DST belief, ignorance, and conflict; VMoER models uncertainty in routing logits or routing temperature; the autonomous-driving framework distinguishes epistemic from aleatoric uncertainty for advice triggering (Shavit et al., 8 Oct 2025, Agbelese et al., 16 Sep 2025, Wang et al., 16 Mar 2026, Yoon et al., 25 May 2026, Li et al., 10 Mar 2026, Abouelazm et al., 28 May 2026). Consequently, numerical calibration metrics are not directly interchangeable across domains.

A third limitation is methodological scope. uMoE reports that “no formal p-value tests” were reported. Bayesian-MoE only Bayesianizes the second linear layer, “block-diagonal (independent) posterior over experts neglects cross-expert correlations,” and evaluations are “restricted to multiple-choice QA.” The Bayesian-routing thesis states that VTSR training stability needs improvement, and UME reports that with more experts “more experts overfit on tiny conflict signals” (Luttner, 2023, Dialameh et al., 12 Nov 2025, Li, 28 Sep 2025, Wang et al., 16 Mar 2026). These are not peripheral caveats; they identify active constraints on what current uncertainty-guided MoE systems can guarantee.

The most explicit future directions are architectural. MoGU proposes replacing Gaussian experts with heavy-tailed or heteroscedastic distributions such as Student-$w_i(x)$49, combining input-based and uncertainty-based gating, sparse activation of only the top-$w_i(x)$50 confident experts, and post-hoc variance scaling or conformal prediction (Shavit et al., 8 Oct 2025). MEGAN suggests incorporating additional experts and online learning of gating-network weights (Agbelese et al., 16 Sep 2025). Bayesian-MoE identifies “a natural next step” in feeding per-expert uncertainty back into the gating network to enable “uncertainty-aware gating,” while the Bayesian-routing thesis proposes dynamic expert expansion and user-interactive uncertainty feedback (Dialameh et al., 12 Nov 2025, Li, 28 Sep 2025). Taken together, these proposals suggest that the field is moving from uncertainty as an auxiliary output toward uncertainty as a control signal for specialization, calibration, deferment, and adaptive computation.