---
title: Uncertainty-Guided Mixture of Experts
url: https://www.emergentmind.com/topics/uncertainty-guided-mixture-of-experts
type: topic
---

# Uncertainty-Guided Mixture of Experts

Uncertainty-guided mixture of experts denotes a family of Mixture-of-Experts (MoE) designs in which uncertainty is not merely reported after prediction, but participates directly in gating, routing, fusion, or expert invocation. In conventional MoEs, a separate gating network maps the input $x$ to weights $w_i(x)$ over experts. The uncertainty-guided literature modifies that template in several ways: by replacing input-based gates with functions of expert variance, by training experts on partitions of uncertain input densities, by fusing evidential or Dempster–Shafer uncertainty across experts, by placing Bayesian distributions over routing decisions, or by using uncertainty thresholds to trigger expert advice [2510.07459] [2312.08083] [2509.12772] [2603.15708] [2603.09453]. Across time-series forecasting, uncertain tabular learning, medical video analysis, segmentation, low-level vision, large language models, and autonomous driving, the recurring objective is to align expert coordination with predictive confidence, calibration, or risk.

## 1. Conceptual scope and design space

The core departure from classical MoE is explicit in "MoGU: Mixture-of-Gaussians with Uncertainty-based Gating" [2510.07459]: input-based gating “ignores each expert’s own confidence in its prediction,” and “decoupling gating from uncertainty can lead to sub-optimal routing.” MoGU therefore defines gating weights from predicted variances rather than from an auxiliary gate network. By contrast, "Training of Neural Networks with Uncertain Data: A Mixture of Experts Approach" introduces "Uncertainty-aware Mixture of Experts" (uMoE), where uncertainty enters during training through sampling, thresholding, clustering into subspaces, and a gating unit that uses the global mode and subspace-mass vector $C_i$ [2312.08083]. "MEGAN: Mixture of Experts for Robust Uncertainty Estimation in Endoscopy Videos" uses a learned gating network that fuses each expert’s predictions and uncertainties [2509.12772]. "Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers" instead confines Bayesian inference to the expert-selection stage, treating uncertainty as a property of routing itself [2603.09453].

Taken together, these papers suggest several recurrent loci for uncertainty in MoE systems.

| Representative model | Uncertainty-guided mechanism | Setting |
|---|---|---|
| MoGU | gating weights derived from each expert’s estimated variance | time series forecasting |
| uMoE | uncertain inputs partitioned into subspaces; gating uses global mode and $C_i$ | tabular regression and classification |
| MEGAN | gating network fuses per-expert probabilities and scalar uncertainties | endoscopy videos |
| UME | DST certainty measures dynamically weigh expert opinions | hierarchical text classification |
| VMoER | Bayesian uncertainty over routing logits or routing temperature | MoE transformers |

A common misconception is that uncertainty-guided MoE refers to a single routing rule. The literature shows otherwise. In some models uncertainty is an expert-output statistic, in others it is an input-distribution descriptor, a Dirichlet or DST quantity, a posterior over router logits, or a decision trigger for expert intervention. This diversity is structurally important because it changes what is being calibrated: forecast dispersion, expert reliability, routing stability, or safety-critical exploration.

## 2. Uncertainty as a gating signal in regression and forecasting

MoGU is the clearest instance of uncertainty-guided gating as a direct replacement for a conventional gate network. Each of $K$ experts models the conditional distribution of the target as a univariate Gaussian,
$$
y_i(x)\sim \mathcal N(\mu_i(x),\sigma_i^2(x)).
$$
The mean is produced by a standard forecasting head, while the variance is produced by a parallel uncertainty head. In the reported implementation, the variance head is “a shallow MLP with one hidden fully-connected layer of the same dimensionality as $g_i(x)$, followed by a Softplus activation for numerical stability,”
$$
\sigma_i^2(x)=\mathrm{Softplus}(a_i(x))=\log(1+\exp(a_i(x))),
$$
with clamping to a small $\epsilon$ during loss computation to avoid division by zero [2510.07459].

Its routing rule is precision-weighted:
$$
g_i(x)=\frac{1/\sigma_i^2(x)}{\sum_{j=1}^K 1/\sigma_j^2(x)}
=\frac{\sigma_i^{-2}(x)}{\sum_j \sigma_j^{-2}(x)}.
$$
The predictive mean and variance are then
$$
\hat y(x)=\sum_i g_i(x)\mu_i(x),
$$
$$
\mathrm{Var}[y\mid x]=\sum_i g_i(x)\bigl(\sigma_i^2(x)+\mu_i^2(x)\bigr)-\hat y(x)^2,
$$
with an equivalent decomposition into aleatoric and epistemic terms. Training uses a weighted Gaussian negative log-likelihood,
$$
\mathcal L_{\mathrm{MoGU}}(x,y)=\sum_{i=1}^K g_i(x)\,\mathcal L_{\mathrm{NLLG}}\bigl(y;\mu_i(x),\sigma_i^2(x)\bigr),
$$
and the paper reports that this weighted NLLG “yields better expert specialization” than the alternative mixture-NLL objective [2510.07459].

Empirically, MoGU evaluates eight multivariate forecasting benchmarks: ETTh1, ETTh2, ETTm1, ETTm2, Electricity load, Weather, Exchange rates, and Influenza-like Illness. It experiments with $K\in\{2,3,4,5\}$ and reports that $K=3$ trades off accuracy and overhead. Expert backbones are iTransformer, PatchTST, and DLinear. Across horizons $24$–$720$, MoGU “consistently outperforms single-expert models and conventional input-gated MoEs.” A concrete example is ETTh1 at horizon $96$ with three iTransformer experts: MoE yields MSE $0.393$ while MoGU attains $0.380$. For uncertainty quality, total uncertainty achieves $R\approx 0.25$–$0.31$ and $\rho\approx 0.22$–$0.30$, and “aleatoric uncertainty alone often correlates more strongly than epistemic” [2510.07459].

uMoE addresses a different uncertainty source: uncertain inputs rather than uncertain outputs. The training set contains uncertain attributes represented by a full probability density $pdf_i(x)$ over the uncertain inputs, and the method applies a “Divide and Conquer” strategy. It draws $M$ samples from each $pdf_i$, keeps the top fraction $p$, clusters the retained samples with $k$-means into $E$ subspaces, and constructs $C_i$, the fraction of probability mass assigned to each subspace. Experts are then trained on local modes with weighted losses, and a gating unit uses the global mode and $C_i$ to combine experts [2312.08083].

This formulation leads to a distinct interpretation of uncertainty-guided MoE. In MoGU, the gate trusts experts that declare low variance. In uMoE, the system first restructures the uncertain input space, then learns experts on the resulting subspaces, and finally uses uncertainty-derived features in the gate. The reported robustness analysis varies $p\in\{0.1,0.2,\dots,1.0\}$ at uncertainty rates $u=40\%,60\%$, observing best $p\approx 0.8$ for $40\%$ uncertainty and best $p\approx 0.6$ for $60\%$ uncertainty. Using Nested CV, uMoE outperformed four baselines on $12/14$ settings, with “typical gains in MSE of 5–10 % or accuracy lifts of 2–5 pp” [2312.08083]. This suggests that uncertainty-guided expert coordination can be instantiated either as output-aware fusion or as input-aware decomposition.

## 3. Evidential and belief-based expert fusion

In classification, uncertainty-guided MoE often appears through evidential or belief-theoretic outputs rather than Gaussian regression heads. MEGAN is a two-stage framework in which $K$ EDL-based models are first trained as “AI experts,” each with a different ground-truth channel or classifier architecture, and a lightweight gating network then fuses their predictions and uncertainties [2509.12772]. For expert $k$, the model produces evidence $e_{ic}^k=\mathrm{Softplus}(z_{ic}^k)\ge 0$, Dirichlet parameters $\alpha_{ic}^k=e_{ic}^k+1$, class probabilities $p_{ic}^k=\alpha_{ic}^k/S_i^k$, and uncertainty $u_i^k=C/S_i^k$. The gating network consumes the per-expert probability vectors, scalar uncertainties, and a shared feature embedding, and outputs fused probability $\hat p_i$ and fused uncertainty $\hat u_i$.

MEGAN’s fusion is explicitly uncertainty-shaped. Its gating loss is
$$
L_{\mathrm{total}}=L_{\mathrm{cls}}+L_{\mathrm{unc}}+L_\epsilon,
$$
where $L_{\mathrm{unc}}$ encourages high fused uncertainty on mistakes and low fused uncertainty on correct predictions, while $L_\epsilon$ further sharpens that behavior. In a large-scale prospective UC clinical trial, MEGAN achieved “a 3.5% improvement in F1-score and a 30.5% reduction in Expected Calibration Error (ECE) compared to existing methods.” On QUASAR, MEGAN-Gated obtained F1 $0.680$ and ECE $0.107$. The same uncertainty estimate also supports sample triage: using class-specific thresholds $t_c$, MEGAN retains $\sim 90\%$ of videos as “confident” with F1 $\approx 0.75$ and forwards $\sim 10\%$ uncertain ones with F1 $\approx 0.35$ for expert review [2509.12772].

"Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning" develops a different belief-based design, UME, around Dempster–Shafer Theory (DST) [2603.15708]. Each expert is a LoRA adapter attached only to the last feed-forward layer of a frozen BERT backbone. The experts are ordered sequentially: early experts see all samples, while later experts are activated only for samples on which earlier experts remain uncertain. For expert $m$, evidence $e_k^m$ induces Dirichlet parameters $\alpha_k^m=e_k^m+1$, strength $S^m=\sum_k \alpha_k^m$, belief masses $b_k^m=e_k^m/S^m$, and uncertainty $u^m=K/S^m$. Pairwise conflict between consecutive experts is
$$
C^m=\sum_{i\neq j} b_i^m b_j^{m-1}.
$$
Dynamic weights $w^m$ are then formed sequentially and the evidence vectors are fused by a temperature-scaled softmax over the weights. The reported effect is strongest on tail labels: UME wins $9$ out of $16$ tails on RCV1-V2, achieves $79.79\%$ on “e311” versus $61.46\%$ by HiTIN, and improves the average Macro-F1 over the strongest baseline by up to $+17.97\%$ for grouped rare labels [2603.15708].

MoDEX extends the same general theme into a structured probabilistic model of uncertainty-aware classification. It models a latent advocate index $L$ with plausibility weights $w_k(\mathbf x)$, and conditioned on $L=k$, the class-probability vector $\boldsymbol\pi$ is drawn from a Dirichlet distribution with concentration $\boldsymbol\alpha_k(\mathbf x)$. The result is a mixture of Dirichlet opinions,
$$
p(\boldsymbol{\pi}\mid \mathbf{x})
= \sum_{k=1}^K w_k(\mathbf{x})\;\mathrm{Dir}\bigl(\boldsymbol{\pi}\mid \boldsymbol{\alpha}_k(\mathbf{x})\bigr),
$$
with concentrations decomposed into shared evidence and class-specific advocacy [2605.25616]. The paper states that MoDEX “strictly generalizes prior Dirichlet-based methods,” admits an interpretation as “a mixture of $K$ EDL experts,” and decomposes epistemic uncertainty into intra-expert uncertainty and inter-expert disagreement. This suggests that uncertainty-guided MoE is converging with semantically interpretable distributional classification.

## 4. Multimodal aleatoric uncertainty in vision and dense prediction

In segmentation, uncertainty-guided MoE must represent distributions over high-dimensional outputs rather than scalar targets or class labels. "Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts" formulates the conditional distribution of a segmentation mask as
$$
p_\theta(Y\mid X)=\sum_{k=1}^K \pi_k(X)\,p_k(Y\mid X),
$$
where the gating network predicts $\pi_k(X)$ and each expert $k$ is stochastic through a latent Gaussian variable $z_k\sim \mathcal N(m_k,\sigma_k I)$ [2212.07328]. The uncertainty representation is explicitly two-level: the gating probabilities estimate how likely the image belongs to each mode, and each expert generates local shape variations by sampling $z_k$. Learning uses a Wasserstein-like loss defined over an optimal-transport coupling between model samples and empirical annotations, with a relaxed transport constraint and a KL penalty on the gate weights.

This architecture targets aleatoric multimodality directly. On LIDC-IDRI, with sample count $16$, the reported results are GED $0.218$ versus best prior $0.264$, matched IoU $0.624$ versus $0.592$, and ECE $0.064\%$ versus $0.214\%$. On modified multimodal Cityscapes at $35$ samples, “Ours–compact” obtains GED $0.142$, M-IoU $0.761$, and ECE $0.35\%$. Ablations show that replacing the OT loss with pairwise IoU or fixing $\pi$ to uniform degrades GED to $0.533$ or $0.282$, while deterministic experts give $0.246$ [2212.07328]. The factual significance is that both stochastic experts and learned gating are treated as critical components of uncertainty modeling.

"Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation" tackles a different visual regime: efficient multi-task image restoration [2312.16610]. Instead of maintaining $E$ independent FFN experts, it introduces a Feature Modulated Expert block with one shared FFN and $E$ lightweight modulation heads. The expert-specific modulated feature is
$$
\mathrm{FM}_i(x)=\gamma^{(i)}\circ x+\beta^{(i)},
$$
and the FME output is
$$
\mathrm{FME}(x)=\mathrm{FFN}\Bigl(\sum_{i=1}^E r_i(x)\,[\gamma^{(i)}\circ x+\beta^{(i)}]\Bigr).
$$
Its Uncertainty-aware Router estimates router uncertainty by MC-dropout, computes the mean $\hat\mu$ and covariance $\hat\Sigma$ of router outputs, and whitens the logits,
$$
\tilde r(x)=\Sigmâ^{-1}[r(x)-\mû]/\|\Sigmâ^{-1}[r(x)-\mû]\|_2,
$$
before softmax and TopK selection [2312.16610].

The reported benefits are both algorithmic and systems-level. On All-Weather with $E=128$, conventional MoE has $309$M parameters and $0.075$ s/image, whereas MoFME has $85$M parameters and $0.046$ s/image, corresponding to $-72.5\%$ parameters and $-39\%$ inference time. On RainCityscapes, MoFME reaches $32.11$ dB PSNR and $0.9691$ SSIM against $31.89$ dB and $0.9674$ for MoE-ViT, with $18.53$M parameters instead of $44.19$M and $37.43$ GMAC instead of $41.31$ [2312.16610]. In this line of work, uncertainty guidance is not only a calibration device; it is also part of efficient expert allocation.

## 5. Bayesian routing, online aggregation, and expert advice

A major recent direction shifts uncertainty from expert outputs to the routing mechanism itself. "Bayesian Mixture of Experts For Large Language Models" applies a structured Laplace approximation to the second linear layer of each expert in fine-tuned MoE LLMs, with an isotropic Gaussian prior over $\theta=\{W_{2,i}\}_{i=1}^E$ and a block-diagonal Kronecker-factored approximation to the Hessian [2511.08968]. For a test input $x_*$, the predictive covariance of the logit vector is
$$
\Lambda(x_*)=
\sum_{i\in\mathcal E_*}
\bigl[\nabla_{W_{2,i}}f\bigr]^\top \Sigma^{(i)} \bigl[\nabla_{W_{2,i}}f\bigr].
$$
The method is explicitly post-hoc and introduces no new parameters. On Qwen1.5-MoE, the reported excerpt gives ECE $31.2\%\rightarrow 3.1\%$, NLL $3.15\rightarrow 0.55$, and ACC $67.4\%\rightarrow 67.8\%$ from MAP to Bayesian-MoE; on DeepSeek-MoE, ECE improves from $\sim 32\%\rightarrow 2.5\%$ and NLL from $\sim 3.4\rightarrow 0.6$ [2511.08968].

The companion Bayesian-routing line places uncertainty directly over router weights, router logits, or the final selection distribution. "Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know" studies weight-space, logit-space, and selection-space routers on IBM-Granite-3.1 3B MoE [2509.23830]. The deterministic baseline has Jaccard similarity $\approx 0.650\pm 0.08$ under Gaussian noise and ECE $0.252$ on OpenBookQA. The logit-space Full-Covariance Variational Router reports Jaccard $0.897$ and ECE $0.015$, while MFVR gives ECE $0.026$ and MCDR gives ECE $0.037$ [2509.23830]. "Variational Routing" refines this approach by freezing the original linear router, attaching an inference network in parallel, and optimizing either a Gaussian ELBO in logit-space,
$$
\mathcal L_{\mathrm{ELBO}}
= \mathbb E_{q_\phi(l\mid u)}[\log p(y\mid l,u)]
- \beta\,\mathrm{KL}(q_\phi(l\mid u)\|p(l\mid u)),
$$
or a temperature-based selection-space objective [2603.09453]. Across tested foundation models, VMoER reports routing-stability gains under noise of $38\%$, a $94\%$ reduction in calibration error, a $12\%$ increase in OoD AUROC, and less than $1\%$ additional FLOPs [2603.09453].

The literature also extends uncertainty guidance beyond static neural routing. "Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making" formulates committee selection as a contextual-bandit problem in which experts vote and the learner must adaptively determine the optimal committee or weight vector [2510.21788]. Its UCB-driven successive elimination algorithm uses confidence intervals
$$
U_i^t=\sqrt{\frac{2\sigma^2\ln(4T)}{n_i(t)}}
$$
to prune suboptimal experts, yielding the regret bound
$$
R_T=O\!\Bigl(\frac{N}{\min_{i\neq j}|p_i-p_j|^2}\ln T\Bigr),
$$
while the weighted-majority variant has
$$
R_T=O\!\bigl(\sqrt{NT\ln T}\bigr).
$$
In simulations with Bernoulli experts, SEE reaches $R_T\approx 5.5\times 10^2$ versus CombUCB’s $7\times 10^3$, and on GSM8K $\theta$-WMV reaches final accuracy $92.7\%$ versus $60.4\%$ for Zooming [2510.21788]. Here uncertainty guidance takes the form of confidence intervals over expert competence.

A safety-critical variant appears in autonomous-driving reinforcement learning. "Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving" computes epistemic uncertainty from disagreement across IQN heads and aleatoric uncertainty from lower-tail quantiles of the risk-sensitive greedy action [2605.30576]. Expert advice is triggered when either uncertainty exceeds an adaptive percentile threshold,
$$
\phi_t=\mathbf 1\!\bigl(U^{\mathrm{epi}}(o_t)>T_t^{\mathrm{epi}}\;\lor\;U^{\mathrm{ale}}(o_t)>T_t^{\mathrm{ale}}\bigr),
$$
and is then governed by a commitment-cooldown strategy and a stochastic early-stop heuristic based on $P_{\mathrm{imp}}$. In CARLA, the method “improving success by 5-7% and reducing failures” over the IQN baseline [2605.30576]. This is not a standard neural MoE layer, but it preserves the central principle that uncertainty should regulate when and how expert behavior is trusted.

## 6. Limitations, misconceptions, and open directions

A first limitation is computational form. MoGU states that its gating is dense—“all experts contribute”—and may therefore be “computationally heavier than sparse MoEs in large-scale LLMs.” The same paper notes that “variance heads can collapse if not carefully regularized, potentially underestimating uncertainty” [2510.07459]. MoFME addresses efficiency by redesigning the expert block itself, whereas Bayesian router methods target less than $1\%$ extra FLOPs by modifying only routing or selected expert layers [2312.16610] [2603.09453]. A plausible implication is that uncertainty-guided MoE divides into dense, calibration-oriented designs and sparse, systems-oriented designs.

A second misconception is that “uncertainty” is uniform across papers. MoGU decomposes total predictive variance into aleatoric and epistemic terms; MEGAN and MoDEX use Dirichlet evidence; UME uses DST belief, ignorance, and conflict; VMoER models uncertainty in routing logits or routing temperature; the autonomous-driving framework distinguishes epistemic from aleatoric uncertainty for advice triggering [2510.07459] [2509.12772] [2603.15708] [2605.25616] [2603.09453] [2605.30576]. Consequently, numerical calibration metrics are not directly interchangeable across domains.

A third limitation is methodological scope. uMoE reports that “no formal p-value tests” were reported. Bayesian-MoE only Bayesianizes the second linear layer, “block-diagonal (independent) posterior over experts neglects cross-expert correlations,” and evaluations are “restricted to multiple-choice QA.” The Bayesian-routing thesis states that VTSR training stability needs improvement, and UME reports that with more experts “more experts overfit on tiny conflict signals” [2312.08083] [2511.08968] [2509.23830] [2603.15708]. These are not peripheral caveats; they identify active constraints on what current uncertainty-guided MoE systems can guarantee.

The most explicit future directions are architectural. MoGU proposes replacing Gaussian experts with heavy-tailed or heteroscedastic distributions such as Student-$t$, combining input-based and uncertainty-based gating, sparse activation of only the top-$M$ confident experts, and post-hoc variance scaling or conformal prediction [2510.07459]. MEGAN suggests incorporating additional experts and online learning of gating-network weights [2509.12772]. Bayesian-MoE identifies “a natural next step” in feeding per-expert uncertainty back into the gating network to enable “uncertainty-aware gating,” while the Bayesian-routing thesis proposes dynamic expert expansion and user-interactive uncertainty feedback [2511.08968] [2509.23830]. Taken together, these proposals suggest that the field is moving from uncertainty as an auxiliary output toward uncertainty as a control signal for specialization, calibration, deferment, and adaptive computation.

Source: https://www.emergentmind.com/topics/uncertainty-guided-mixture-of-experts