---
title: Expert-Guided Sampling Strategies
url: https://www.emergentmind.com/topics/expert-guided-sampling
type: topic
---

# Expert-Guided Sampling Strategies

Searching arXiv for recent papers on expert-guided sampling and closely related formulations.
arXiv search query: "expert-guided sampling OR uncertainty-guided expert sampling OR expert-sample routing MoE"
Expert-guided sampling is a family of selection, routing, and acquisition strategies in which sampling decisions are steered by an expert signal rather than by uniform choice or unconstrained stochasticity. In the literature, the expert signal may be a human annotation, a domain prior over optima, a specialist policy, a verifier, a routing distribution over Mixture-of-Experts (MoE) subnetworks, or an external foundation model. The common objective is to concentrate computation, data acquisition, or exploration where it most improves diversity, stability, safety, calibration, or sample efficiency.

## 1. Scope and conceptual taxonomy

The surveyed literature suggests that “expert-guided sampling” is not a single method class but a recurrent design pattern appearing at different stages of the pipeline: inference-time generation, reinforcement learning, Bayesian optimization, active data acquisition, evaluation subset construction, explainability supervision, and model compression.

| Setting | Guiding expert signal | Representative papers |
|---|---|---|
| LLM and MoE inference | Router scores, difficulty estimates, verifier scores, specialized experts | [2602.02443], [2604.10335] |
| Control and optimization | Human demonstrations, guardian interventions, expert prior over optimum | [2110.06831], [2212.08232], [2002.11256] |
| Annotation, interpretability, and evaluation | Expert ROIs, expert disagreement, confidence-guided case selection | [2509.08007], [2601.02409], [2403.16594], [2412.02173], [2406.07967] |

Two distinctions are especially important. First, “expert” is overloaded: in MoE work it refers to sparse subnetworks, whereas in clinical or robotics work it denotes human raters, radiologists, or expert policies. Second, the guidance signal may act on different objects: tokens, experts, trajectories, data points, prompts, calibration samples, or human-evaluation items. This suggests that the unifying property is not the source of expertise alone, but the use of expertise to bias sampling toward decisions with higher expected utility.

## 2. Inference-time routing and generation in language models

In fine-grained MoE inference, "Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE" identifies a routing pattern in which router scores contain a “certain head” of high-confidence experts and an “uncertain tail” of low-confidence candidates [2602.02443]. Empirically, reducing activated experts by half does not hurt greedy decoding accuracy, but multi-sample pass@n drops sharply. The method therefore preserves the top-ranked experts deterministically and samples the remaining experts from a broader candidate pool using router logits and Gumbel-Top-K sampling. With token representation $\mathbf{h}$, router logits $\mathbf{g} = \mathbf{h}\cdot\mathbf{W}_g$, probabilities $\mathbf{p}=\mathrm{softmax}(\mathbf{g})$, and selected experts $\mathcal{S}$, the output is aggregated as
$$
\mathbf{o}=\sum_{i\in\mathcal{S}} \tilde{p}_i \cdot \mathrm{Expert}_i(\mathbf{h}).
$$
The default configuration is
$$
k_{\text{keep}}=\lfloor k/2\rfloor+1,\quad \tau=1.0,\quad r=4k.
$$
On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification; across 20 model-task pairs, the reported average gain is 4.3% for BoN and 3.1% for WMV, with runtime overhead within $\pm1\%$ [2602.02443].

"Adaptive Multi-Expert Reasoning via Difficulty-Aware Routing and Uncertainty-Guided Aggregation" implements expert-guided sampling by predicting problem difficulty and uncertainty from problem text, then adapting both the number and diversity of candidate generations [2604.10335]. The router computes
$$
U(x)=\frac{1}{2}H(p(x))+\frac{1}{2}\left(1-2\left|p_{\text{hard}}(x)-0.5\right|\right).
$$
When $U<0.35$, the system generates one deterministic candidate; when $0.35\le U<0.55$, each expert generates one candidate at low temperature; when $U\ge0.55$, each expert generates two candidates at temperatures 0.0 and 0.15. Three LoRA-adapted experts—Algebraic Expert, Intuitive Expert, and Step-by-Step Expert—are coupled with correction, finalization, neural verification, and clustering-based aggregation. On GSM8K, AMR achieved 75.28% accuracy while only using the original training data [2604.10335].

A plausible implication is that recent expert-guided inference methods decouple diversity from token-level temperature. In MoE routing, diversity is injected into expert selection; in multi-expert reasoning, diversity is allocated conditionally on estimated difficulty and uncertainty.

## 3. Safe exploration, posterior sampling, and offline reinforcement learning

In Bayesian optimization, "Incorporating Expert Prior Knowledge into Experimental Design via Posterior Sampling" formalizes expert guidance as a prior distribution $\pi(\boldsymbol{x}^*)$ over the unknown global optimum location [2002.11256]. The posterior becomes
$$
p(\boldsymbol{x}^* \mid \mathcal{D}_n,\pi)\propto p(\boldsymbol{x}^* \mid \mathcal{D}_n)\pi(\boldsymbol{x}^*),
$$
and posterior sampling is implemented by drawing functions from the posterior GP, finding their maximizers, weighting these maximizers by $\pi(\boldsymbol{x}_i^*)$, and sampling the next experiment accordingly. The method is analyzed as no-regret; when the prior is informative it accelerates convergence, and when the prior is mis-specified the GP posterior eventually dominates. The same paper notes that an extremely misleading prior can worsen initial performance [2002.11256].

In safe reinforcement learning, "Safe Driving via Expert Guided Policy Optimization" introduces a guardian consisting of an expert policy $\mathcal{E}$ and a switch function $\mathcal{T}$ that intervenes when the agent proposes unsafe actions [2110.06831]. The switch is
$$
\mathcal{T}(s,a,\mathcal{E})=(\hat{a},\hat{c})=
\begin{cases}
(a,0), & \text{if } a\in\mathcal{A}_\eta(s)\\
(a^E\sim\mathcal{E}(\cdot|s),1), & \text{otherwise.}
\end{cases}
$$
EGPO combines constrained optimization to prevent the agent from exploiting interventions with Conservative Q-Learning on partial demonstrations. In human-in-the-loop experiments it achieves 90% success rate with only 15,000 environment steps, versus 185,000 for the best RL baseline, and reports test cost 0.56 versus 1.26 for the expert and 2.21 for the best Safe RL baseline [2110.06831].

"Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling" uses an ensemble of $M$ Q-networks to estimate epistemic uncertainty by variance,
$$
\sigma^2(s,a)=\frac{1}{M}\sum_{i=1}^M\left(Q_{\theta^i}(s,a)-\mu(s,a)\right)^2,
$$
and switches between a sub-optimal-agent buffer and a human-demonstration buffer according to a threshold $\epsilon$ [2212.08232]. The reported outcome is more sample-efficient learning than naïve mixing, with more than $2\times$ faster convergence in several environments and up to $5\times$ more human demonstrations required by the naïve method to match performance in Monolith [2212.08232].

Across these works, expert-guided sampling functions as a control mechanism on exploration: expert priors bias query selection in expensive optimization, guardian intervention restricts unsafe trajectory sampling, and uncertainty gates when scarce expert demonstrations should enter the training stream.

## 4. Prompt refinement, finite-population acquisition, and evaluation subset construction

"Keeping Experts in the Loop: Expert-Guided Optimization for Clinical Data Classification using Large Language Models" introduces SamplEase, an iterative sampling algorithm within StructEase that identifies high-value cases for prompt refinement [2412.02173]. Starting from an LLM-generated initial prompt, the system samples 10% of unlabeled cases, classifies them, computes confidence
$$
\mathrm{Confidence}=\exp\left(\frac{1}{n}\sum_{i=1}^{n}\log P_i\right),
$$
and, for each class, selects the 10 lowest-confidence examples for expert labeling. Mismatches between model and expert labels are then inserted as new few-shot examples in the next prompt. Macro F1 improved from 0.935 to 0.986 after two refinements, and smart sampling outperformed random expert labeling with F1 0.974 versus 0.959 [2412.02173].

"Active sampling: A machine-learning-assisted framework for finite population inference with optimal subsamples" treats sampling as adaptive importance sampling guided by surrogate predictions on unseen data [2212.10024]. If outcomes were known, the optimal probabilities satisfy
$$
\pi_i^* \propto \left|\nabla h(\mathbf{t}_{\mathbf{y}})^T \mathbf{y}_i\right|,
$$
while with predicted means and covariances the method defines
$$
c_i=\left[\nabla h(\mathbf{t}_{\mathbf{y}})^T \hat{\mathbf{y}}_i\right]^2+\nabla h(\mathbf{t}_{\mathbf{y}})^T \hat{\Sigma}_i \nabla h(\mathbf{t}_{\mathbf{y}}),
\qquad
\pi_i\propto \sqrt{c_i}.
$$
The inclusion of prediction variance prevents over-committing to unreliable predictions. In simulation-based safety assessment, the paper reports RMSE reductions of 20–50% and up to 46% fewer expensive simulations relative to simple random sampling and traditional importance sampling [2212.10024].

For human evaluation of NLG, "Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling" proposes CASF, combining a Learner, a Systematic Sampler, and a Constrained Controller to select representative evaluation subsets [2406.07967]. The Learner predicts sample quality from automatic metrics; the Systematic Sampler stratifies by predicted quality; the Constrained Controller minimizes redundancy under interval-preserving rules. Across 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks, CASF achieved 93.18% top-ranked system recognition accuracy, ranked first or second on 90.91% of the human metrics, and reached 0.83 overall inter-system ranking Kendall correlation [2406.07967].

These works share a common principle: annotation or evaluation budgets are treated as scarce resources, and expert-guided sampling is used to maximize marginal information gain rather than label volume alone.

## 5. Explainability-guided acquisition and disagreement-guided supervision

In medical few-shot learning, "Expert-Guided Explainable Few-Shot Learning for Medical Image Diagnosis" augments a prototypical network with radiologist-provided ROIs and a Grad-CAM-based explanation loss [2509.08007]. For Grad-CAM heatmap $G$ and expert mask $M$,
$$
\mathcal{L}_{\text{exp}}=1-\frac{2\cdot |G\cap M|}{|G|+|M|},
$$
and the total loss is
$$
\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{proto}}+\alpha\cdot \mathcal{L}_{\text{exp}}.
$$
The paper reports accuracy improvements from 77.09% to 83.61% on BraTS and from 54.33% to 73.29% on VinDr-CXR, with $\alpha=0.10$ described as empirically optimal [2509.08007].

"Expert-Guided Explainable Few-Shot Learning with Active Sample Selection for Medical Image Analysis" extends this logic into a closed-loop active-learning framework [2601.02409]. xGAL scores samples by predictive uncertainty and attention misalignment:
$$
\mathcal{H}(x)=-\sum_{k=1}^{N}p(y=k\mid x)\log p(y=k\mid x),
$$
$$
D_{\text{exp}}(x)=1-\frac{2\cdot |\mathrm{CAM}_{\hat{y}}(x)\cap \mathrm{ESM}(x)|}{|\mathrm{CAM}_{\hat{y}}(x)|+|\mathrm{ESM}(x)|},
$$
$$
\mathrm{Score}(x)=\lambda\cdot \mathcal{H}(x)+(1-\lambda)\cdot D_{\text{exp}}(x).
$$
On BraTS, VinDr-CXR, and SIIM-COVID-19, the reported accuracies are 92%, 76%, and 62%, respectively, and under severe data constraints xGAL achieves 76% accuracy with only 680 samples versus 57% for random sampling [2601.02409].

"EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentation" treats inter-rater variability itself as an expert guidance signal [2403.16594]. With multiple expert masks $\{\mathbf{M}_j\}_{j=1}^{Y}$, the ground-truth disagreement heatmap is
$$
\mathbf{H}=\mathrm{Var}(\mathbf{M}_1,\ldots,\mathbf{M}_Y),
$$
and model uncertainty is estimated by variance across segmentation heads,
$$
\hat{\mathbf{H}}=\mathrm{Var}(\hat{\mathbf{M}}_1,\ldots,\hat{\mathbf{M}}_{N_D}).
$$
A disagreement-guidance module aligns these with an RMSE loss, while random sampling of annotator-provided labels during training exposes the model to annotation variability. The method reports 55% and 23% improvement in correlation on average with expert disagreements at the image and pixel levels, better calibration, competitive segmentation performance, and only a single forward pass [2403.16594].

A common misconception is that expert-guided sampling in medical AI only means asking experts which images to label. These papers show that expert guidance can also operate on where the model should attend, which disagreements should be preserved, and which unlabeled cases should be prioritized because they are both uncertain and explanation-misaligned.

## 6. External expert models, quantization-time sampling, and recurrent limitations

"Expert Knowledge-Guided Decision Calibration for Accurate Fine-Grained Tree Species Classification" uses a large pre-trained vision foundation model, BioCLIP2, as a "Domain Expert" and consults it selectively through uncertainty-guided decision calibration [2601.16498]. The Local Prior Guided Knowledge Extraction Module uses CAM-based masks from the backbone to guide the domain expert toward discriminative regions, while the Uncertainty-Guided Decision Calibration Module fuses backbone and expert logits as
$$
\hat{z}=z_b+\lambda\cdot z_e.
$$
EKDC-Net is described as a lightweight plug-and-play module that improves backbone accuracy by 6.42% and precision by 11.46% using only 0.08M additional learnable parameters [2601.16498]. Although the paper centers on calibration rather than sampling per se, it exemplifies a closely related pattern: uncertainty determines when expert knowledge should influence the final decision.

In MoE compression, "MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance" frames calibration-set construction itself as an expert-guided sampling problem [2505.03804]. Expert-Balanced Self-Sampling optimizes for both data-model alignment and expert usage balance,
$$
\mathcal{D}^*=\arg\min_{\mathcal{D}}\left\{\operatorname{PPL}(\mathcal{M},\mathcal{D})\cdot \exp\left(\frac{\sigma(\mathcal{M},\mathcal{D})}{\tau}\right)\right\},
$$
while Affinity-Guided Quantization weights per-sample quantization error by token-expert affinity:
$$
\mathcal{L}(\hat{\mathbf{W}})=\sum_{i=1}^{n} c_i \cdot \|\mathbf{W}\mathbf{x}_i-\hat{\mathbf{W}}\mathbf{x}_i\|_F^2.
$$
The paper reports more than 10 points accuracy gain in HumanEval for DeepSeekMoE-16B under 4-bit quantization, together with over $3.2\times$ memory reduction and decoder speed above $1.2\times$ [2505.03804].

Several limitations recur across the literature. Expert quality bounds achievable performance in EGPO, and removing the CQL loss or intervention minimization damages stability and safety [2110.06831]. Extremely misleading priors can waste early queries in posterior-sampling Bayesian optimization [2002.11256]. In active finite-population inference, ignoring prediction uncertainty can lead to potential variance explosion [2212.10024]. In explainability-supervised few-shot learning, overweighting the explanation loss harms learning, whereas underweighting makes the alignment signal too weak [2509.08007]. Conversely, not all expert-guided methods require retraining: Expert-Sample is explicitly training-free and plug-and-play at inference time [2602.02443].

Taken together, these results suggest that expert-guided sampling is most effective when the guidance signal is both selective and uncertainty-aware: high-confidence structure is preserved, low-confidence regions are explored, and expert intervention is concentrated where purely data-driven procedures are most brittle.

Source: https://www.emergentmind.com/topics/expert-guided-sampling