---
title: Expert-Controlled CFG in Medical VQA
url: https://www.emergentmind.com/topics/expert-controlled-classifier-free-guidance-expert-cfg
type: topic
---

# Expert-Controlled CFG in Medical VQA

Searching arXiv for the cited Expert-CFG and related CFG papers.
Expert-Controlled Classifier-Free Guidance (Expert-CFG) is a training-free, expert-in-the-loop framework for aligning medical vision-language models (MedVLMs) with clinical expertise, particularly for medical visual question answering (MedVQA). It detects when a base MedVLM is uncertain, selectively invokes human expertise, retrieves supporting references, extracts or edits key medical highlights, and then applies a classifier-free-guidance formulation adapted to autoregressive generation so that token-level decoding is steered toward expert-approved content without changing model weights [2507.09209]. In its canonical usage, the term refers to this MedVLM reliability framework; a broader reading in adjacent CFG research suggests a more general design pattern in which guidance becomes an explicitly controllable layer rather than a single fixed scalar.

## 1. Conceptual definition and problem setting

Expert-CFG addresses two operational questions: **RQ1 – When is a MedVLM response unreliable?** and **RQ2 – How to inject expert knowledge efficiently without retraining?** The motivating setting is a medical assistant based on a 4.2B-parameter MedVLM that can answer questions about radiology or pathology images, yet remains vulnerable to probabilistic uncertainty, hallucinations, and misalignment with clinical expertise, especially for fine-grained medical details such as laterality, projection views, or subtle pathological patterns [2507.09209].

The framework is explicitly described as **training-free**: the MedVLM weights remain frozen; only token embeddings and attention scores are rescaled on-the-fly during inference. “Expert-Controlled” denotes that the conditioning signal is derived from human expert highlights, possibly assisted by GPT-4 keyword extraction on retrieved captions. “Classifier-Free” denotes that the method does not train a separate classifier; instead it constructs conditional and unconditional contexts from the same model and combines their outputs to obtain guided decoding [2507.09209].

A recurrent misconception is that Expert-CFG is simply diffusion-style CFG transplanted verbatim into language generation. In fact, the framework adapts CFG to autoregressive MedVLMs at the token-logit level and defines the two branches through expert-highlight masking in embeddings and attention, rather than through an external classifier or a separate conditional model [2507.09209].

## 2. Reliability gating through uncertainty estimation

The first stage is baseline MedVLM inference. A pre-trained 4.2B-parameter MedVLM, based on Phi-3-Vision or Phi-3.5-Vision and adapted to medicine via LoRA using 1.3M PubMedVision VQA samples, takes an image and question and generates an answer using greedy decoding. The second stage computes uncertainty from the resulting token distributions and routes only high-entropy cases into the expert-control path [2507.09209].

The uncertainty measure is sequence-level predictive entropy computed from token-level autoregressive distributions. For a generated sequence $\mathbf{x}=\{x_1,\dots,x_N\}$, the normalized predictive entropy is
\[
\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).
\]
Because longer sequences would otherwise have larger summed entropy, the framework normalizes by length $N$. High predictive entropy indicates many almost-equally probable tokens; low predictive entropy indicates concentrated distributions. The method uses a single greedy decoding run rather than Monte Carlo dropout or ensembles, and unreliable answers are selected either by a fixed threshold or by taking the top-$k\%$ highest-entropy outputs [2507.09209].

Empirically, entropy-based reliability detection achieves ROC-AUC greater than 0.8 on VQA-RAD, SLAKE, and PathVQA. The analysis further links high entropy to sparse training coverage, rare answers, unseen phrases such as abbreviations, and difficult answer types including size, degree, left/right, and numbers [2507.09209]. Expert load is controlled by this routing mechanism, and the reported gains saturate beyond roughly 5% of samples, which motivates the paper’s emphasis on selective intervention rather than exhaustive annotation [2507.09209].

## 3. Retrieval, reference grounding, and expert highlighting

Once an answer is flagged as uncertain, Expert-CFG retrieves supporting material from a multimodal medical corpus and converts it into a sparse expert signal. The knowledge base is built from all training QA pairs from VQA-RAD, SLAKE, and PathVQA, rephrased as captions and excluding negative answers, together with ROCO (70k radiology image–text pairs) and PEIR (33k pathology image–text pairs). Retrieval uses BioMedCLIP encoders and FAISS for Top-K search [2507.09209].

After retrieval, GPT-4 is used twice: first to extract medical entities or keywords from each retrieved caption, and then to match those keywords to the query context and select relevant key terms. Experts review and optionally edit those terms. The final annotation is therefore a sparse set of clinically meaningful words or phrases such as “fat stranding,” “posterior-anterior (PA) view,” or “left lower lobe consolidation.” In open-ended VQA experiments, the ground-truth answer string itself is often used as the highlight; in closed yes/no questions, entity extraction from retrieved text plus question matching provides highlight candidates [2507.09209].

The highlights are encoded as a binary token mask
\[
\mathbf{m}=\{m_1,\dots,m_N\},
\]
where $m_i=1$ if token $x_i$ belongs to the expert highlight set and $m_i=0$ otherwise. This mask mediates both branches of the subsequent CFG procedure. A second misconception follows from retrieval-augmented generation practice: retrieved references alone do not guarantee factual alignment. The reported case studies show that Expert-RAG can still leave conflicts between internal and external knowledge unresolved, whereas Expert-CFG uses the highlights to directly reshape generation dynamics [2507.09209].

## 4. Token-level classifier-free guidance in autoregressive MedVLMs

The core CFG mechanism is defined at the token level. Let $P_{\Theta}(x_i \mid x_{<i}, \mathbf{k})$ denote the conditional next-token probability given expert annotations $\mathbf{k}$, and let $P_{\Theta}(x_i \mid x_{<i})$ denote the corresponding unconditional branch. The guided sequence distribution is
\[
\hat{P}_{\Theta}(\mathbf{x} \mid \mathbf{k}) \propto P_{\Theta}(\mathbf{x} \mid \mathbf{k})^{\gamma} \, P_{\Theta}(\mathbf{x})^{\gamma-1}
\propto \prod_{i=1}^{N} P_{\Theta}(x_i \mid x_{<i}, \mathbf{k})^{\gamma} \, P_{\Theta}(x_i \mid x_{<i})^{\gamma - 1},
\]
which yields the guided token logit
\[
\mathbb{P}(x_i) = \gamma \log P_{\Theta}(x_i \mid x_{<i}, \mathbf{k}) - (\gamma - 1) \log P_{\Theta}(x_i \mid x_{<i}).
\]
The reported guidance strength is $\gamma=1.5$ [2507.09209].

The conditional and unconditional branches are not created by different prompts alone, but by different embedding and attention treatments of the highlight mask. If $f(\cdot)$ is the input embedding layer and $c_i=f(x_i)$ is the normal conditional context, then the unconditional context rescales highlighted tokens as
\[
\bar{c}_i = (\alpha - 1) m_i \cdot f(x_i) + f(x_i).
\]
Thus, if $m_i=1$, then $\bar{c}_i=\alpha f(x_i)$, while non-highlight tokens are unchanged. The paper uses $\alpha=0.01$, so the unconditional branch largely ignores highlighted tokens while preserving global context [2507.09209].

In the normal branch, self-attention is also reweighted toward highlighted tokens. If $e_i$ is a raw attention score, the modified activation is
\[
h_i = \log(\beta) \cdot m_i + e_i,
\]
which induces
\[
p_i = \frac{\exp(h_i)}{\sum_{j=1}^N \exp(h_j)}
= \frac{\beta^{m_i} \exp(e_i)}{\sum_{j=1}^N \beta^{m_j} \exp(e_j)}.
\]
For highlighted tokens, attention is multiplied by $\beta$, with $\beta=3$ in the reported configuration. In the unconditional branch, attention to highlighted tokens is suppressed by a scaled negative mask $-\delta \mathbf{m}$, with $\delta$ chosen so that $\log(\beta)+2$, ensuring that deactivation overwhelms the positive highlight bias [2507.09209].

At inference time, each generation step therefore consists of constructing a normal branch and an unconditional branch, obtaining two next-token distributions, combining them through the guided logit formula, and decoding greedily. No retraining is performed; the intervention acts only through token embeddings, attention scores, and logit recombination [2507.09209].

## 5. Benchmarks, quantitative performance, and qualitative behavior

Expert-CFG is evaluated on three MedVQA benchmarks: VQA-RAD with 3,515 QA pairs and 315 radiology images, SLAKE with 7,000 QA pairs and 642 images across multiple modalities, and PathVQA with 32,799 QA pairs and 4,998 pathology images. Closed questions are evaluated by accuracy, while open questions are evaluated by recall of ground-truth tokens within generated answers. The compared variants include baseline MedVLMs, naive RAG, 5% RAG, 5% Expert-RAG, and 5% Expert-CFG, where expert-augmented methods are applied only to the top 5% highest-entropy samples unless otherwise stated [2507.09209].

| Model | Baseline overall accuracy | 5% Expert-CFG overall accuracy |
|---|---|---|
| Phi3.5V-Med 4.2B | VQA-RAD 82.04; SLAKE 85.20; PathVQA 65.24 | VQA-RAD 84.70; SLAKE 87.75; PathVQA 69.56 |
| Phi3V-Med 4.2B | VQA-RAD 79.38; SLAKE 83.98; PathVQA 64.43 | VQA-RAD 81.82; SLAKE 86.33; PathVQA 68.91 |

These results show consistent gains of +2.66, +2.55, and +4.32 for Phi3.5V-Med, and +2.44, +2.35, and +4.48 for Phi3V-Med across the three datasets. The paper further reports that on SLAKE closed-set, BioMed-VITAL 13B attains 90.70, while Phi3.5V-Med + Expert-CFG 4.2B reaches 91.59, indicating that a smaller model with limited expert annotations can match or exceed some 13B-scale MedVLM baselines [2507.09209].

Qualitatively, the framework is presented as addressing conflicts between internal model beliefs and external references. Retrieval-only conditioning provides limited or inconsistent gains, and retrieval hit rate tops out around 50–56% even at Top-4. By contrast, Expert-CFG is reported to raise token probabilities on clinically critical phrases and to produce correct and more complete answers in cases where the base model omitted or misdiagnosed findings such as “fat stranding” [2507.09209]. This supports the paper’s narrower claim that the method aligns internal generation with expert-highlighted phrases rather than merely appending evidence to the context.

## 6. Relation to the broader literature on controllable CFG

A broader reading of recent CFG work suggests that Expert-CFG also names a wider control paradigm in which guidance is decomposed into interpretable levers. In diffusion models, $\beta$-CFG replaces a constant guidance scale with a time- and state-dependent effective scale,
\[
s(t,x_t)=\beta(t)\cdot\omega\,\|\epsilon_c(x_t)-\epsilon_\varnothing(x_t)\|^{-\gamma},
\]
combining a Beta-distribution time window with gradient-based adaptive normalization; the reported effect is better FID while maintaining text-to-image CLIP similarity at a level similar to reference CFG [2502.10574]. This suggests one axis of “expert control” through schedule shape, normalization exponent, and global scale.

A second line of work turns guidance selection into online control. Dynamic CFG via online feedback evaluates candidate guidance scales at each reverse step using latent-space experts such as CLIP, a discriminator, a human preference reward model, a text rendering expert, and a numerical reasoning expert, then greedily selects the best scale. On Imagen 3, this yields up to 53.8% human preference win-rate for overall preference and up to 55.5% on prompts targeting text rendering [2509.16131]. A plausible implication is that Expert-CFG can be understood not only as human-in-the-loop highlighting, but also as expert-in-the-loop scheduling when “expert” denotes a learned evaluator.

Theoretical and training-free control results point in a similar direction. C$^2$FG derives an exponential guidance schedule from strict upper bounds on conditional–unconditional score discrepancy, while VAGS modulates the nominal flow guidance scale by a bounded factor that combines denoising stage with cosine similarity between relevant velocity fields, requiring no extra forward passes [2603.08155] [2605.15661]. Stage-wise analysis further argues that early strong guidance erodes global diversity and late strong guidance suppresses fine-grained variation, providing a formal rationale for time-varying control [2509.22007]. These adjacent results do not define Expert-CFG in the same way as the MedVLM paper, but they suggest a coherent research theme: classifier-free guidance increasingly functions as a controllable interface rather than a fixed heuristic.

## 7. Limitations, assumptions, and future directions

The most explicit limitation is expert dependency. Expert-CFG requires expert annotations or highlight validation, even if only for a small fraction of cases. Semi-automated GPT-4 keyword extraction reduces the burden but does not remove the need for clinician review. A second limitation is knowledge-base coverage: retrieval hit rate is limited, especially for numeric or attribute-based questions, and CFG cannot fix missing concepts if neither the base model nor the knowledge base contains them [2507.09209].

The empirical scope is also narrow. The reported experiments focus on short-answer VQA rather than longer medical reports, and the paper notes that greedy decoding may be sub-optimal for long sequences while entropy estimation for long texts remains underexplored. Although radiology and pathology are covered, additional evaluation is still needed across more subspecialties and modalities. The framework also assumes access to a reasonably competent base MedVLM and to clinicians capable of validating or editing keyword highlights [2507.09209].

The stated future directions are correspondingly concrete: extending Expert-CFG to longer outputs and multi-turn clinical dialogues, combining it with automated expert models such as radiology classifiers or segmentation models to generate pseudo-expert highlights at scale, improving uncertainty estimation for multi-sentence answers, and broadening evaluation to trustworthiness and calibration benchmarks such as CARES and MultiMedEval [2507.09209]. Beyond medicine, the same paper proposes analogous use in legal analysis, scientific QA, and safety-critical engineering QA, where experts validate key constraints or factual anchors before inference-time guidance is applied [2507.09209].

Source: https://www.emergentmind.com/topics/expert-controlled-classifier-free-guidance-expert-cfg