Expert-Controlled CFG in Medical VQA

• Expert-CFG is a training-free framework that intercepts uncertain MedVLM responses and integrates expert-verified highlights to steer token-level generation.
• It employs entropy-based reliability detection and dynamic reweighting of embeddings and attention, achieving ROC-AUC over 0.8 on key MedVQA benchmarks.
• The method enhances answer accuracy by aligning internal model outputs with clinical evidence without modifying the underlying model weights.

Searching arXiv for the cited Expert-CFG and related CFG papers. Expert-Controlled Classifier-Free Guidance (Expert-CFG) is a training-free, expert-in-the-loop framework for aligning medical vision-LLMs (MedVLMs) with clinical expertise, particularly for medical visual question answering (MedVQA). It detects when a base MedVLM is uncertain, selectively invokes human expertise, retrieves supporting references, extracts or edits key medical highlights, and then applies a classifier-free-guidance formulation adapted to autoregressive generation so that token-level decoding is steered toward expert-approved content without changing model weights (Liang et al., 12 Jul 2025). In its canonical usage, the term refers to this MedVLM reliability framework; a broader reading in adjacent CFG research suggests a more general design pattern in which guidance becomes an explicitly controllable layer rather than a single fixed scalar.

1. Conceptual definition and problem setting

Expert-CFG addresses two operational questions: RQ1 – When is a MedVLM response unreliable? and RQ2 – How to inject expert knowledge efficiently without retraining? The motivating setting is a medical assistant based on a 4.2B-parameter MedVLM that can answer questions about radiology or pathology images, yet remains vulnerable to probabilistic uncertainty, hallucinations, and misalignment with clinical expertise, especially for fine-grained medical details such as laterality, projection views, or subtle pathological patterns (Liang et al., 12 Jul 2025).

The framework is explicitly described as training-free: the MedVLM weights remain frozen; only token embeddings and attention scores are rescaled on-the-fly during inference. “Expert-Controlled” denotes that the conditioning signal is derived from human expert highlights, possibly assisted by GPT-4 keyword extraction on retrieved captions. “Classifier-Free” denotes that the method does not train a separate classifier; instead it constructs conditional and unconditional contexts from the same model and combines their outputs to obtain guided decoding (Liang et al., 12 Jul 2025).

A recurrent misconception is that Expert-CFG is simply diffusion-style CFG transplanted verbatim into language generation. In fact, the framework adapts CFG to autoregressive MedVLMs at the token-logit level and defines the two branches through expert-highlight masking in embeddings and attention, rather than through an external classifier or a separate conditional model (Liang et al., 12 Jul 2025).

2. Reliability gating through uncertainty estimation

The first stage is baseline MedVLM inference. A pre-trained 4.2B-parameter MedVLM, based on Phi-3-Vision or Phi-3.5-Vision and adapted to medicine via LoRA using 1.3M PubMedVision VQA samples, takes an image and question and generates an answer using greedy decoding. The second stage computes uncertainty from the resulting token distributions and routes only high-entropy cases into the expert-control path (Liang et al., 12 Jul 2025).

The uncertainty measure is sequence-level predictive entropy computed from token-level autoregressive distributions. For a generated sequence $\mathbf{x}=\{x_1,\dots,x_N\}$, the normalized predictive entropy is

$$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$

Because longer sequences would otherwise have larger summed entropy, the framework normalizes by length $N$. High predictive entropy indicates many almost-equally probable tokens; low predictive entropy indicates concentrated distributions. The method uses a single greedy decoding run rather than Monte Carlo dropout or ensembles, and unreliable answers are selected either by a fixed threshold or by taking the top-$k\%$ highest-entropy outputs (Liang et al., 12 Jul 2025).

Empirically, entropy-based reliability detection achieves ROC-AUC greater than 0.8 on VQA-RAD, SLAKE, and PathVQA. The analysis further links high entropy to sparse training coverage, rare answers, unseen phrases such as abbreviations, and difficult answer types including size, degree, left/right, and numbers (Liang et al., 12 Jul 2025). Expert load is controlled by this routing mechanism, and the reported gains saturate beyond roughly 5% of samples, which motivates the paper’s emphasis on selective intervention rather than exhaustive annotation (Liang et al., 12 Jul 2025).

3. Retrieval, reference grounding, and expert highlighting

Once an answer is flagged as uncertain, Expert-CFG retrieves supporting material from a multimodal medical corpus and converts it into a sparse expert signal. The knowledge base is built from all training QA pairs from VQA-RAD, SLAKE, and PathVQA, rephrased as captions and excluding negative answers, together with ROCO (70k radiology image–text pairs) and PEIR (33k pathology image–text pairs). Retrieval uses BioMedCLIP encoders and FAISS for Top-K search (Liang et al., 12 Jul 2025).

After retrieval, GPT-4 is used twice: first to extract medical entities or keywords from each retrieved caption, and then to match those keywords to the query context and select relevant key terms. Experts review and optionally edit those terms. The final annotation is therefore a sparse set of clinically meaningful words or phrases such as “fat stranding,” “posterior-anterior (PA) view,” or “left lower lobe consolidation.” In open-ended VQA experiments, the ground-truth answer string itself is often used as the highlight; in closed yes/no questions, entity extraction from retrieved text plus question matching provides highlight candidates (Liang et al., 12 Jul 2025).

The highlights are encoded as a binary token mask

$\mathbf{m}=\{m_1,\dots,m_N\},$

where $m_i=1$ if token $x_i$ belongs to the expert highlight set and $m_i=0$ otherwise. This mask mediates both branches of the subsequent CFG procedure. A second misconception follows from retrieval-augmented generation practice: retrieved references alone do not guarantee factual alignment. The reported case studies show that Expert-RAG can still leave conflicts between internal and external knowledge unresolved, whereas Expert-CFG uses the highlights to directly reshape generation dynamics (Liang et al., 12 Jul 2025).

4. Token-level classifier-free guidance in autoregressive MedVLMs

The core CFG mechanism is defined at the token level. Let $P_{\Theta}(x_i \mid x_{<i}, \mathbf{k})$ denote the conditional next-token probability given expert annotations $\mathbf{k}$, and let $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$0 denote the corresponding unconditional branch. The guided sequence distribution is

$$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$1

which yields the guided token logit

$$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$2

The reported guidance strength is $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$3 (Liang et al., 12 Jul 2025).

The conditional and unconditional branches are not created by different prompts alone, but by different embedding and attention treatments of the highlight mask. If $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$4 is the input embedding layer and $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$5 is the normal conditional context, then the unconditional context rescales highlighted tokens as

$$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$6

Thus, if $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$7, then $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$8, while non-highlight tokens are unchanged. The paper uses $$\text{PE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{x_i} P_{\Theta}(x_i \mid \mathbf{x}_{<i}) \log P_{\Theta}(x_i \mid \mathbf{x}_{<i}).$$9, so the unconditional branch largely ignores highlighted tokens while preserving global context (Liang et al., 12 Jul 2025).

In the normal branch, self-attention is also reweighted toward highlighted tokens. If $N$0 is a raw attention score, the modified activation is

$N$1

which induces

$N$2

For highlighted tokens, attention is multiplied by $N$3, with $N$4 in the reported configuration. In the unconditional branch, attention to highlighted tokens is suppressed by a scaled negative mask $N$5, with $N$6 chosen so that $N$7, ensuring that deactivation overwhelms the positive highlight bias (Liang et al., 12 Jul 2025).

At inference time, each generation step therefore consists of constructing a normal branch and an unconditional branch, obtaining two next-token distributions, combining them through the guided logit formula, and decoding greedily. No retraining is performed; the intervention acts only through token embeddings, attention scores, and logit recombination (Liang et al., 12 Jul 2025).

5. Benchmarks, quantitative performance, and qualitative behavior

Expert-CFG is evaluated on three MedVQA benchmarks: VQA-RAD with 3,515 QA pairs and 315 radiology images, SLAKE with 7,000 QA pairs and 642 images across multiple modalities, and PathVQA with 32,799 QA pairs and 4,998 pathology images. Closed questions are evaluated by accuracy, while open questions are evaluated by recall of ground-truth tokens within generated answers. The compared variants include baseline MedVLMs, naive RAG, 5% RAG, 5% Expert-RAG, and 5% Expert-CFG, where expert-augmented methods are applied only to the top 5% highest-entropy samples unless otherwise stated (Liang et al., 12 Jul 2025).

Model	Baseline overall accuracy	5% Expert-CFG overall accuracy
Phi3.5V-Med 4.2B	VQA-RAD 82.04; SLAKE 85.20; PathVQA 65.24	VQA-RAD 84.70; SLAKE 87.75; PathVQA 69.56
Phi3V-Med 4.2B	VQA-RAD 79.38; SLAKE 83.98; PathVQA 64.43	VQA-RAD 81.82; SLAKE 86.33; PathVQA 68.91

These results show consistent gains of +2.66, +2.55, and +4.32 for Phi3.5V-Med, and +2.44, +2.35, and +4.48 for Phi3V-Med across the three datasets. The paper further reports that on SLAKE closed-set, BioMed-VITAL 13B attains 90.70, while Phi3.5V-Med + Expert-CFG 4.2B reaches 91.59, indicating that a smaller model with limited expert annotations can match or exceed some 13B-scale MedVLM baselines (Liang et al., 12 Jul 2025).

Qualitatively, the framework is presented as addressing conflicts between internal model beliefs and external references. Retrieval-only conditioning provides limited or inconsistent gains, and retrieval hit rate tops out around 50–56% even at Top-4. By contrast, Expert-CFG is reported to raise token probabilities on clinically critical phrases and to produce correct and more complete answers in cases where the base model omitted or misdiagnosed findings such as “fat stranding” (Liang et al., 12 Jul 2025). This supports the paper’s narrower claim that the method aligns internal generation with expert-highlighted phrases rather than merely appending evidence to the context.

6. Relation to the broader literature on controllable CFG

A broader reading of recent CFG work suggests that Expert-CFG also names a wider control paradigm in which guidance is decomposed into interpretable levers. In diffusion models, $N$8-CFG replaces a constant guidance scale with a time- and state-dependent effective scale,

$N$9

combining a Beta-distribution time window with gradient-based adaptive normalization; the reported effect is better FID while maintaining text-to-image CLIP similarity at a level similar to reference CFG (Malarz et al., 14 Feb 2025). This suggests one axis of “expert control” through schedule shape, normalization exponent, and global scale.

A second line of work turns guidance selection into online control. Dynamic CFG via online feedback evaluates candidate guidance scales at each reverse step using latent-space experts such as CLIP, a discriminator, a human preference reward model, a text rendering expert, and a numerical reasoning expert, then greedily selects the best scale. On Imagen 3, this yields up to 53.8% human preference win-rate for overall preference and up to 55.5% on prompts targeting text rendering (Papalampidi et al., 19 Sep 2025). A plausible implication is that Expert-CFG can be understood not only as human-in-the-loop highlighting, but also as expert-in-the-loop scheduling when “expert” denotes a learned evaluator.

Theoretical and training-free control results point in a similar direction. C$k\%$0FG derives an exponential guidance schedule from strict upper bounds on conditional–unconditional score discrepancy, while VAGS modulates the nominal flow guidance scale by a bounded factor that combines denoising stage with cosine similarity between relevant velocity fields, requiring no extra forward passes (Gao et al., 9 Mar 2026, Luo et al., 15 May 2026). Stage-wise analysis further argues that early strong guidance erodes global diversity and late strong guidance suppresses fine-grained variation, providing a formal rationale for time-varying control (Jin et al., 26 Sep 2025). These adjacent results do not define Expert-CFG in the same way as the MedVLM paper, but they suggest a coherent research theme: classifier-free guidance increasingly functions as a controllable interface rather than a fixed heuristic.

7. Limitations, assumptions, and future directions

The most explicit limitation is expert dependency. Expert-CFG requires expert annotations or highlight validation, even if only for a small fraction of cases. Semi-automated GPT-4 keyword extraction reduces the burden but does not remove the need for clinician review. A second limitation is knowledge-base coverage: retrieval hit rate is limited, especially for numeric or attribute-based questions, and CFG cannot fix missing concepts if neither the base model nor the knowledge base contains them (Liang et al., 12 Jul 2025).

The empirical scope is also narrow. The reported experiments focus on short-answer VQA rather than longer medical reports, and the paper notes that greedy decoding may be sub-optimal for long sequences while entropy estimation for long texts remains underexplored. Although radiology and pathology are covered, additional evaluation is still needed across more subspecialties and modalities. The framework also assumes access to a reasonably competent base MedVLM and to clinicians capable of validating or editing keyword highlights (Liang et al., 12 Jul 2025).

The stated future directions are correspondingly concrete: extending Expert-CFG to longer outputs and multi-turn clinical dialogues, combining it with automated expert models such as radiology classifiers or segmentation models to generate pseudo-expert highlights at scale, improving uncertainty estimation for multi-sentence answers, and broadening evaluation to trustworthiness and calibration benchmarks such as CARES and MultiMedEval (Liang et al., 12 Jul 2025). Beyond medicine, the same paper proposes analogous use in legal analysis, scientific QA, and safety-critical engineering QA, where experts validate key constraints or factual anchors before inference-time guidance is applied (Liang et al., 12 Jul 2025).