MedQA-Followup: Interactive Medical QA

Updated 7 July 2026

The paper introduces a novel interactive QA framework that leverages iterative follow-up questioning to bridge a 22.3% accuracy gap over static models.
MediQ decomposes patient cases into a minimal initial intake and a hidden fact set, using modular components like question generation and confidence calibration for decision-making.
Empirical findings show that calibrated abstention and structured dialogue effectively mitigate premature answers and improve diagnostic performance.

MedQA-Followup refers to MedQA-derived settings in which medical question answering is no longer treated as a static, fully specified multiple-choice task, but as an interactive process of clarifying incomplete cases through follow-up questions. In MediQ, which introduced the canonical interactive reformulation, the Expert System initially receives only a minimal intake $k_0$ —age, gender, and chief complaint—while a Patient System holds the full record $K=\{k_0,k_1,\dots,k_n\}$ and returns record-grounded answers to atomic questions. The Expert must decide whether to abstain, ask a follow-up question, or commit to the original MedQA option. Later work used MedQA-derived follow-up settings to study structured state tracking, information-seeking diagnosis, and multi-turn robustness under misleading context (Li et al., 2024, Meng et al., 3 Oct 2025, Wong et al., 19 Feb 2026, Manczak et al., 14 Oct 2025).

1. Origins in MedQA and the shift from static to interactive reasoning

The MedQA benchmark was introduced as a large-scale free-form multiple-choice OpenQA dataset from medical exams, with an English USMLE split of 10,178 train, 1,272 dev, and 1,273 test questions. In a manual sample of 100 USMLE dev questions, 98.0% were classified as Type 2 patient case studies, rather than single-knowledge-point items, which made MedQA a natural substrate for case-based follow-up reformulation (Jin et al., 2020). In its original form, however, MedQA supplies the entire vignette before answering.

MediQ identifies the central limitation of that setup: most medical QA benchmarks provide all necessary details in one paragraph, whereas real clinical encounters begin with incomplete information, and safe diagnosis requires iterative elicitation. The paper argues that LLMs are trained to answer any question even with incomplete context or insufficient knowledge, which leads to hallucinations and premature decisions in interactive settings (Li et al., 2024). A later survey places MediQ in the “Incomplete Information Background” category, in contrast to MedQA’s “Precise Information Background”; this suggests that MedQA-Followup is not primarily a new answer space, but a change in the assumptions about what information is initially visible and how it must be acquired (Xiao et al., 22 Oct 2025).

2. MediQ and the construction of iMedQA

MediQ converts single-turn medical benchmarks into interactive tasks by splitting each original MedQA record into a visible intake $k_0$ and a hidden fact set $K$ . The Expert sees the MedQA multiple-choice question and only $k_0$ , while the Patient sees the full record. At turn $t$ , the Expert either asks an atomic follow-up question $q_t$ or answers; if it asks, the Patient returns $r_t \subseteq K$ , and the knowledge state is updated as

$K_0=\{k_0\}, \qquad K_t=K_{t-1}\cup r_t.$

The target is to expand $K_t$ until it approximates the sufficient subset $K=\{k_0,k_1,\dots,k_n\}$ 0 and then answer accurately (Li et al., 2024).

The Expert System is organized around five modules: Initial Assessment, Abstention, Question Generation, Information Integration, and Decision. The Initial Assessment identifies likely diagnoses and knowledge gaps from $K=\{k_0,k_1,\dots,k_n\}$ 1 and the MCQ. The Abstention module decides whether confidence is sufficient to answer. If not, Question Generation produces one atomic, medically grounded follow-up about missing evidence such as additional symptoms, physical examination, labs, medications, or social history. Information Integration appends the resulting question-answer pair to the conversation log, after which the system re-evaluates and eventually emits FINAL CHOICE: <letter> (Li et al., 2024).

Patient simulation is a critical design choice. MediQ evaluates three Patient variants—Direct, Instruct, and Fact-Select—and uses Fact-Select for the benchmark because it decomposes the record into atomic facts and selects only facts relevant to the Expert’s atomic question. On MedQA dev automated metrics, Direct achieved Factuality 55.9 and Relevance 75.5, Instruct 62.8 and 78.6, and Fact-Select 89.1 and 79.9. The Patient is instructed to refrain with a standardized response such as “The patient cannot answer this question…” when the requested fact is unavailable, making unanswerable and repeated turns analyzable post hoc (Li et al., 2024).

3. Abstention, confidence estimation, and evaluation protocol

MedQA-Followup operationalizes reliability through abstention. MediQ studies three decision rules for ask-versus-answer control: numerical confidence $K=\{k_0,k_1,\dots,k_n\}$ 2, binary confidence $K=\{k_0,k_1,\dots,k_n\}$ 3, and a Likert-style scale $K=\{k_0,k_1,\dots,k_n\}$ 4. In the scale formulation,

$K=\{k_0,k_1,\dots,k_n\}$ 5

The system further combines rationale generation (RG), in which the model first explains its confidence judgment, with self-consistency (SC), in which the abstention prompt is repeated $K=\{k_0,k_1,\dots,k_n\}$ 6 times and aggregated by averaging or mode. In Scale abstention, RG made confidence more conservative and better calibrated, reducing Expected Calibration Error from 2.9 to 2.1 (Li et al., 2024).

Evaluation is not limited to final MCQ accuracy. MediQ reports diagnostic accuracy, interaction efficiency measured by the number of follow-up questions, abstention rate as a proxy for information-seeking behavior, coverage, and robustness to irrelevant or noisy turns. It also defines an empirical full-information upper bound from the original static MedQA paragraph and a gap-closure metric,

$K=\{k_0,k_1,\dots,k_n\}$ 7

This measures how much of the performance gap between the incomplete-information baseline and the full-information condition is recovered by interactive questioning (Li et al., 2024).

The same framework formalizes Patient quality with factuality and relevance metrics and allows robustness analysis under filtered versus unfiltered dialogue histories. This makes MedQA-Followup simultaneously a reasoning benchmark, a confidence-estimation problem, and a memory-management problem, since the same collected evidence can help or hurt depending on how it is stored and re-presented to the model (Li et al., 2024).

4. Empirical findings on iMedQA

The central empirical result of MediQ is negative in one sense and constructive in another: directly prompting strong LLMs to ask follow-up questions often degrades performance, but calibrated abstention policies partly recover the lost accuracy (Li et al., 2024).

Model	Full / Initial / Basic interactive / Best abstention
GPT-4	79.7% ± 1.1 / 54.5% ± 1.4 / 55.4% ± 1.3 / 66.1% ± 1.3
GPT-3.5	55.8% ± 1.4 / 45.6% ± 1.4 / 42.2% ± 1.3 / 50.2% ± 1.4
Llama-3-70B	84.7% ± 1.0 / 58.5% ± 1.4 / 55.1% ± 1.3 / 60.9% ± 1.4

Across iMedQA, the best interactive system—Scale+RG+SC—improved accuracy by 22.3% over the Basic interactive baseline and by 12.1% over the non-interactive Initial condition, while closing 51.2% of the Initial→Full gap. Even so, a 10.3% gap to the full-information upper bound remained. On iCraft-MD, a dermatology benchmark converted in the same way, GPT-4 Best reached 84.3% ± 3.1 compared with Full 91.4% ± 2.4 and Initial 67.9% ± 3.9, showing a similar pattern (Li et al., 2024).

The failure analysis is equally important. Basic interactive systems often ask zero follow-ups and answer prematurely. Repeated or unanswerable follow-up questions degrade performance by distracting the Expert. Filtering unanswerable questions and repeated questions each added +2 percentage points, and combined filtering plus paragraph reformatting (Unique-Para) yielded +5.7 percentage points over Basic and +2.3 points over the non-interactive Initial baseline. MediQ also reports that dialogue format itself can hinder evidence use relative to concise paragraph summaries, indicating that the bottleneck is not only what information is acquired but how it is represented for downstream reasoning (Li et al., 2024).

5. Structured and knowledge-guided successors

Later MedQA-Followup systems retained the interactive premise but replaced flat dialogue history with more explicit diagnostic state.

System	Mechanism	Reported effect
TriMediQ	UMLS-style triplets, patient-specific knowledge graph, graph encoder, projector, frozen expert LLM	up to 10.4% improvement in accuracy over five baselines on iMedQA (Meng et al., 3 Oct 2025)
MedClarify	differential diagnosis of size $K=\{k_0,k_1,\dots,k_n\}$ 8, question selection by diagnostic expected information gain	reduces diagnostic errors by ~27 percentage points compared to a standard single-shot LLM baseline (Wong et al., 19 Feb 2026)
KG-Followup	PrimeKG augmentation with active in-context learning	outperforms state-of-the-art methods by 5%–8% in recall (Sun et al., 1 Mar 2026)

TriMediQ addresses a specific weakness of MediQ-style systems: raw dialogue logs contain facts in sentence form without explicit links. It converts patient responses into clinically relevant triplets, incrementally builds a knowledge graph $K=\{k_0,k_1,\dots,k_n\}$ 9, encodes the graph into $k_0$ 0, mean-pools to $k_0$ 1, and projects this into a prefix $k_0$ 2 that conditions a frozen expert LLM. On iMedQA, Llama-3-70B improved from 60.9 to 71.3 and Qwen-2.5-72B from 61.5 to 71.4; on iCraft-MD, the same pattern held, and the advantage widened with dialogue depth (Meng et al., 3 Oct 2025).

MedClarify replaces abstention-threshold heuristics with an explicitly information-theoretic policy. It first constructs a differential diagnosis, then generates questions designed to refute the current top diagnosis while supporting alternatives, and ranks them by

$k_0$ 3

with $k_0$ 4, $k_0$ 5, and $k_0$ 6. The system performs Bayesian-style belief updates across turns, applies temperature scaling with $k_0$ 7, and achieved an ECE of 0.16 versus 0.37 for a naive multi-turn baseline. Under joint masking of case features, it improved over the naive multi-turn baseline by +8.1 percentage points on NEJM, +4.9 on MediQ, and +4.1 on MedQA, for an aggregated +5.7 points; the abstract summarizes the overall effect as reducing diagnostic errors by ~27 percentage points versus a standard single-shot baseline (Wong et al., 19 Feb 2026).

KG-Followup targets pre-diagnostic follow-up generation rather than final diagnosis selection. It links extracted entities from the patient-doctor conversation to PrimeKG, builds intersected subgraphs through breadth-first search, samples shortest reasoning paths between source entities and candidate diagnoses, and uses active in-context learning from KG-informed hard cases. On ClinicalInquiryBench, Claude Sonnet with Active ICL reached 0.77 recall with 20 questions, and on FollowupBench it reached 0.82 recall with 40 questions. The paper emphasizes recall because valid clinical follow-ups may not match a single reference wording (Sun et al., 1 Mar 2026).

A parallel extension shifts attention from question planning to evidence verification. MedTrust-Guided Iterative RAG combines citation-aware reasoning, Medical Gap Analysis, and Negative Knowledge Assertions for cases with inadequate support. On MedQA, Qwen3-8B improved from 68.9 with standard RAG to 70.1 with DPO-aligned verification, while LLaMA3.1-8B improved from 61.6 for PostAttr to 63.1 with the SFT variant (Ning et al., 16 Oct 2025).

6. Robustness, realism, and the limits of MedQA-derived follow-up evaluation

A later paper reuses the name MedQA-Followup for a different but related purpose: systematic evaluation of multi-turn robustness under controlled interventions. That framework distinguishes shallow robustness from deep robustness, and indirect contextual framing from direct suggestion. Across five models, the paper finds that shallow robustness can look acceptable while deep robustness is severely degraded. The most striking result is that Claude Sonnet 4 drops from 91.2% baseline accuracy to 13.5% under RAG-style misleading context, and indirect context-based interventions are often more harmful than direct suggestions. In compounding experiments, 85% of combinations were sub-additive rather than super-additive, indicating bounded but still serious exploitability (Manczak et al., 14 Oct 2025).

Real patient threads expose a different limitation: even interactive MedQA-like benchmarks remain stylized. ThReadMed-QA, built from 2,437 fully answered r/AskDocs threads with 8,204 question-answer pairs, evaluates five state-of-the-art LLMs on authentic multi-turn patient follow-ups. Even GPT-5 achieves only 41.2% fully-correct responses overall, all models degrade significantly from turn 0 to turn 2 with $k_0$ 8, wrong-answer rates roughly triple by the third turn, and Error Propagation Rate shows that a single wrong turn raises the probability of a subsequent wrong turn by 1.9–6.1x. This suggests that MedQA-derived follow-up tasks still understate the instability that emerges in open, patient-authored conversations (Munnangi et al., 11 Mar 2026).

The broader evaluation landscape also weakens any equation of medical QA with MedQA-like factual recall. MedArena reports that only about one-third of clinician-submitted questions resembled MedQA-style factual recall, while approximately 20% of interactions were multi-turn, and clinicians cited Depth and Detail and Presentation and Clarity more often than raw factual accuracy when explaining preferences. MedQA-CS reaches the same conclusion from an OSCE-inspired direction: models that can exceed 90% on USMLE-style MCQ still score far lower on instruction-following clinical skills, with strong systems typically averaging about 48–62 across InfoGatherQA, Physical Exams, Closure, and Differential Diagnosis (Wu et al., 13 Mar 2026, Yao et al., 2024).

These results sharpen the limitations already noted in MediQ. Only MedQA and Craft-MD had sufficiently detailed records for conversion in that work; the benchmark remains multiple-choice; the Patient System depended on a paid API; and cost and latency increase with question count, rationale generation, and self-consistency. MediQ explicitly states that these systems are for benchmarking, not deployment, and that abstention cannot replace clinical judgment. In that sense, MedQA-Followup is best understood as a controlled laboratory for studying when medical LLMs should ask, what they should ask, how evidence should be accumulated across turns, and how fragile those decisions remain under realistic conversational pressure (Li et al., 2024).