DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

Abstract: Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revise a differential diagnosis, and stop when the diagnosis is sufficiently supported. Most medical AI benchmarks instead reveal the relevant context upfront and score only the final answer, making unsupported correct guesses, premature closure, inefficient workups, and poor uncertainty updating invisible. We introduce DDX-TRACE, a physician-adjudicated benchmark for multimodal neuroradiology that evaluates diagnostic trajectories under hidden evidence over 211 challenging cases. Each case begins with limited clinical history; models request imaging studies in free form, receive matched image bundles when available, update a probabilistic differential diagnosis after each turn, and stop with a localized final diagnosis. Evaluating state-of-the-art VLMs, we find that final diagnosis scores can substantially misrepresent workup quality: models may guess plausible diagnoses without essential evidence, request useful studies but misinterpret raw images, or acquire evidence inefficiently while updating uncertainty poorly. Controlled evidence variants isolate bottlenecks in planning, visual evidence extraction, and downstream differential reasoning. DDX-TRACE shifts medical AI evaluation from final answers to evidence-supported diagnostic trajectories.

• The paper demonstrates that existing VLMs can generate correct diagnoses but often rely on unsupportive workups lacking essential evidence.
• It introduces a sequential, process-aware evaluation where models request imaging studies, update differential diagnoses, and undergo detailed route and endpoint analysis.
• Results reveal significant gaps between endpoint accuracy and clinically sufficient workups, highlighting bottlenecks in active evidence acquisition and uncertainty handling.

DDx-TRACE: Advancing Process-Aware Evaluation for Medical Diagnostic Trajectories in VLMs

Motivation and Benchmark Design

Conventional medical AI benchmarks typically assess models based on their ability to answer clinical questions from fully specified prompts—patient history, imaging, and findings are provided upfront, and only the final answer is scored. This paradigm fails to capture several critical aspects of clinical practice: unsupported correct guesses, premature closure, inefficient diagnostics, and inadequate belief updating remain invisible. DDx-TRACE is proposed as a solution: an annotated benchmark for neuroradiology where models must actively request imaging studies, sequentially update probabilistic differential diagnoses, and finalize with a localized diagnosis, all from an initially limited clinical history. This reframing aims to measure not just outcome correctness, but the route taken, evidence acquisition, and uncertainty updating.

Figure 1: Conventional benchmarks reveal evidence upfront and score only endpoint answers; DDx-TRACE requires sequential evidence acquisition with intermediate probabilistic updates.

Each DDx-TRACE case starts with clinical history only; models must dynamically request imaging, receive image bundles, update a four-item differential list with probabilities after each turn, and eventually "stop" with a localized diagnosis. Evaluation encompasses endpoint correctness, route quality (essential-evidence recall, workup order, unmatched request rate), efficiency, and confidence metrics, providing a granular view of diagnostic trajectories beyond surface-level accuracy.

Figure 2: Schema: Model receives only clinical history, requests imaging in free-form, updates differential after each turn, stops with a localized diagnosis. Benchmark evaluates both endpoint and route metrics.

Benchmark Construction and Clinical Annotation

DDx-TRACE leverages 211 EuroRad-derived neuroradiology cases (191 retained in the official split) with 785 requestable imaging bundles and 1,609 images spanning 28 modalities. Each case is annotated by at least two board-certified physicians. Annotations include:

• Step importance: labeling exams as essential, optional, unnecessary
• Preferred order: partial order over imaging exams
• Case rarity/difficulty: physician-assessed
• Metadata correction: modality, acquisition, anatomical region, contrast
• Rubrics: specific diagnostic and localization criteria

Disagreements (e.g., essential vs. optional) are adjudicated by senior physicians. These labels underpin process-aware evaluation of models, ensuring not only endpoint correctness but clinical sufficiency of the diagnostic route.

Process-Aware Evaluation Protocol

DDx-TRACE formalizes diagnosis as a sequential evidence acquisition problem: models receive only limited history at $t=0$. At each turn, models request one exam in natural language or stop; the resolver returns matched bundles if available. After each new evidence, a four-item differential (with probability) is updated. At stop, the model outputs a final diagnosis and localization answer.

Evaluation metrics include:

• Endpoint metrics: $S_{\mathrm{dx}}$ (diagnosis), $S_{\mathrm{loc}}$ (localization), $S_{\mathrm{ddx}}$ (differential-list quality)
• Route metrics: essential evidence recall ($S_{\mathrm{ER}}$), optional burden ($B_{\mathrm{opt}}$), unmatched request rate ($B_{\mathrm{unm}}$), order concordance ($S_{\mathrm{order}}$)
• Efficiency: time to diagnostic guess ($T_{\mathrm{guess}}$), time to clinically supported diagnosis ($T_{\mathrm{clin}}$), average number of requested exams, stopping behavior
• Confidence metrics: probability mass on exact/acceptable/unmatched diagnoses ($S_{\mathrm{dx}}$0, $S_{\mathrm{dx}}$1)

These metrics collectively ensure that correct diagnoses unsupported by acquired evidence are not equivalently scored as evidence-supported workups.

Figure 3: A correct endpoint diagnosis may arise from an insufficient diagnostic route: evidence acquisition, ordering, and clinical sufficiency can diverge from endpoint credit.

Experimental Analysis: Diagnostic Trajectories Versus Endpoint Accuracy

Thirteen VLMs (frontier, open-weight, medical/radiology-adapted) were benchmarked. Core findings include:

• Endpoint accuracy and process quality select different leaders: Gemini 3.1 Pro and GPT-5.4 achieve highest diagnosis scores ($S_{\mathrm{dx}}$2), but Gemini 3 Flash dominates process metrics (essential recall, order concordance, confidence alignment). Thus, final answer accuracy does not imply clinically sufficient workups.
• Passive evidence settings obscure acquisition/planning failures: When models are given all images at once or oracle findings, scores increase substantially, demonstrating that active acquisition, evidence planning, and image understanding are significant bottlenecks.

Figure 4: Passive evaluation (all-images-at-once, oracle findings) improves $S_{\mathrm{dx}}$3, hiding failures in acquisition/planning that DDx-TRACE exposes.

Figure 5: Comparison of passive endpoint score ($S_{\mathrm{dx}}$4) with active trajectory score ($S_{\mathrm{dx}}$5) reveals a benchmark gap between endpoint accuracy and workup quality.

• Correct diagnostic guesses are rarely supported by essential evidence: Even the best models achieve clinically sufficient diagnosis (correct guess + all essential evidence acquired) in only ~30% of cases.

Diagnostic Trajectory and Efficiency

Sequential evidence acquisition improves diagnosis for frontier models; open-weight and domain-specialized VLMs produce smaller or unstable gains. Some models request more exams but do not translate evidence into better diagnosis, while others stop early without sufficient workup.

Figure 6: Diagnosis trajectory: Frontier models' scores improve as evidence is acquired; others plateau.

Efficiency–accuracy tradeoff is explicit: models on the Pareto frontier (Gemini 3.1 Pro, GPT-5.4 Mini) achieve highest accuracy with more requests, but stopping early can lead to unsupportive diagnoses.

Figure 7: Pareto frontier: Diagnostic accuracy ($S_{\mathrm{dx}}$6) versus number of requests, illustrating tradeoff.

Slice analysis (rarity/difficulty) shows weak dependence—performance bottlenecks are more attributable to planning and evidence acquisition rather than case difficulty.

Figure 8: Model performance remains stable across physician-rated rarity and difficulty slices; bottlenecks arise from evidence acquisition/planning limitations.

Confidence Alignment and Calibration

Reliability diagrams reveal frontier models (Gemini 3 Flash, Gemini 3.1 Pro) best align confidence with diagnosis quality, whereas open-weight and radiology-adapted models tend to allocate high probability to unmatched or implausible alternatives.

Figure 9: Calibration analysis: Frontier models have the strongest confidence–diagnosis alignment; domain-specialized models are poorly calibrated.

Benchmark Composition and Annotation Interface

DDx-TRACE consists of 191 evaluation cases, 811 imaging examinations, and 1,609 images. Annotation is performed via a web-based platform supporting evidence labeling, exam order specification, difficulty/rarity assessment, and metadata/rubric correction.

Figure 10: Distribution of official cases across pathology/tag, rarity/difficulty, exam/contrast attributes.

Figure 11: Distribution of imaging evidence and benchmark characteristics: case origin, patient age, modality, exam/image counts.

Figure 12: Annotation interface: Physicians label exam importance, order, rarity, difficulty, and correct metadata/rubric artifacts.

Implications and Future Directions

DDx-TRACE demonstrates that existing state-of-the-art VLMs are capable of plausible diagnosis, but lack clinically sufficient workup strategies: correct answers often arise without proper evidence acquisition or belief updating. Major bottlenecks include:

• Active evidence acquisition: Planning, requesting studies, and sequencing
• Visual evidence extraction: Correctly interpreting raw images
• Uncertainty-aware reasoning: Updating differential diagnoses and stopping only when sufficient support exists

The practical implication is that deployment-ready medical AI cannot rely on endpoint benchmarks; process-aware evaluation is crucial. DDx-TRACE encourages model development toward active acquisition, evidence integration, and reliable uncertainty handling.

Future expansions should generalize beyond neuroradiology, incorporate full imaging studies and other specialities, and strengthen the dataset for training and evaluation. Process-aware metrics are essential to characterize clinical AI limitations and ensure trustworthy diagnostic agents.

Conclusion

DDx-TRACE reframes medical AI benchmarking as process-aware evaluation of diagnostic trajectories under partial information, not just endpoint answer accuracy. Model assessment incorporates acquisition route, evidence sufficiency, order, efficiency, and calibration, exposing limitations invisible to traditional benchmarks. Results indicate substantial gaps between plausible diagnosis and clinically supported workups across existing VLMs. This benchmark sets a new standard for evaluating and guiding development of reliable, clinically relevant multimodal diagnostic agents.