---
title: 'DeepTumorVQA: CT Tumor Diagnosis Benchmark'
url: https://www.emergentmind.com/topics/deeptumorvqa
type: topic
---

# DeepTumorVQA: CT Tumor Diagnosis Benchmark

DeepTumorVQA is a tumor-centric medical visual question answering benchmark for volumetric computed tomography that was introduced to test whether vision-language models can progress from lesion recognition to clinically grounded diagnostic reasoning. In its 2025 presentation, it was described as a diagnostic VQA benchmark for abdominal tumors in CT scans, comprising 9,262 CT volumes, 3.7M slices, and 395K expert-level question-answer pairs across four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning [2505.18915]. In its 2026 formulation, DeepTumorVQA was recast as a hierarchical 3D CT benchmark for stage-wise evaluation of medical VLMs and tool-augmented agents, with 476K questions across 42 clinical subtypes on the same 9,262 3D CT volumes, plus explicit tool-interaction environments and ground-truth tool-use traces [2605.09679]. The benchmark’s defining premise is that tumor diagnosis is a multi-stage evidence chain, and that evaluation should localize failures at the level of perception, quantification, compositional reasoning, or clinical inference rather than obscure them within a single aggregate score.

## 1. Historical emergence and conceptual rationale

DeepTumorVQA was proposed against a background in which medical VQA benchmarks were characterized as mostly 2D, small, or overly simplistic, and therefore poorly matched to real radiological workflows for volumetric abdominal CT [2505.18915]. The 2025 benchmark asked a direct question: whether current vision-language models are precise and intelligent enough for clinical diagnosis from 3D CT, especially in tumor-centric settings requiring small lesion recognition, numerical measurement, cross-slice integration, and application of medical rules.

The 2026 hierarchical benchmark sharpened that motivation. Its authors argued that existing evaluations collapse capability into a single accuracy score, even though a model may recognize lesions but fail to measure them, may measure correctly but fail in downstream reasoning, may answer from priors without genuine visual grounding, or may fail specifically in tool use rather than in clinical interpretation [2605.09679]. DeepTumorVQA therefore frames diagnosis as a staged progression from lower-level perceptual primitives to higher-level clinical conclusions.

This design aligns with a broader methodological shift in medical multimodal research. Rather than treating VQA as generic image-text fusion, DeepTumorVQA treats oncologic QA as an evidence-tracing problem in which the validity of a high-level answer depends on access to lower-level measurable facts. This suggests that the benchmark is intended not merely as a leaderboard instrument, but as a failure-analysis framework for clinically oriented VLMs and agents.

## 2. Dataset composition and benchmark construction

Across its 2025 and 2026 descriptions, DeepTumorVQA is built from **9,262 3D CT volumes** drawn from **17 public abdominal CT datasets** spanning **88 medical centers** [2505.18915]. The 2025 paper reports **395K expert-level question-answer pairs**, with a split of **355,962 training QA pairs from 8,334 CT** and **39,650 testing QA pairs from 928 CT**. The 2026 hierarchical version reports **476K questions total**, including **428K train** and **48K test**, and evaluates models on a **balanced 10,000-question subset** from the test split, stratified across **42 subtypes** and spanning **991 CT volumes** [2605.09679].

The source corpora include abdominal datasets such as CHAOS, Pancreas-CT, BTCV, LiTS, CT-ORG, WORD, AMOS22, KiTS, the Medical Segmentation Decathlon CT tasks, AbdomenCT-1K, FLARE’23, and the RSNA 2023 Abdominal Trauma Detection dataset, among others [2505.18915]. The 2025 description reports **7,629 lesions annotated by 23 board-certified radiologists**, distributed across **3,067 liver**, **4,078 kidney**, **351 pancreatic**, and **131 colon** lesions. The 2026 benchmark reports **11,319 lesions**, **43 anatomical structures per volume**, and annotations by **23 radiologists** [2605.09679].

Question generation is structured rather than free-form. The 2025 paper describes a modular, template-based process inspired by CLEVR-style compositional programs, in which structured metadata are extracted from segmentation masks and reports and then converted into benchmark questions [2505.18915]. The 2026 benchmark states that subtype-specific deterministic programs generate QA pairs from **70+ structured attributes** per scan, including organ volumes, lesion volumes, mean HU values, lesion counts, lesion diameters, cross-organ ratios, and staging fields [2605.09679]. Higher-level answers are therefore programmatically grounded in measurable primitives rather than authored as unconstrained natural-language prompts.

A concise summary of the 2026 stage structure is given below.

| Stage | Subtypes | Representative tasks |
|---|---:|---|
| Recognition | 9 | liver lesion existence, kidney cyst existence, PDAC existence, splenomegaly detection |
| Measurement | 5 | organ volume, organ HU, lesion volume, organ HU ratio, tumor burden percentage |
| Visual reasoning | 16 | lesion counting, largest lesion diameter, lesion location, kidney volume comparison |
| Medical reasoning | 12 | fatty liver, pancreatic steatosis, portal hypertension, PDAC vs PNET, pancreatic T staging |

The discrepancy between the **29 subtypes** reported in 2025 and the **42 clinical subtypes** reported in 2026 is itself informative. It suggests a later broadening of subtype coverage and a more explicit formalization of the benchmark’s evidence hierarchy.

## 3. Evidence hierarchy and clinical task semantics

DeepTumorVQA is organized around four reasoning stages: **Recognition**, **Measurement**, **Visual Reasoning**, and **Medical Reasoning** [2505.18915] [2605.09679]. The benchmark’s central innovation is that higher-level questions remain independently scorable as ordinary VQA items, while their ground-truth evidence chains are defined over lower-level primitives.

Recognition tests lesion or abnormality existence. In the 2025 version, it includes questions such as whether the liver, kidney, pancreas, or colon contains lesions, tumors, or cysts [2505.18915]. In the 2026 hierarchy, recognition includes organ, lesion, and pathological finding detection, with splenomegaly detection explicitly noted as sitting near the recognition/measurement boundary because it depends on a quantitative threshold [2605.09679].

Measurement asks the model to extract numerical evidence from CT or metadata. Reported measurement tasks include organ volume, organ mean Hounsfield Unit, lesion volume, organ HU ratio, and tumor burden percentage [2505.18915] [2605.09679]. The 2025 paper evaluates numerical free-text answers with **mean relative accuracy**, whereas the 2026 benchmark also supports multiple-choice evaluation and tool-based retrieval of exact quantities.

Visual reasoning composes perceptual and quantitative primitives. Representative tasks include lesion counting, largest lesion diameter, lesion location, lesion attenuation, left-vs-right kidney volume comparison, organ aggregation, lesion outlier detection, adjacency, organ enlargement, bilateral asymmetry, and multi-organ burden comparison [2505.18915] [2605.09679]. These tasks require integrating multiple measurements or spatial relations rather than reading out a single attribute.

Medical reasoning applies medical rules or external clinical knowledge to image-derived evidence. Examples include fatty liver diagnosis, hepatic steatosis grading, pancreatic steatosis, pancreatic cyst resectability, pancreatic lesion resectability, pancreatic tumor staging, PDAC vs PNET, portal hypertension, renal mass characterization, pseudocyst determination, and splenomegaly grading [2505.18915] [2605.09679]. The 2025 appendix gives explicit task logic for several subtypes, including `lesion outlier` as whether the largest lesion is more than **3×** the second largest, `pancreatic cyst resectability` as a binary classification based on cyst volume \(> 3.0\ \mathrm{cm}^3\), and `pancreatic steatosis` using a pancreas/spleen HU ratio with the rule \(<0.7 = Yes\) [2505.18915].

The benchmark’s evidence-chain formalism is most apparent in questions such as fatty liver, where the final answer is clinically categorical but the ground truth depends on lower-level measurements such as liver HU and spleen HU [2605.09679]. This structure makes it possible to distinguish a clinically correct answer supported by correct intermediate evidence from a correct final answer produced for the wrong reason.

## 4. Evaluation protocols, metrics, and tool-interaction environments

The 2025 benchmark uses **task-specific metrics**: **accuracy** for multiple-choice questions, **exact match** for free-text categorical answers, and **mean relative accuracy (MRA)** for numerical free-text outputs, with the note that free-text numerical subtypes are evaluated using MRA and that higher is better [2505.18915]. The 2026 hierarchical benchmark evaluates direct inference on **10,000 fixed multiple-choice questions** with greedy decoding and reports answer accuracy by subtype, task type, and overall performance, alongside latency and trajectory statistics for tool-using agents [2605.09679].

The 2026 paper defines MRA for free-form numeric scoring as
$$
\mathrm{MRA}(\hat{y}, y) = \max\!\Big(0,\; 1 - \tfrac{|\hat{y}-y|}{|y|}\Big),
$$
with special handling when \(y = 0\) [2605.09679]. For agent trajectories, it reports tool-set Jaccard,
$$
\frac{|GT \cap Actual|}{|GT \cup Actual|},
$$
as well as parameter accuracy, average steps, and valid prediction rate.

A major extension in the 2026 benchmark is the introduction of **tool-interaction environments** in a ReAct-style loop of up to **8 steps** [2605.09679]. The available tools are:

- **`segment_organ(target)`**: returns segmentation statistics for **43 targets**.
- **`measure(target, type)`**: returns values such as **volume**, **mean HU**, **diameter**, and **count**.
- **`lookup_medical_knowledge(query)`**: returns criteria from a curated **27-entry knowledge base**.
- **`crop_region(organ)`**: returns organ-focused crop images in vision mode.

The tools may return oracle quantities from expert masks, predicted quantities from **TotalSegmentator**, or crop-based visual outputs [2605.09679]. DeepTumorVQA also provides deterministic **ground-truth tool-use traces** for all **42 subtypes**, implemented with regex-style parameter matching. This makes it possible to evaluate not only whether an agent answered correctly, but also whether it selected the right tools, invoked them in the right order, avoided redundant loops, and completed the intended evidence chain.

This tool-augmented setting materially changes the meaning of benchmark success. Direct reasoning evaluates raw multimodal inference; agent evaluation additionally measures planning, quantification access, and the translation of tool outputs into clinically valid decisions.

## 5. Empirical results and the measurement bottleneck

The 2025 DeepTumorVQA evaluation benchmarks **RadFM**, **M3D**, **Merlin**, and **CT-CHAT**, with two M3D language-backbone variants, and finds a consistent performance ordering: **measurement tasks are easiest**, **recognition and reasoning are harder**, and **medical reasoning is hardest** [2505.18915]. RadFM is reported as the strongest model, with **0.812 recognition accuracy in free-text**, **0.629 medical reasoning accuracy in free-text**, and the highest **total average free-text score of 0.555**; in multiple-choice evaluation, RadFM also leads with a **total average of 0.662** [2505.18915]. The same paper concludes that current VLMs are still not meeting clinical needs, especially for small tumor detection, nuanced lesion localization, 3D integration, and clinically grounded reasoning.

Several finer-grained findings from the 2025 study are notable. Lesion size alone is reported as **not** being a reliable predictor of recognition, whereas **HU contrast** is: for RadFM, sensitivity rises with larger lesion-to-organ HU difference, while size trends are inconsistent for liver and pancreatic lesions [2505.18915]. The paper also shows that anatomical preprocessing can dominate nominal architecture differences: wrapping models with **nnU-Net-based organ localization** yields major recognition gains, and **nnM3D-LLaMA2** improves kidney tumor sensitivity from **0%** to **80.9%**, surpassing RadFM on that task [2505.18915].

The 2026 hierarchical benchmark reframes these findings as a stage-wise bottleneck analysis. Across zero-shot 2D models, **Recognition** is reported at about **50–58%**, whereas **Measurement** drops to about **24–32%**, near chance [2605.09679]. Pretrained 3D specialists without task-specific fine-tuning are described as surprisingly weak, while after fine-tuning the **best 2D LoRA model reaches 66.3%** and the **best 3D fine-tuned model reaches 59.8%** [2605.09679]. Two radiologists evaluated on a subset achieve **45.0%** and **54.5%**, with the paper noting that exact quantitative measurement from raw CT is difficult even for humans [2605.09679].

The dominant conclusion of the 2026 benchmark is explicit: **reliable quantitative measurement is the primary bottleneck** [2605.09679]. This is not a marginal observation. It is presented as the mechanism through which failures propagate upward: once measurement collapses, both visual reasoning and medical reasoning degrade because their prerequisite evidence is unavailable or unreliable.

## 6. Tool augmentation, transfer, and broader significance

Tool augmentation in DeepTumorVQA substantially alters the performance profile of current systems. When oracle tools are introduced, **Measurement** accuracy rises from roughly **29–31%** to **64–65%**, and **Medical reasoning** rises from roughly **27–28%** to **51–54%** [2605.09679]. With predicted tools, the paper reports that **predicted mode retains about 93% of the oracle-to-direct gain overall**, indicating that imperfect but structured quantification can still unlock much of the latent reasoning capability of VLMs [2605.09679]. By contrast, crop-only visual tools can make performance worse than direct inference because models hallucinate numeric estimates from organ crops rather than accessing quantitative evidence directly.

The same paper shows that ground-truth tool traces can supervise agents. **Meissa-4B**, trained on about **20K tool traces**, improves **overall accuracy from 46.0% to 63.8%**, raises **Measurement** accuracy to **78.3%**, and reduces the step-limit hit rate from about **17–18%** to **0.2–0.3%** [2605.09679]. This result indicates that the benchmark is not only evaluative but also instructional: its traces can function as supervision for tool orchestration and evidence-grounded reasoning.

DeepTumorVQA has also become a reference point within tumor-VQA research. The tumor-centric reasoning framework **TumorChain** reports explicit zero-shot generalization on DeepTumorVQA, achieving **73.30** in recognition, **53.31** in visual reasoning, **45.93** in medical reasoning, and **57.51** average, which the paper attributes to interleaved chain-of-thought, ROI-guided local evidence extraction, and auxiliary abnormality supervision [2603.05867]. The phrase **“DeepTumorVQA-style benchmark”** is used directly in **UCSF-PDGM-VQA**, a brain-tumor MRI benchmark built from real 3D multi-sequence studies, indicating that DeepTumorVQA has come to denote a broader design pattern: clinically grounded, volumetric, tumor-centered VQA with meaningful reasoning demands [2605.17140].

Related benchmarks also clarify the confounds that DeepTumorVQA attempts to expose. The contamination-controlled oncology VQA benchmark of 2026 shows that visual reliance is highly dataset-specific: liver questions genuinely require the image, whereas **Lung CT** can be essentially solvable without it, with blind accuracy matching or exceeding sighted accuracy in some settings [2606.02809]. In brain tumor MRI, UCSF-PDGM-VQA reports **modality collapse**, including cases in which multimodal systems perform similarly or better with a blank image and strong option-order bias in the original generated dataset [2605.17140]. These findings do not arise from DeepTumorVQA itself, but they sharpen the significance of its hierarchical evidence-chain design: aggregate accuracy alone is not a trustworthy proxy for genuine visual reasoning.

DeepTumorVQA’s broader importance therefore lies in its redefinition of medical VQA evaluation. It treats tumor diagnosis as a structured pipeline from recognition to quantification to compositional visual reasoning to medical rule application, then extends that pipeline into explicit tool use and trace supervision [2605.09679]. A plausible implication is that the benchmark’s enduring contribution is methodological rather than merely numeric: it provides a concrete framework for separating failures of seeing, measuring, reasoning, and acting in 3D oncologic AI systems.

Source: https://www.emergentmind.com/topics/deeptumorvqa