RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology

Abstract: Cancer screening is a reasoning task. A radiologist observes findings, compares them to prior scans, integrates clinical context, and reaches a diagnostic conclusion confirmed by pathology. We present RadThinking, a Visual Question Answering (VQA) dataset that makes this reasoning explicit and trainable. RadThinking releases VQA pairs at three difficulty tiers. Foundation VQAs are atomic perception questions. Single-step reasoning VQAs apply one clinical rule. Compositional VQAs require multi-step chain-of-thought to reach a guideline category such as LI-RADS-5. For every compositional VQA, we release the chain of foundation VQAs that solves it. The chain follows the rules of the governing clinical reporting standard. The dataset spans 20,362 CT scans from 9,131 patients across 43 cancer groups, plus 2,077 verified healthy controls with >1-year follow-up. To our knowledge, RadThinking is the first cancer-screening VQA corpus that stratifies questions by reasoning depth and grounds compositions in clinical reporting standards. The foundation tier supplies atomic perception supervision. The compositional tier supplies chain-of-thought data and verifiable rewards for reinforcement-learning recipes such as DeepSeek-R1 and OpenAI o1. RadThinking enables systematic training and evaluation of whether AI systems can reason about cancer, not merely detect it.

• The paper introduces RadThinking as the first VQA dataset for longitudinal clinical reasoning in radiology, integrating temporal scans and clinical variables.
• It categorizes VQA pairs by reasoning depth, from perceptual to multi-step compositional questions aligned with clinical standards.
• Robust expert validation and high annotation accuracy establish its potential for advancing AI in cancer screening tasks.

RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology

Motivation and Background

The RadThinking dataset addresses a critical deficiency in publicly available radiology benchmarks: the near-exclusive focus on visual perception rather than clinical reasoning. Conventional CT datasets provide spatial tumor annotations but omit the structured interpretive process that underpins expert diagnosis, namely, the integration of temporal change, clinical context, and structured reporting standards. In clinical practice, radiologists synthesize observations from longitudinal imaging, correlate findings against patient history, and apply rigorous guideline-based reasoning, yet AI models trained on perception-centric datasets lack explicit supervision or evaluation along this interpretive axis.

RadThinking is proposed as a comprehensive Visual Question Answering (VQA) corpus that operationalizes the clinical reasoning workflow. By incorporating longitudinal scan sequences, structured clinical variables, and adherence to reporting standards, it enables the training and evaluation of AI systems on reasoning tasks that extend beyond visual detection. This dataset is presented as the first VQA resource in cancer screening to stratify questions according to reasoning depth and to ground multi-step compositions in established clinical grammars.

Figure 1: Left: six timepoints from a hepatocellular carcinoma case demonstrate the evolution of reasoning complexity and reasoning steps per scan. Right: RadThinking dataset characteristics, including CT scan and patient coverage.

Dataset Structure and Content

Data Curation and Cohort

RadThinking aggregates data from abdominal, pelvic, and thoracic CT scans across 10 European hospitals spanning 2012–2025 with rigorous IRB and ethics governance. The cohort consists of over 9,000 patients and nearly 20,400 scans (exact numbers redacted in draft), with $>$1-year follow-up for healthy controls and confirmed pathology for cancer cases. Each patient record contains a longitudinal trajectory of scans, with associated radiology reports, extracted clinical variables, and voxel-wise tumor masks over 19 organ screening targets, 12 of which lack prior public CT tumor annotations.

Figure 1's right panel outlines the dataset size and organ coverage.

VQA Structure: Reasoning Tiers

RadThinking VQA pairs are categorized at three reasoning depths:

• Foundation Tier: Atomic perception questions, e.g., lesion size, present/absent, scan modality. Answers are directly accessible from one field, providing supervision for visual primitives.
• Single-Step Reasoning Tier: Questions requiring one explicit clinical rule applied to a foundation answer, such as applying size thresholds or determining eligibility for reporting standards.
• Compositional Tier: Multi-step chain-of-thought questions that require sequential aggregation and synthesis of observation, temporal comparison, and clinical context according to a standard's explicit rules (e.g., LI-RADS, RECIST, TNM). Each compositional VQA is decomposed into a chain of foundation VQAs, with the determinative logic strictly following the protocol of the governing standard.

Box 1 in the manuscript gives an explicit example for a complex LI-RADS classification, delineating the stepwise sequence from atomic observations through to guideline-driven conclusion.

Multimodal Artifacts and Annotations

Each patient-level JSON record encapsulates: scan metadata, tumor masks, foundation and compositional VQA chains, de-identified clinical text, structured clinical variables, and pathology labels for positive cases. Annotation follows a multi-stage process: initial labeling by 28 radiologist residents, dual independent board-certified review, and adjudication for discrepancies. Inter-annotator Dice for spatial annotation is 62.2%. Feature extraction, temporal label agreement, and report parsing accuracy metrics are all $\geq$94.6%.

Compositional Reasoning via Clinical Standards

A distinguishing methodological feature is the formalization of clinical reporting standards as compositional grammars. For each organ, standards such as LI-RADS (liver), PI-RADS (prostate), BI-RADS (breast), and Bosniak (renal cysts) dictate the observable atomic features and the deterministic rules linking them to risk categories or recommendations (see Table~\ref{tab:standards} in the manuscript). The dataset strictly follows these grammars when structuring VQA decompositions. This approach ensures the compositional reasoning tasks are directly aligned with clinical decision-making rather than mere synthetic logical puzzles.

Cohort & Cancer-Type Stratification

RadThinking captures a broad spectrum of oncologic complexity across 43 unified cancer groups, derived from consolidating 55 raw clinical labels using expert-defined mappings. As depicted in Figure 2, the distribution is heavily long-tailed, with five cancer categories accounting for over half the cancer-positive cohort. Notably, a substantial subset comprises rare cancers and extra-organ primaries identified by metastatic deposits, requiring cross-modality and cross-organ reasoning.

Figure 2: The cancer-type distribution highlights the imbalance and clinical heterogeneity characteristic of population-level screening cohorts, emphasizing the dataset's coverage of both prevalent and rare cancer groups.

Longitudinal cases are prevalent: 36.3% of cancer-positive patients have 2–26 scans each, facilitating the modeling of disease progression and temporal reasoning tasks that mirror real-world diagnostic challenges.

Reasoning Complexity and Validation

The majority of cases demand reasoning that transcends image-level perception: 39.2% are classified as integrative (requiring multi-source synthesis), and 36.4% as ambiguous (entailing genuine inter-radiologist uncertainty). Only 12.9% are labeled as perceptual (i.e., decisive from a single scan). This distribution validates the dataset's focus on the "hard" territory of clinical decision-making.

Integral to the resource is its explicit annotation of reasoning complexity—perceptual, temporal, integrative, or ambiguous—based on reproducible rules reflecting information demands and radiologist agreement. Eight-radiologist validation yields high reliability ($\kappa = 0.947$).

Model Training and Evaluation Paradigm

RadThinking is positioned for supervised and reinforced training (SFT/RL) of vision-LLMs (VLMs), supplying:

• Several hundred thousand VQA pairs at foundation and compositional difficulty.
• Grounded multimodal chain-of-thought (CoT) supervision, aligned with contemporary curricula in VLM pretraining and fine-tuning stacks.
• Verifiable reward axes for RL, including direct pathology match, organ-level malignancy/metastasis, risk-category accuracy, temporal change correctness, and enforced output structure.

The dataset is unique in providing explicit, pathology-anchored reward signals and multi-tiered complexity labels for robust benchmark stratification—facilitating a granular diagnosis of model failure modes.

Implications and Future Prospects

Practical Implications: RadThinking enables the systematic development and evaluation of VLMs capable of multi-step, guideline-grounded clinical reasoning—a necessary capability for reliable deployment in cancer screening and longitudinal disease monitoring. By structuring the question landscape across atomic to compositional tiers, it provides a natural curriculum for curriculum learning strategies, multi-task transfer, and RL from clinical standards.

Theoretical Implications: The dataset operationalizes the concept of hierarchical, standard-grounded reasoning in a real-world, high-complexity domain. It sets a precedent for large-scale, structure-based medical datasets that go beyond perception. Explicit chain-of-thought annotations offer unique opportunities for mechanistic interpretability, error tracing, and analysis of reasoning generalization in deep VLMs.

Prospective Directions may include:

• Cross-modal generalization to other imaging modalities (MR, mammography, ultrasound) as clinical standards exist for these domains.
• Uncertainty quantification research leveraging the ambiguous complexity category.
• Benchmarking continual learning and lifelong AI systems on longitudinal real-world disease trajectories.
• Studying the transferability of chain-of-thought inductive biases from medicine to other domains requiring compositional and temporal reasoning.

Conclusion

RadThinking introduces a structured, multimodal VQA corpus for longitudinal cancer screening, distinguished by its explicit modeling of radiological reasoning chains under clinical standards. The dataset’s breadth across organs, cancers, and reasoning depths, combined with rigorous expert validation and reward signal design, addresses a significant limitation of current radiology AI benchmarks. RadThinking is positioned as a reference dataset for future research in clinical reasoning, interpretable vision-language AI, and curriculum-driven medical model development.