RSNA-RadioQA: Diagnostic QA Benchmark

• RSNA-RadioQA is a radiology question-answering benchmark designed to evaluate diagnostic reasoning through clinical vignettes and text summaries of imaging findings.
• It isolates diagnostic inference by using curated, text-only data derived from RSNA case collections, enabling focused evaluation of language model reasoning.
• The framework supports retrieval-augmented methods and agentic pipelines that improve diagnostic accuracy and generate clinically useful, evidence-based reasoning artifacts.

RSNA-RadioQA denotes a radiology question-answering benchmark and, more broadly, an RSNA-aligned evaluation paradigm in which a model receives a clinical vignette and a textual description of imaging findings and is expected to return the most likely diagnosis or a closely related radiologic concept. In the original formulation, it was built from 80 questions derived from the RSNA Case Collection; later work reused those 80 questions together with a 24-question ExtendedQA set to form a 104-item composite benchmark for retrieval-based and agentic evaluation (Arasteh et al., 2024, Wind et al., 1 Aug 2025). The concept sits within a wider RSNA ecosystem that includes expert-validated multimodal benchmarks, radiology report evaluation frameworks, image-grounded detection/localization models, and workflow-oriented quality-assurance systems (Wei et al., 21 Jan 2026, Shi et al., 13 Jun 2026, Wismueller et al., 2022).

1. Historical emergence and conceptual scope

A direct precursor to RSNA-RadioQA appeared at the 2016 Scientific Assembly and Annual Meeting of the Radiological Society of North America, where a live demonstration presented a radiology multiple-choice question-answering system that accepted both a radiology question and a medical image, then exposed a cognitive workflow involving text analysis, image analysis, and reasoning. That system, referred to as “Eyes of Watson,” classified questions into differential diagnosis versus patient management, extracted concepts from the vignette, ranked imaging pipelines using DICOM metadata and text similarity, and visualized its inference path with Sankey diagrams (Kanjaria et al., 2020). The emphasis was not only answer generation, but also whether machine reasoning was similar enough to that of radiologists to be trusted as assistance.

The later RSNA-RadioQA benchmark narrowed the problem into a reproducible diagnostic QA task for LLMs. The defining move was to decouple radiologic reasoning from direct image perception: the benchmark supplied clinical history and textualized imaging findings, but not image pixels, and required a single concise answer. This design made RSNA-RadioQA especially suitable for testing whether LLMs could use radiology knowledge, differential reasoning, and external retrieval under controlled conditions (Arasteh et al., 2024).

Within current literature, the term also has a broader infrastructural meaning. It refers not only to the 80-question benchmark itself, but to an RSNA-centered family of tasks spanning diagnostic QA, image-based evidence grounding, report-based evaluation, and clinical QA workflows. This broader reading is explicit in later work that treats RSNA-RadioQA as one of two core radiology QA benchmarks and uses it as the main anchor for evaluating retrieval-based and agentic LLM systems (Wind et al., 1 Aug 2025).

2. Benchmark design and task formulation

The original RSNA-RadioQA benchmark was constructed from 80 peer-reviewed cases from the RSNA Case Collection. For each case, the clinical history and the image characteristics from the figure legend were converted into a single question asking for the most likely diagnosis or, in a smaller number of cases, a focused radiologic concept such as a classification or syndrome. Direct hints at the diagnosis or differential diagnoses were removed, and the benchmark was intentionally text-only because the study concentrated on LLMs without image processing capabilities (Arasteh et al., 2024).

Its question style follows radiology board and teaching-case conventions. Input consists of a short clinical scenario and a textual summary of imaging findings; output is a single concise sentence. The task therefore measures diagnostic reasoning over radiologic case descriptions rather than visual perception. A second benchmark, RadioQA, was later created by board-certified radiologists to mitigate contamination from publicly available RSNA case material. In the agentic retrieval study, the 80 RSNA-RadioQA items were kept unchanged and combined with a 24-question ExtendedQA set; every question in the resulting 104-item composite benchmark was presented as a four-option multiple-choice question (Wind et al., 1 Aug 2025).

This construction has two consequences. First, RSNA-RadioQA is unusually well aligned with radiology pedagogy: it encodes the style of conference teaching files and subspecialty case conferences rather than generic medical QA. Second, its text-only design makes it possible to isolate retrieval, reasoning, and answer selection from image encoder performance. This suggests that RSNA-RadioQA functions less as a full multimodal benchmark than as a controlled substrate for studying radiologic inference, factuality, and knowledge use.

3. RadioRAG and online retrieval-augmented radiology QA

The first dedicated retrieval framework built on RSNA-RadioQA was RadioRAG. It retrieves domain-specific information from Radiopaedia in real time rather than relying on a fixed local corpus. The retrieval pipeline first uses GPT-3.5-turbo to extract up to five keywords from the question; for each keyword, it queries Radiopaedia and selects the five most relevant articles, yielding up to 25 articles per question. These articles are split into 1,000-token chunks with 200-token overlap, embedded with text-embedding-ada-002, and stored in a temporary Chroma vector database. The original question is then embedded, and the top three chunks are retrieved by cosine similarity and passed as context to the answering model (Arasteh et al., 2024).

Under this setup, LLMs were evaluated in zero-shot mode with and without retrieval. The models included GPT-3.5-turbo, GPT-4, Mistral-7B, Mixtral-8x7B, and Llama3 in 8B and 70B variants. RadioRAG improved diagnostic accuracy across most models, with relative accuracy increases ranging up to 54% for different LLMs, although the magnitude of improvement varied. GPT-3.5-turbo and Mixtral-8x7B showed notable gains, whereas Mistral-7B-instruct-v0.2 showed no improvement, indicating that retrieval effectiveness is model-dependent rather than uniformly beneficial (Arasteh et al., 2024).

Methodologically, RadioRAG established several features that became central to later RSNA-RadioQA work. It made online radiology retrieval operational, showed that Radiopaedia can act as a high-yield external knowledge source for radiologic differential diagnosis, and framed factuality as a retrieval-grounding problem rather than only an internal parametric-knowledge problem. At the same time, it exposed the fragility of single-step retrieval: irrelevant context can degrade answers, and dynamic web retrieval introduces dependence on source coverage, latency, and reproducibility.

4. Agentic retrieval and the expansion of RSNA-RadioQA evaluation

A later study re-evaluated RSNA-RadioQA with an agentic retrieval framework. Here, each question is processed as a JSON object with a stem and four answer options. Mistral Large first performs keyword abstraction; a supervisor agent implemented with GPT-4o-mini then drafts a research plan with one section per diagnostic option. Dedicated research agents query Radiopaedia through SearXNG, constrained to site:radiopaedia.org, and are allowed to reformulate queries iteratively, with up to four retrieval attempts per option. The resulting evidence is synthesized into a structured report with an introduction, per-option sections, and a neutral conclusion, and this report becomes the context for the final answering model (Wind et al., 1 Aug 2025).

This agentic framework was evaluated on 24 LLMs spanning parameter scales from 0.5B to more than 670B, including general-purpose, reasoning-optimized, and clinically fine-tuned models. On the 104-question composite benchmark derived from RSNA-RadioQA and ExtendedQA, agentic retrieval significantly improved mean diagnostic accuracy over zero-shot prompting, 73% versus 64%, and over conventional online RAG, 73% versus 68%. The largest gains occurred in small and mid-sized systems, such as Qwen 2.5-7B, which improved from 55% to 71%, and Mistral Large, which improved from 72% to 81%; by contrast, very large models above 200B parameters showed minimal changes, typically less than 2% improvement (Wind et al., 1 Aug 2025).

The same study added a more explicit account of factual grounding. Retrieved context was judged clinically relevant in 46% of questions, mean hallucination rate under agentic retrieval was 9.4%, and clinically fine-tuned models also benefited, with MedGemma-27B improving from 71% to 81% (Wind et al., 1 Aug 2025). A radiologist answering the same 104 questions improved from 51% to 68% when given only the agentic Radiopaedia report, which indicates that the retrieved evidence itself has diagnostic value independent of the answering model. This suggests that RSNA-RadioQA has evolved from a static benchmark into a testbed for human-AI evidence mediation: not only whether a model answers correctly, but whether the retrieval process generates clinically useful intermediate reasoning artifacts.

5. Relation to RSNA multimodal benchmarks and report-centered QA

RSNA-RadioQA occupies only one part of a broader RSNA benchmarking landscape. A complementary development is REVEAL-CXR, described as the first RSNA multimodal LLM benchmark focused specifically on chest radiographs. REVEAL-CXR was built from 13,735 deidentified chest radiographs and reports from MIDRC, sampled 1,000 studies for expert review, and produced a final benchmark of 200 chest radiographic studies with 12 benchmark labels, divided into 100 released and 100 holdout studies. Each case was reviewed by three chest radiologists, and the holdout set is reserved for RSNA-managed evaluation (Wei et al., 21 Jan 2026).

REVEAL-CXR differs from RSNA-RadioQA in task structure. Whereas RSNA-RadioQA is an open-ended diagnostic QA task derived from teaching cases, REVEAL-CXR is a label-verified abnormality benchmark oriented toward multimodal LLM evaluation on original DICOM chest radiographs. Yet the relationship is direct: REVEAL-CXR can be wrapped into RadioQA-style yes/no or list-all-findings questions, and the paper explicitly notes that its labels can serve as the objective reference for correctness in such settings (Wei et al., 21 Jan 2026). This marks an important shift from case-level diagnostic reasoning toward expert-validated image-grounded answer checking.

A parallel shift occurs in report evaluation. ReportQA treats the radiology report itself as context and evaluates whether an LLM judge can answer clinically grounded QA pairs derived from structured entity and attribute trees. It covers MIMIC-CXR, CTRG-Brain, CT-RATE, and AMOS-MM, yielding approximately 660K QA pairs across 6,857 reports, and defines a report-level QAScore as the harmonic mean of a positive consistency score and an exponentially penalized negative score. On RadEvalX, QAScore shows better alignment with radiologist judgments than existing metrics, with absolute Pearson, Spearman, and Kendall correlations of 0.4507, 0.4612, and 0.3910, respectively (Shi et al., 13 Jun 2026). In the RSNA-RadioQA context, this introduces a rigorous answer-centric evaluation grammar for future systems that must generate or justify reports, not merely return a diagnosis.

6. Imaging-grounded building blocks and workflow-oriented QA

Although the original RSNA-RadioQA benchmark is text-only, surrounding RSNA literature provides image-grounded components directly relevant to question answering. On the RSNA Pneumonia Detection Challenge dataset, a radiomics-plus-contrastive framework aligned image embeddings with 102-dimensional radiomic feature vectors extracted from bounding-box regions and achieved 0.886 accuracy, 0.927 F1, and 0.923 AUC with the ResNet-18AttRadi model. The same work explicitly connected such representations to answering questions like “Does this chest radiograph show pneumonia? If so, where?” and to justifying answers with ROI-focused evidence (Han et al., 2021).

A different RSNA pneumonia line used an ensemble of two 13-layer CNNs to generate a 17×17 heatmap that guides a second classifier, achieving precision 0.84, recall 0.80, F1 score 0.82, and AUROC 0.95 on the RSNA test set. The authors presented the heatmap as a QA-support tool because it visualizes why the algorithm is confident about pneumonia and whether it is focusing on plausible radiological features rather than spurious structures (Singh, 2020). Earlier RSNA challenge work based on CoupleNet similarly provided robust bounding-box localization, which can ground answers to questions about presence and location of focal opacities (Team, 2018).

Workflow-oriented QA extends beyond answer generation into discrepancy detection. AQUARIUS formalized radiology quality assurance as a comparison between AI-based image analysis and NLP-derived report labels, followed by expert review only of discordant cases. In a study of 1,936 emergent head CT scans, targeted expert review of only 29 discordant cases reduced human QA effort by 98.5% and identified six non-reported true ICH-positive cases (Wismueller et al., 2022). This is not RSNA-RadioQA in the narrow benchmark sense, but it operationalizes the same logic: ask whether image-derived and report-derived answers agree, and concentrate human review where the answers diverge.

These threads show that RSNA-RadioQA is best understood as a layered stack. At the top sits benchmarked diagnostic reasoning over case descriptions; beneath it are image-grounded detectors, attention maps, radiomic embeddings, report-based QA metrics, and discordance-driven QA workflows. A plausible implication is that mature RSNA-RadioQA systems will combine all of these: image evidence, retrieved knowledge, structured reporting, and targeted quality assurance.

7. Limitations and future directions

The original RSNA-RadioQA benchmark has clear limitations. It is small, with 80 questions, and later composite evaluation still totals only 104 items when ExtendedQA is included (Arasteh et al., 2024, Wind et al., 1 Aug 2025). It is also text-only: the benchmark measures diagnostic reasoning over curated case descriptions rather than image interpretation. Because the RSNA Case Collection is public, contamination of model training data is a legitimate concern; this is precisely why later work introduced non-public expert-authored questions alongside RSNA-RadioQA (Arasteh et al., 2024, Wind et al., 1 Aug 2025).

Retrieval-based variants introduce further constraints. RadioRAG depends on Radiopaedia as an external source, and the agentic system still found clinically relevant context in less than half of questions (Arasteh et al., 2024, Wind et al., 1 Aug 2025). Online retrieval also creates latency and reproducibility issues: agentic runs are substantially slower than zero-shot inference, and dynamic web content can shift over time. These studies therefore suggest a trade-off between contemporaneous knowledge access and stable benchmarking.

The current direction of travel in the literature is nevertheless coherent. RSNA-derived QA is moving toward expert-validated multimodal case sets, structured label ontologies, report-based QA metrics, and agentic retrieval pipelines. Future RSNA-RadioQA systems will plausibly incorporate actual imaging, curated and versioned retrieval corpora, structured entity-attribute schemas, and workflow-aware QA modules that compare images, reports, and external knowledge in one loop. The combined evidence suggests that RSNA-RadioQA is no longer only a diagnostic quiz benchmark; it is becoming a general framework for evaluating how radiology AI systems answer, justify, and audit clinically meaningful questions.