ReportQA: QA-Based Radiology Report Evaluation

Abstract: Radiology report evaluation is essential for advancing automated report generation. Natural language generation metrics have limited clinical relevance. Clinical efficacy (CE) metrics evaluate important medical findings, but focus mainly on presence and cover only a limited set of entities. Due to heavy reliance on manual annotations, it is difficult for CE metrics to extend clinical entities or attributes. In clinical practice, radiology reports serve as a medium for information transfer. Clinicians use them to perform downstream diagnostic tasks without directly inspecting images. Based on this insight, we propose ReportQA, a clinical-related and flexible radiology report evaluation framework, supporting detailed quantitative analysis of radiology report generation systems. We first collect datasets covering multiple imaging modalities and anatomical regions. We then construct knowledge trees of clinical entities and attributes with radiologist guidance, and use LLMs to extract structured information from raw reports. Next, we generate QA pairs from predefined templates and apply quality control through self-filtering and report-based filtering. During evaluation, the report is treated as context, and an LLM acts as a judge model to answer the QA pairs. Based on the resulting QA accuracy, we introduce QAScore metric. Compared with existing metrics, QAScore shows better alignment with radiologist judgments. Experiments on multiple state-of-the-art vision-LLMs reveal that current report-based inference paradigms struggle to learn fine-grained clinical representations and exhibit strong negative prior biases. In contrast, question-driven inference provides a more effective alternative. For reproducibility and extensibility, we release the knowledge trees, structured reports, and QA pairs, along with the pipeline code for QA construction and evaluation.

• The paper introduces ReportQA, a QA-based framework that reframes radiology report evaluation to achieve clinical fidelity and extensibility.
• It employs a four-stage pipeline to transform free-form reports into structured formats and generate QA pairs for robust clinical assessment.
• Evaluated via the composite QAScore, ReportQA outperforms traditional metrics by capturing fine-grained clinical details, especially in 2D imaging.

ReportQA: QA-Based Evaluation for Radiology Report Generation

Motivation and Limitations of Existing Radiology Report Evaluation

Radiology report generation (RRG) has garnered substantial attention due to its potential to streamline clinical workflows and standardize report quality. However, the development of automated RRG systems is impeded by the inadequacy of evaluation metrics that fail to capture clinical correctness. Conventional natural language generation (NLG) metrics, such as BLEU, prioritize lexical overlap, disregarding the clinical accuracy of report content. Clinical efficacy (CE) metrics, including CheXbert and RadGraph, improve clinical relevance but restrict evaluation to the presence of select entities, lacking coverage of fine-grained attributes and being dependent on extensive manual annotation.

ReportQA is designed to overcome these methodological limitations. By conceptualizing the evaluation as a QA task aligned with real clinical usage—where clinicians extract diagnostic information from reports without directly inspecting images—it achieves both clinical fidelity and extensibility across entities, attributes, modalities, and anatomical regions.

Framework Architecture of ReportQA

ReportQA operates as a four-stage pipeline. Initially, datasets covering diverse imaging modalities and anatomical regions are curated. Free-form reports are transformed into structured formats guided by radiologist-defined knowledge trees that enumerate clinical entities and attributes. Subsequently, QA pairs are systematically generated via predefined templates, spanning base (leaf-level), hierarchical (ancestor-level), and negative categories. QA pairs undergo self-filtering and report-based filtering, utilizing LLMs to ensure context dependency and reliability.

During evaluation, the generated report is treated as context for an LLM judge model, which answers the QA pairs. Performance is assessed through QA accuracy, with the introduction of a composite QAScore metric that measures both positive consistency and negative avoidance.

Figure 1: Overview of the ReportQA pipeline, from dataset collection to structured report extraction, QA pair generation, filtering, and evaluation using LLMs.

Dataset Construction and Structured Information Extraction

ReportQA leverages datasets such as MIMIC-CXR, CTRG-Brain, CT-RATE, and AMOS-MM, encompassing 2D and 3D imaging. Radiologists develop knowledge trees for each dataset, encompassing entities (findings, diagnoses) and a spectrum of clinical attributes (presence, location, distribution, margin, chronicity, enhancement, etc.). LLMs undertake a two-stage extraction: preliminary structured outputs inform refinement of the knowledge trees, followed by remapping and filtering for standardization.

Figure 2: Example transformation from raw report to structured format and generated QA pairs for CT-RATE.

QA Pair Generation, Filtering, and Evaluation

Base QA pairs target distinct (entity, attribute) combinations, while hierarchical pairs incorporate ontology-aware reasoning for ancestor nodes and negative pairs test for the absence of entities. Filtering stages eliminate context-independent or unreliable pairs, ensuring that retained pairs are robustly dependent on report content.

Statistically, filtering yields approximately 660,000 high-quality QA pairs across 6,857 reports, maintaining high QA density per report.

QAScore: Metric Definition and Radiologist Alignment

QAScore synthesizes performance across positive clinical entity preservation (via base and hierarchical questions) and avoidance of clinically irrelevant false positives (via negative questions). It is defined as the harmonic mean of positive and negative scores, penalizing reports that either fail to include key findings or erroneously introduce spurious entities.

ReportQA demonstrates superior alignment with radiologist judgments versus established metrics. Correlation analyses on RadEvalX reveal QAScore achieves the highest Pearson (0.4507), Spearman (0.4612), and Kendall (0.3910) coefficients among all metrics, outperforming CheXbert, GREEN, and RaTEScore.

Model Evaluation: Performance, Attribute Difficulty, and Paradigm Bias

Proprietary VLMs (e.g., GPT-5.4) consistently outperform open-source medical and general VLMs, particularly in 2D chest X-ray evaluation, but face substantial challenges on 3D datasets, indicating current limitations in 3D volumetric understanding.

Attribute-level performance analysis reveals a pronounced hierarchy: presence and location are generally tractable, while margin, chronicity, enhancement, and internal features are rarely predicted correctly across models.

Figure 3: Distribution of attribute-level accuracy across dataset-model pairs, elucidating the gradient of attribute difficulty for VLMs.

Supervised fine-tuning for report-based inference yields only modest gains in coarse-grained entity recognition; fine-grained attributes remain inadequately addressed. Question-driven inference allows models to better leverage image and report cues, activating clinical entity recognition more reliably, but remains vulnerable to textual priors.

Figure 4: Performance comparison of SFT and question-driven models on base, hierarchical, and negative questions for CTRG-Brain.

Report-based paradigm activates a strong negative prior bias, with models defaulting to absence for most entities unless directly provided image context. In contrast, question-driven inference exhibits a mild positive bias, informed by the specificity of clinical queries.

Figure 5: Comparative analysis of inference paradigms demonstrates divergent prior biases and the impact of image input.

Architectural and Scaling Analysis

Experimentation with judge models of different parameter sizes (Qwen3.5 family) underscores a trade-off: larger models deliver improved QA filtering and evaluation accuracy but demand heavier compute. Increasing input slices marginally enhances performance but does not address foundational clinical understanding. Prompt specificity improves task adherence in specialized VLMs but has limited effect in general VLMs; simply increasing model size fails to yield robust gains for the complex RRG task, indicating a bottleneck at structured visual-textual alignment, not model capacity.

Implications and Future Directions

ReportQA operationalizes QA-based evaluation for RRG, supporting systematic quantitative analysis with fine clinical granularity. The superior alignment with radiologist judgment highlights its utility for practical deployment and benchmark development. Release of knowledge bases, structured reports, QA pairs, and pipeline code enables reproducibility, extensibility, and further research in radiology report evaluation.

Theoretical implications suggest that question-driven inference paradigms hold promise for mitigating prior biases and improving fine-grained representation learning, especially if combined with mixed positive and negative training data and further optimization of domain-specific prompt engineering. Practical application will benefit as VLMs advance in 3D image interpretation and structured clinical reasoning. Future research should explore adaptive ontological modeling, continual domain adaptation, and multimodal fusion for comprehensive clinical report understanding in high-stakes medical environments.

Conclusion

ReportQA provides a clinically aligned, extensible, and scalable framework for RRG evaluation. By reframing the task as QA, it exposes hidden model deficits, particularly in fine-grained clinical reasoning and 3D imaging. Current VLMs remain limited by prior biases and lack robust attribute-level understanding, especially in volumetric contexts. The framework's modularity and public release will catalyze future advances in medical AI evaluation, supporting rigorous benchmarking and practical translation in radiology workflows.

(2606.15037)