CLeaR Framework for Radiology Report Evaluation

• CLeaR is a clinically grounded structured framework that converts free-text radiology reports into a tabular form with detailed condition and attribute values.
• It compares candidate and reference reports at the attribute level—including presence, temporal context, severity, location, and recommendation—for precise clinical evaluation.
• The framework enables nuanced performance analysis of report generation systems, improving model selection and error localization with high clinical relevance.

CLeaR, or CLEAR, is a clinically grounded framework for radiology report evaluation that converts free-text reports into a structured table of condition–attribute values and compares candidate and reference reports at the attribute level rather than only at the lexical or report level. In the formulation introduced in “CLEAR: A Clinically-Grounded Tabular Framework for Radiology Report Evaluation,” the goal is to evaluate report generation systems in a way that better matches how radiologists identify findings and describe them across clinically important dimensions, including presence, temporal context, severity, location, and recommendation (Jiang et al., 22 May 2025).

1. Clinical motivation and evaluation target

CLEAR is motivated by limitations in three broad classes of radiology-report metrics: lexical metrics such as BLEU, ROUGE, METEOR, and BERTScore; clinical label or entity metrics such as CheXpert/CheXbert label similarity, RadGraph F1, RaTEScore, and MEDCON; and LLM-as-a-judge metrics such as GREEN, CheXprompt, FineRadScore, and RadFact (Jiang et al., 22 May 2025). The central criticism is that these approaches often lack granularity, have limited interpretability for clinicians, and may align poorly with clinical goals when small wording differences alter clinical meaning.

The framework therefore shifts the evaluation question from textual similarity to clinically structured agreement. Rather than asking whether two reports are similar as documents, CLEAR asks whether the candidate report identifies the right conditions and whether each positively identified condition is described correctly along clinically important dimensions. This is operationalized through a tabular representation in which rows correspond to conditions and columns correspond to attributes, enabling condition-wise and attribute-wise comparison (Jiang et al., 22 May 2025).

A common misconception is to regard CLEAR as a single aggregated metric. The framework explicitly does not collapse evaluation into one scalar. Instead, it yields a multidimensional score vector spanning presence detection, categorical descriptive attributes, and free-text descriptive attributes. This design is intended to preserve interpretability for radiologists and failure localization for model developers (Jiang et al., 22 May 2025).

2. Tabular schema and formal definition

CLEAR operates on a fixed condition set $C$ derived from the CheXpert label set, excluding “No Finding”:

$$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$

For each condition $c \in C$, CLEAR models six attributes (Jiang et al., 22 May 2025). The first is presence $S_1$, with value space

$$S_1 \in \{\text{"Positive"}, \text{"Unclear"}, \text{"Negative"}\}.$$

The remaining five describe only positive findings. First occurrence $S_2$ captures temporal status:

$$S_2 \in \{\text{"Previous"}, \text{"Current"}, \text{"N/A"}\}.$$

Change $S_3$ models longitudinal evolution:

$$S_3 \in \{\text{"improving"}, \text{"stable"}, \text{"worsening"}, \text{"mixed"}, \text{"N/A"}\}.$$

Severity $S_4$ uses

$$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$0

Descriptive location $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$1 is a set of free-text phrases,

$$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$2

or “N/A” when no location is mentioned. Recommendation $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$3 is likewise a set of free-text phrases or “N/A” (Jiang et al., 22 May 2025).

For a report $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$4, the framework defines its structured representation as

$$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$5

Presence is filled for all conditions, whereas attributes $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$6 through $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$7 are only meaningful for conditions classified as positive in Stage 1. CLEAR uses LLMs to implement a mapping

$$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$8

The Label Extraction Module produces $$C = \{\text{Atelectasis}, \text{Cardiomegaly}, \text{Consolidation}, \text{Edema}, \text{Enlarged Cardiomediastinum},\ \text{Fracture}, \text{Lung Lesion}, \text{Lung Opacity}, \text{Pleural Effusion}, \text{Pleural Other}, \text{Pneumonia}, \text{Pneumothorax}, \text{Support Devices}\}$$9 for all conditions, and the Description Extraction Module produces $c \in C$0 for conditions with $c \in C$1. For a ground-truth report $c \in C$2 and a candidate report $c \in C$3, the system extracts two tables and compares them attribute-wise and condition-wise (Jiang et al., 22 May 2025).

The paper notes that some prompts and internal discussion mention an “Urgency” attribute, but the benchmark uses the six attributes above. This distinction matters because the benchmarked formulation is presence plus five descriptive attributes, not a broader open-ended schema (Jiang et al., 22 May 2025).

3. Scoring methodology and interpretability

CLEAR decomposes evaluation into two stages. Stage 1 evaluates condition presence. Stage 2 evaluates descriptive attributes for positive conditions (Jiang et al., 22 May 2025).

For presence, CLEAR treats “Positive” and “Negative” as separate binary targets and reports class-wise F1. The class-wise score is

$c \in C$4

Reported metrics include Pos F1@13, Pos F1@5, Pos F1 (micro), and the analogous Neg F1@13, Neg F1@5, and Neg F1 (micro). Pos F1@5 is macro-F1 over the five most frequent conditions: Pneumothorax, Pneumonia, Edema, Pleural Effusion, and Consolidation (Jiang et al., 22 May 2025).

For first occurrence, change, and severity, the framework uses exact-match accuracy over positive condition–report pairs:

$c \in C$5

It reports micro accuracy, condition-averaged accuracy, and report-averaged accuracy. Because these attributes are posed as multiple-choice question answering in prompts, the paper treats exact-match accuracy as appropriate (Jiang et al., 22 May 2025).

For location and recommendation, the values are phrase sets rather than categorical labels. CLEAR uses phrase-level similarity with optimal matching:

$c \in C$6

The similarity function may be BLEU-4 or ROUGE-L at the phrase level. In addition, CLEAR uses OpenAI’s o1-mini as an LLM-based similarity judge that produces a score in $c \in C$7 for ground-truth and generated phrase lists (Jiang et al., 22 May 2025).

The resulting output is a multidimensional evaluation profile rather than a scalar. This makes explicit distinctions such as correct presence but incorrect severity, or correct condition detection but missing recommendation. The paper’s motivating examples emphasize that a report may receive a high lexical or graph-based similarity score while still misdescribing whether a finding is worsening, mild rather than severe, or associated with a follow-up action. CLEAR externalizes these discrepancies at the condition–attribute cell level (Jiang et al., 22 May 2025).

4. Labeling scheme, expert curation, and CLEAR-Bench

CLEAR’s presence schema begins from MIMIC-CXR-JPG’s CheXpert-like labels and refines them in collaboration with radiologists into five certainty categories: Confidently Present, Likely Present, Neutral, Likely Absent, and Confidently Absent, plus a separate “Mentioned vs. Unmentioned” flag. For downstream evaluation these are mapped to a three-class space: Positive, Negative, and Unclear (Jiang et al., 22 May 2025).

To curate presence labels for model development, the authors start from the MIMIC-CXR-JPG test set, use GPT-4o to relabel each report according to MIMIC’s own labeling guidelines, identify disagreements with the original labels, and ask a board-certified radiologist to re-annotate only the mismatched conditions. Cases with more than five mismatches are dropped. This yields 550 studies with high-quality labels across all 13 conditions for fine-tuning and evaluation of label extractors (Jiang et al., 22 May 2025).

CLEAR-Bench is the benchmark dataset used to assess clinical alignment. It contains 100 chest X-ray studies from the MIMIC-CXR-JPG validation and test sets, excludes training samples used in curation and excludes normal or “No Finding” studies. Each report is annotated over the same 13 conditions and six attributes used by the framework (Jiang et al., 22 May 2025).

Presence labels in CLEAR-Bench are produced by three board-certified radiologists. Each radiologist uses the five certainty categories for each of the 13 conditions. The final gold label is obtained by mapping present and absent categories to Positive and Negative, keeping Neutral as Unclear, then applying majority vote and resolving remaining disagreements by consensus discussion (Jiang et al., 22 May 2025).

For descriptive attributes, the authors first generate attribute suggestions with Llama-3.1-8B-Instruct and GPT-4o for each positive condition. Two radiologists then review these outputs, label each attribute as incorrect, partially correct, or correct, and provide corrected content when needed. In CLEAR-Bench, the review scores use $c \in C$8, $c \in C$9, and $S_1$0. The corrected attributes form the gold labels for first occurrence, change, severity, descriptive location, and recommendation. The benchmark also stores radiologists’ Likert-style scores for correlation analysis with CLEAR’s automatic metrics (Jiang et al., 22 May 2025).

5. Experimental performance and clinical alignment

For Stage 1 label extraction, the evaluated models include Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct in base and fine-tuned forms, Llama-3.1-70B-Instruct and Llama-3.3-70B-Instruct in zero-shot and 5-shot settings, GPT-4o in zero-shot and 5-shot settings, and the baselines CheXpert and CheXbert (Jiang et al., 22 May 2025).

The best-performing CLEAR variants substantially exceed prior labelers. For Positive F1 averaged over 13 conditions, CheXbert scores 0.695, CheXpert 0.674, and the best CLEAR variant 0.805, corresponding to a reported +15.8% absolute improvement over CheXbert. For Positive F1@5, CheXbert scores 0.833 and the best CLEAR variant 0.940. For Positive micro F1, CheXbert scores 0.897 and the best CLEAR variant 0.934. Negative detection shows an even larger gap: Neg F1@13 is 0.498 for CheXbert, 0.522 for CheXpert, and 0.744 for the best CLEAR medium model with 5-shot prompting, which the paper reports as +42.5% over prior state of the art (Jiang et al., 22 May 2025).

The paper attributes these patterns to model scale and supervision regime. Larger models are strongest on positive condition detection, while few-shot prompting and fine-tuning particularly improve negative detection, which often depends on implicit negatives such as “lungs are clear.” Fine-tuning small models on curated labels substantially boosts their negative F1 (Jiang et al., 22 May 2025).

For Stage 2 attribute extraction, GPT-4o achieves expert-judged average scores of 0.793 for first occurrence, 0.853 for change, 0.786 for severity, 0.865 for location, and 0.936 for recommendation. The corresponding micro accuracies or similarities reported for GPT-4o are 0.737 for first occurrence, 0.754 for change, 0.671 for severity, 0.785 for location using o1-mini similarity, and 0.888 for recommendation using o1-mini similarity. Llama-3.1-8B-Instruct is weaker overall but still reaches 0.760 expert average and 0.739 o1-mini similarity for location (Jiang et al., 22 May 2025).

Clinical alignment is measured by Pearson correlation between CLEAR’s automated metrics and expert ratings over GPT-4o outputs. Condition-averaged accuracy correlates with expert judgment at 0.894, report-averaged accuracy at 0.908, and micro accuracy at 0.915. Among similarity metrics for free-text attributes, o1-mini similarity reaches 0.994, ROUGE-L 0.977, and BLEU-4 0.811. The paper interprets these results as evidence that CLEAR’s automatic metrics are very strongly aligned with radiologist judgments, with o1-mini similarity nearly matching expert assessment for location and recommendation evaluation (Jiang et al., 22 May 2025).

6. Position within prior work, practical use, and limitations

CLEAR extends prior label-based metrics such as CheXpert and CheXbert by retaining a CheXpert-derived condition space for presence while adding five descriptive attributes per condition. Relative to graph or entity metrics such as RadGraph F1, RadGraph2, MEDCON, and RaTEScore, it replaces implicit structural interpretation with explicit attribute slots such as severity and progression. Relative to aggregate correctness metrics such as RadCliQ, GREEN, CheXprompt, FineRadScore, and RadFact, it uses a condition-by-attribute grid rather than a collapsed error count (Jiang et al., 22 May 2025).

In practical terms, the framework is intended for model selection, benchmarking, debugging, and failure analysis. It allows comparisons such as whether one model is better at detecting positives while another is better at generating recommendations. It also exposes systematic errors at the condition–attribute level, such as underestimation of edema severity or omission of fracture recommendations. A plausible implication is that such outputs are more actionable for iterative model development than a single report-level score, because they localize failure modes to clinically meaningful cells in the table.

The paper also delineates several limitations. CLEAR is image-free: it compares candidate reports to reference reports rather than to the underlying images, so a reference report may itself omit or mischaracterize abnormalities. Its scope is limited to 13 CheXpert chest X-ray conditions and does not cover broader anatomical domains such as breast imaging, cardiology, or gastrointestinal imaging. CLEAR-Bench contains 100 studies and the curated training set contains 550 studies, which are relatively small because expert annotation is expensive. This suggests that the framework prioritizes annotation quality and attribute fidelity over benchmark scale (Jiang et al., 22 May 2025).

A further point of clarification concerns what CLEAR evaluates. It is not a report-generation model and not a general-purpose radiology reasoning benchmark. It is an evaluation framework whose principal contribution is a clinically grounded, expert-curated, attribute-level representation and comparison scheme for radiology report quality. Within that scope, its reported contribution is a more comprehensive and clinically interpretable evaluation whose automatic metrics are strongly aligned with clinical judgment (Jiang et al., 22 May 2025).