RadJudge: Clinical Benchmark for X-Ray Reports
- RadJudge is a specialized benchmark evaluating automated chest X-ray report scoring against nuanced radiologist judgment in critical clinical scenarios.
- It comprises 30 curated cases across 10 clinically nuanced categories, requiring consensus among three expert cardiothoracic radiologists.
- The suite probes evaluator weaknesses by testing context sensitivity, severity prioritization, and recognition of clinically equivalent differences in borderline edge cases.
RadJudge is a targeted pass–fail benchmark for chest X-ray report evaluation introduced as part of the CRIMSON framework, an LLM-based, clinically grounded metric for generative radiology report assessment. It is not itself the scoring metric; rather, it is a curated validation suite designed to test whether an automatic evaluator behaves as expert cardiothoracic radiologists do in clinically nuanced edge cases. In the CRIMSON formulation, RadJudge asks whether candidate reports should be ranked in the same order as radiologists would rank them, or treated as clinically indistinguishable when no meaningful difference exists, thereby probing context sensitivity, severity prioritization, and patient-safety relevance beyond generic error counting (Baharoon et al., 6 Mar 2026).
1. Position within the CRIMSON evaluation framework
CRIMSON was introduced to assess chest X-ray report generation in terms of diagnostic correctness, contextual relevance, and patient safety rather than textual similarity alone. The framework incorporates full clinical context, including patient age, indication, and guideline-based decision rules, and it prevents normal or clinically insignificant findings from exerting disproportionate influence on the overall score. Within that broader system, RadJudge functions as a stringent validation instrument: it tests whether the metric’s outputs align with expert judgment in difficult pairwise comparisons where the clinically preferable report is not obvious from lexical overlap or coarse structured matching (Baharoon et al., 6 Mar 2026).
RadJudge is best understood in relation to the other validation components used for CRIMSON.
| Component | Format | Primary role |
|---|---|---|
| ReXVal | Radiologist-annotated clinically significant error counts | Correlation with expert error counts |
| RadJudge | 30 curated pass–fail cases across 10 categories | Agreement with expert ordering in edge cases |
| RadPref | 100 cases with two candidate reports, a reference report, and 1–5 ratings | Correlation with graded radiologist preferences |
This structure is central to the paper’s argument. ReXVal evaluates aggregate correlation with clinically significant error counts; RadPref evaluates alignment with graded pairwise preferences; RadJudge is sharper and more adversarial, because it asks whether the evaluator gets the ordering right in carefully designed borderline scenarios rather than merely approximating an average score (Baharoon et al., 6 Mar 2026).
2. Benchmark design, adjudication, and operating rule
RadJudge is described as a curated suite of 30 cases spanning 10 clinically nuanced categories. Multiple candidate reports are compared within each case, and three cardiothoracic radiologists independently review them. Agreement among the radiologists is required to establish the reference preference, reflecting the fact that the cases are intentionally borderline and clinically subtle. The benchmark is explicitly pass–fail: a metric passes a case if it ranks the reports according to the expert ordering, or if it assigns effectively equal scores when the radiologists judge the reports to be clinically indistinguishable (Baharoon et al., 6 Mar 2026).
The criterion for clinical indistinguishability is operationalized as a score difference within 0.01. This detail is methodologically important. RadJudge does not reward arbitrary numerical separation between reports when expert readers regard the outputs as functionally equivalent from a clinical perspective. The benchmark therefore evaluates not only directional preference but also whether the evaluator respects equivalence classes of acceptable reporting. In that sense, RadJudge is less a general ranking benchmark than a test of clinically disciplined discrimination (Baharoon et al., 6 Mar 2026).
RadJudge also differs structurally from RadPref. RadPref is the larger pairwise preference benchmark: 100 cases, each with two candidate reports and a reference report, with each candidate rated 1–5 by three cardiothoracic radiologists. RadJudge, by contrast, is not a graded preference dataset. Its purpose is to determine whether the evaluator can make or withhold distinctions in the same places that radiologists do, which makes it a more concentrated probe of evaluator behavior under edge-case stress (Baharoon et al., 6 Mar 2026).
3. Clinical edge cases and the logic of the scenarios
The scenarios in RadJudge are designed to expose failure modes of metrics that ignore context, severity, or the asymmetry between clinically dangerous and clinically trivial discrepancies. The paper states that the suite probes urgent omissions versus benign hallucinations, context-dependent findings, diagnostic over-interpretation and under-interpretation, and cases reflecting the reality of imperfect reference reports that omit localization or age-expected benign findings. These are precisely the circumstances in which simple overlap metrics and many structured metrics tend to fail, because they either treat all discrepancies too similarly or collapse them into coarse significant/non-significant buckets (Baharoon et al., 6 Mar 2026).
A canonical example concerns omission of aortic atherosclerosis. The paper’s discussion emphasizes that such an omission may be clinically quite different in a 75-year-old versus a 25-year-old. This is a direct test of context dependence: the same textual omission need not carry the same clinical weight across patients. The suite also includes cases in which mention of a normal or expected finding should not inflate a score, thereby penalizing metrics that reward irrelevant normality statements or benign detail inflation (Baharoon et al., 6 Mar 2026).
Another class of scenarios concerns partial correctness. A candidate may correctly identify a lesion but use slightly off localization or an imprecise descriptor. RadJudge tests whether the evaluator recognizes that such reports may still be clinically acceptable, rather than over-penalizing minor wording deviations. Conversely, it includes cases where one candidate makes a benign hallucination while another misses an urgent abnormality; in those comparisons, a clinically aligned evaluator must prioritize the patient-safety-relevant omission over the low-impact hallucination. The benchmark is therefore built around tradeoffs that are common in real radiological review but poorly captured by text-similarity measures (Baharoon et al., 6 Mar 2026).
4. Dependence on CRIMSON’s clinically grounded scoring model
RadJudge is tightly coupled to the design philosophy of CRIMSON. CRIMSON first extracts findings from the reference and candidate reports, but normal findings are excluded from evaluation because they can introduce stylistic noise. Discrepancies are then categorized using an error taxonomy comprising false findings, missing findings, and attribute errors. Each finding is assigned one of four clinical significance levels—urgent, actionable non-urgent, non-actionable, or expected/benign—which determine its scoring weight (Baharoon et al., 6 Mar 2026).
The significance rubric is specified as
This weighting scheme encodes the principle that an urgent miss should count much more than a benign discrepancy, and that expected benign findings should not distort the score at all. RadJudge exists in part to verify that these weighting choices produce evaluator behavior that matches radiologist judgment in hard comparisons rather than only in aggregate statistics (Baharoon et al., 6 Mar 2026).
CRIMSON also uses an attribute-level taxonomy for matched findings, with eight attribute dimensions: location/laterality, severity/extent, morphology, quantitative measurements, certainty, underinterpretation, overinterpretation, and temporal/comparison descriptors. Attribute errors are split into significant versus negligible, with
The paper gives concrete examples of this distinction. Wrong lung laterality is significant, whereas “apical” versus “lateral” within the same lobe is negligible. For pulmonary nodules, a measurement discrepancy larger than 2 mm is significant if the nodule is smaller than 6 mm, and larger than 4 mm if the nodule is 6 mm or bigger, reflecting guideline-based practice. These rules are not decorative; they are the machinery that allows CRIMSON to grant partial credit when the right abnormality is identified but some clinically meaningful detail is wrong (Baharoon et al., 6 Mar 2026).
The final CRIMSON score is severity-aware and normalized to lie in , with 0 interpreted as roughly equivalent to a normal template, 1 as a perfect report, and negative values indicating that the report contains more clinically weighted errors than correct content. A plausible implication is that RadJudge serves as an external check that this normalized score behaves sensibly when translated into pairwise choices, especially when the clinically correct action is to treat two reports as effectively equivalent (Baharoon et al., 6 Mar 2026).
5. Empirical results and validation significance
The reported headline result on RadJudge is that CRIMSON is the only metric that gets all 30 out of 30 cases correct. Prior metrics do substantially worse, correctly resolving fewer than 35% of the cases. Within the paper’s validation narrative, this is one of the strongest pieces of evidence for the metric, because it shows that the framework’s context sensitivity, severity weighting, and avoidance of over-penalizing benign or normal findings translate into expert-aligned behavior in clinically difficult decision cases (Baharoon et al., 6 Mar 2026).
RadJudge sits inside a three-part validation strategy. On ReXVal, CRIMSON is reported to align strongly with clinically significant error counts annotated by six board-certified radiologists, with Kendall’s tau and Pearson’s . On RadJudge, it passes all cases. On RadPref, it achieves the strongest alignment with radiologist preferences among the compared metrics. The combined interpretation offered by the paper is that CRIMSON is not merely correlated with radiologist annotations in the aggregate; it also behaves correctly in the types of clinically consequential comparisons that matter for deployment-facing report evaluation (Baharoon et al., 6 Mar 2026).
The practical implication stated in the paper is that if a benchmark or metric cannot distinguish an urgent omission from a benign wording difference, model developers may optimize the wrong objective and improve surface fluency without improving safety. RadJudge therefore functions as a safeguard against evaluator misalignment: it checks whether the scoring system rewards what radiologists actually care about, namely dangerous misses, clinically consequential attribute errors, and the deprioritization of irrelevant normal or expected content (Baharoon et al., 6 Mar 2026).
6. Scope, misconceptions, and relation to adjacent work
A common misconception is to treat RadJudge as synonymous with CRIMSON. The paper explicitly separates them: CRIMSON is the clinically grounded metric, whereas RadJudge is one of the two new benchmarks introduced to validate it. A second misconception is to read RadJudge as a generic preference dataset. It is instead a targeted pass–fail suite whose function is to test whether an evaluator makes the same distinctions—or declines to make distinctions—that cardiothoracic radiologists make in especially tricky cases (Baharoon et al., 6 Mar 2026).
RadJudge should also not be conflated with “LLM-RadJudge,” an earlier LLM-based radiology-report evaluation framework that compares a candidate report to a reference by counting clinically meaningful discrepancies across six predefined error categories. That earlier framework reported GPT-4-turbo Kendall’s tau against radiologist judgments on ReXVal and used a two-stage prompting strategy inspired by Chain-of-Thought and Chain-of-Density, but it is a scoring method rather than the curated pass–fail benchmark defined in CRIMSON (Wang et al., 2024).
In a broader methodological context, RadJudge is adjacent to recent work on evaluating judges themselves. JudgeBench evaluates LLM judges on objectively correct versus incorrect response pairs across knowledge, reasoning, mathematics, and coding, while the Judge Reliability Harness stress-tests judges under formatting changes, paraphrasing, verbosity shifts, stochastic repetition, and agent transcript edits (Tan et al., 2024, Dev et al., 5 Mar 2026). This suggests that RadJudge occupies a narrower but more clinically specific niche. Its core question is not whether a judge is generally robust across domains, but whether a radiology-report evaluator respects radiological severity, patient context, and clinical equivalence in the particular edge cases where benchmark optimization can otherwise become clinically misleading.
Taken together, RadJudge is best characterized as a clinically grounded sanity check for automated radiology-report evaluation. It operationalizes whether an evaluator understands that urgent omissions matter more than benign hallucinations, that context alters significance, and that some textual differences are clinically negligible even when they are lexically salient. Within the CRIMSON framework, that role makes it a central instrument for distinguishing mere correlation with annotations from genuine alignment with radiologist judgment (Baharoon et al., 6 Mar 2026).