ReXrank Benchmark Overview

• ReXrank Benchmark is a standardized evaluation framework that rigorously assesses AI performance in radiograph interpretation, automated report generation, and knowledge graph relationship explanations.
• It leverages the large-scale ReXGradient-160K dataset alongside harmonized public datasets with defined splits for training, validation, and secure leaderboard testing.
• The benchmark features detailed metrics—including BLEU-2, BERTScore, RadGraph-F1, and others—and a robust submission process to ensure transparent, reproducible comparisons.

The ReXrank Benchmark is a standardized, public evaluation framework for rigorously benchmarking AI models in chest radiograph interpretation, automated radiology report generation, and, in a parallel context, relationship explanation between entity pairs in knowledge bases. Central to ReXrank’s clinical imaging variant is the large-scale ReXGradient-160K dataset, featuring 160,000 chest X-ray studies paired with free-text reports, enabling reproducible comparisons and robust generalization assessment across clinical domains (Zhang et al., 1 May 2025, Zhang et al., 2024). For entity-relationship explanations, ReXrank implements algorithmic standards and evaluation protocols for enumerating and ranking minimal explanations in graph-structured knowledge bases (Fang et al., 2011). This benchmark addresses the absence of universal yardsticks for both automated report generation quality and entity-pair explanation utility, facilitating head-to-head model comparison under transparent protocols.

1. Data Resources and Corpus Organization

The clinical imaging arm of ReXrank is defined by the ReXGradient-160K dataset, comprising $N_{\text{total}}=160{,}000$ chest X-ray studies from $P=109{,}487$ patients at 79 medical sites spanning three U.S. health systems (Zhang et al., 1 May 2025). Each study typically contains multiple images and a comprehensive free-text radiology report. The dataset is stratified into distinct splits:

Split Type	# Studies	Clinical Role	Access Protocol
Training	140,000	Model fitting	Unrestricted
Validation	10,000	Hyperparameter tuning	Unrestricted
Public Test	10,000	Progress monitoring/debug	Unrestricted
Private Test	10,000	Leaderboard ranking	Restricted/API

All data splits (except the private test set) are accessible on Hugging Face after license agreement (https://huggingface.co/datasets/rajpurkarlab/ReXGradient-160K), with detailed metadata schemas and preprocessing instructions. Official ReXrank leaderboard standings derive exclusively from the held-out private test set, which is accessed through secure submission to https://rexrank.ai (Zhang et al., 1 May 2025, Zhang et al., 2024). The broader evaluation corpus incorporates three additional public datasets: MIMIC-CXR, IU-Xray, and CheXpert Plus, harmonized via schema unification and standardized splits (Zhang et al., 2024).

2. Evaluation Metrics and Protocols

ReXrank implements an extensive suite of metrics to evaluate model output. For chest radiograph reporting, assessment covers both linguistic fidelity and clinical accuracy, through eight complementary measures (Zhang et al., 2024):

1. BLEU-2: Quantifies bigram overlap with brevity penalty.
2. BERTScore: Measures contextual similarity via pretrained BERT embeddings.
3. SembScore: Vector similarity of CheXbert-extracted 14-pathology indicators.
4. RadGraph-F1: Entity/relation overlap parsed by RadGraph.
5. RadCliQ-v1: Composite metric aggregating BLEU-2, BERTScore, SembScore, and RadGraph-F1; inverted for ranking so $1/\mathrm{RadCliQ-v1}$ is “higher is better.”
6. RaTEScore: Entity-aware diagnostic accuracy with learned weights.
7. GREEN: LLM-based clinically significant error count, scaled by sentence count.
8. FineRadScore: LLM-assigned line-level correction severity, inverted for ranking.

Models can be evaluated on findings-only or combined findings+impression tasks by restricting/concatenating candidate and reference texts. All metrics are normalized for ranking consistency and reported with mean $\pm$ 95% confidence intervals ($CI=1.96\cdot\sigma/\sqrt{N}$) (Zhang et al., 2024). For classification performance, ReXrank reports accuracy, micro-averaged F1-score, and AUROC, defined as

$$\mathrm{AUROC} = \int_0^1 \mathrm{TPR}(\mathrm{FPR})\,d(\mathrm{FPR})$$

ranking models on overall correctness, sensitivity-specificity balance, and performance across both common and rare findings (Zhang et al., 1 May 2025).

3. Leaderboard Architecture and Submission Process

The ReXrank leaderboard (https://rexrank.ai) allows researchers to submit Dockerized inference routines or JSON prediction files for both findings-only and findings+impression tasks across all datasets (Zhang et al., 2024). Submissions are automatically evaluated on secure servers against the private test set for ReXGradient and respective splits for public datasets. Aggregate model rankings default to $1/\mathrm{RadCliQ-v1}$, with secondary rankings available per metric and dataset. The protocol enforces strict data separation—private test results are only revealed post-submission, precluding data leakage and promoting unbiased generalization assessment (Zhang et al., 1 May 2025, Zhang et al., 2024). Early baselines (e.g., MedVersa, GPT-4V, CheXagent) demonstrate micro F1-scores $\sim0.75$ and AUROC $0.85$–$0.90$ on common findings; distributional challenges emerge for domain-shifted test sets such as CheXpert Plus (Zhang et al., 2024). Leaderboard results are presented with point estimates and confidence intervals for reproducibility.

4. Algorithmic Framework for Entity-Pair Relationship Explanation

In the knowledge-graph context, the ReXrank Benchmark formalizes and operationalizes the enumeration and ranking of relationship explanations between entity pairs. Each explanation is defined as a pair $(p,I)$ where $p=(V',E',\ell',v_s,v_e)$ is a minimal pattern graph and $I$ is the set of instance mappings (Fang et al., 2011). Patterns must exhibit essentiality (each node/edge lies on a simple $v_s$–$v_e$ path) and be non-decomposable (cannot be split into disjoint subgraphs sharing only $v_s$ and $v_e$).

Enumeration proceeds via a two-phase framework:

• PathEnum: Enumerates all simple $u_s$–$u_e$ paths up to size $n-1$.
• PathUnion: Iteratively merges path-patterns into larger minimal patterns, with duplicate and non-minimal structures pruned efficiently.

Ranking explanations leverages several “interestingness” measures:

• $M_\mathrm{size}$: Pattern size
• $M_\mathrm{rw}$: Random-walk–based connectivity
• $M_\mathrm{count}$: Instance cardinality
• $M_\mathrm{monocount}$: Minimum distinct mappings across variables (anti-monotonic; enables top-$k$ pruning)
• $M_\mathrm{pos}$: Distributional rarity via empirical position/z-score

Empirical evaluation reports enumeration time, ranking time under top-$k$ pruning, and user-study–driven relevance scores (DCG metrics). Released reference implementations support reproducible results and encourage submissions of improved enumeration/ranking methods. Gold explanations for held-out entity pairs allow objective Precision@$k$ and NDCG@$k$ benchmarking (Fang et al., 2011).

5. Robustness, Generalization, and Empirical Insights

Analysis across baseline models reveals key trends:

• Domain robustness: Models trained on multiple datasets (e.g., CheXpertPlus_CheX_MIMIC) demonstrate superior generalization compared to single-source variants.
• Task granularity: Generating full reports yields marginal performance drops (except for systems using specialized decoders for findings vs. impressions).
• Test-set difficulty: IU-Xray is consistently easier; CheXpert Plus exhibits higher variance and difficulty due to its small, distributionally distinct test fold.
• Metric stability: ReXGradient’s private test set generates tight confidence intervals (∼0.01), supporting benchmark reliability (Zhang et al., 2024).

In the context of relationship explanation, user studies highlight the efficacy of distributional (rarity-based) measures over simple aggregate or structural measures; 64% of top-5 human-preferred explanations are non-path patterns, underscoring the significance of minimal pattern enumeration (Fang et al., 2011).

6. Extensibility to Additional Modalities and Domains

ReXrank’s modular architecture allows expansion beyond chest radiographs. Potential extensions include:

• CT scans: Multi-section structure with 3D image backbones; domain-specific metrics such as lesion volumetry F1.
• MRI: Integration of functional data and radiology–pathology correlations.
• Ultrasound/mammography: Region proposal–driven descriptors and enhanced entity–relation metrics.

In graph-based explanation, attribute-constrained, personalized, or context-aware ranking can be incorporated, and the protocol accommodates alternate backbone architectures and larger/more diverse knowledge bases (Zhang et al., 2024, Fang et al., 2011). This flexibility positions ReXrank as a template for universal benchmarking of AI-powered interpretation, reporting, and explanation systems.

7. Community Resources and Protocol Governance

ReXrank provides open-source scoring code, reference algorithms, and publicly available data slices (e.g., DBpedia graph subset, anonymized entity-pair logs for relationship explanations) (Fang et al., 2011). Challenge participation is governed by terms of use, data privacy protocols, and a formal submission API. Gold-standard annotations—human-labeled report pairs (for imaging) or explanation sets (for knowledge graphs)—form the basis for leaderboard ranking and qualitative assessment. Continuous community engagement is encouraged through extension proposals, method submissions, and leaderboard participation.

Collectively, the ReXrank Benchmark establishes a comprehensive, transparent infrastructure advancing research in medical imaging AI and explainable knowledge-base reasoning. Its protocols ensure reproducibility, scalability, and clinical relevance, with evaluation grounded in expansive data resources, rigorous stratification, and multifaceted metric suites (Zhang et al., 1 May 2025, Zhang et al., 2024, Fang et al., 2011).