CheXpert: Chest X-ray Benchmark & Labeling Framework

• CheXpert is a comprehensive chest X-ray dataset and labeling framework featuring 224K radiographs from 65K patients with 14 annotated observations.
• The framework leverages a rule-based NLP pipeline and its BERT-based variant to explicitly preserve uncertainty, enabling nuanced label strategies.
• CheXpert Plus extends the ecosystem into a multimodal radiology corpus with paired reports, DICOM images, metadata, and RadGraph annotations.

CheXpert is a public chest-radiograph benchmark and report-labeling framework centered on automated interpretation of chest X-rays. In its original release, it contains 224,316 chest radiographs from 65,240 patients and annotates 14 observations by running a rule-based natural-language-processing pipeline over radiology reports, explicitly preserving uncertainty rather than collapsing it into a binary label space (Irvin et al., 2019). The resource is notable not only as a dataset, but as an ecosystem: it includes patient-disjoint evaluation splits with radiologist reference annotations, a widely reused label ontology, multiple uncertainty-handling protocols, downstream model comparisons against experts, and later extensions such as CheXpert Plus, which adds paired reports, DICOM images, demographics, metadata, and RadGraph annotations at scale (Garbin et al., 2021, Chambon et al., 2024).

1. Corpus design and released resources

The original CheXpert release is described as comprising 224,316 chest radiographs from 65,240 patients, with frontal and lateral views represented and 14 observation labels attached to each study (Irvin et al., 2019). The datasheet further characterizes the images as digital chest X-rays in DICOM, originally 12- to 16-bit grayscale, converted to 8-bit JPEG/PNG for release, resized and center-cropped to $320\times320$ pixels, with pixel-intensity histogram equalization and de-identification applied (Garbin et al., 2021). The benchmark design includes a large training split plus a validation set of 200 studies and a held-out test set of 500 studies, with no patient appearing in more than one split (Garbin et al., 2021).

CheXpert Plus extends the resource from an image benchmark into a multimodal radiology corpus. It contains 223,228 unique chest X-ray images in DICOM format, corresponding PNG exports, 187,711 unique radiology reports, and 64,725 distinct patients; each image carries up to 47 metadata fields, each study has 14 pathology labels, and Findings and Impression sections include pre-computed RadGraph annotations (Chambon et al., 2024).

Resource	Scale	Core contents
CheXpert	224,316 radiographs; 65,240 patients	14 observations, uncertainty labels, radiologist-annotated validation/test
CheXpert Plus	223,228 images; 187,711 reports; 64,725 patients	DICOM + PNG, reports, metadata, RadGraph, model zoo

The significance of this design lies in the combination of large-scale weak supervision and small, high-quality expert evaluation sets. That combination made CheXpert both a training resource and a standardized benchmark for model comparison, while CheXpert Plus broadened the benchmark toward VLMs, report generation, fairness analysis, and multimodal pretraining (Irvin et al., 2019, Chambon et al., 2024).

2. Label ontology and report labeling pipeline

CheXpert’s label space consists of 14 observations: Enlarged Cardiomediastinum, Cardiomegaly, Lung Opacity, Lung Lesion, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture, Support Devices, and No Finding (Irvin et al., 2019). For each observation, the report labeler outputs Positive, Negative, Uncertain, or blank if the report does not mention the finding (Irvin et al., 2019). The special “No Finding” label is positive if and only if none of the other 13 observations is Positive or Uncertain (Irvin et al., 2019).

The original labeler is a three-stage rule-based NLP pipeline operating on the Impression section. First, mention extraction uses a curated dictionary of phrases. Second, mention classification applies three sequential rule sets over a Universal Dependency parse: pre-negation uncertainty rules, negation rules, and post-negation uncertainty rules. Third, mention aggregation maps all mentions for each observation into a single study label by precedence: Positive overrides Uncertain, which overrides Negative, which overrides blank (Irvin et al., 2019). The datasheet emphasizes that this explicit modeling of uncertainty is a defining feature of the dataset (Garbin et al., 2021).

A major subsequent development is CheXpert++, a BERT-based approximation to the rule-based labeler. It attaches 14 task-specific linear heads to a clinical BERT encoder, with four output classes per task: no mention, uncertain, negative, and positive. CheXpert++ achieves 99.81% parity with the original CheXpert labeler, processes approximately 680K MIMIC-CXR sentences in about 1.53 hours on one GPU versus about 2.75 hours for the original rule-based system on 32 CPU processes, and provides differentiable probabilistic outputs instead of deterministic regex-based labels (McDermott et al., 2020). In disagreement analysis, a board-certified radiologist preferred CheXpert++ in 60% of sampled disagreements versus 27% for the original labeler, and a one-iteration active-learning-inspired retraining step improved accuracy on a clinician-labeled gold set from 72.0% to 79.1% (McDermott et al., 2020).

This progression from symbolic rules to neural approximation illustrates an important feature of the CheXpert ecosystem: the labeler is itself an object of study, not merely a preprocessing utility.

3. Uncertainty labels, evaluation protocol, and expert comparison

Uncertainty is central to CheXpert. The original benchmark investigates several training strategies for labels marked $u$: U-Ignore, U-Zeroes, U-Ones, U-SelfTrained, and U-MultiClass (Irvin et al., 2019). U-Ignore masks uncertain labels out of the loss; U-Zeroes maps uncertainty to 0; U-Ones maps it to 1; U-SelfTrained first trains under U-Ignore and then replaces uncertain labels with model probabilities; U-MultiClass treats $\{0,1,u\}$ as a three-class target and renormalizes the binary probability at inference (Irvin et al., 2019). The datasheet complements this with operational guidance: thresholds $\tau\in[0,1]$ may be selected on the validation split to maximize $F_1$, and evaluation commonly reports AUROC, AUPRC, sensitivity, specificity, and $F_1$ (Garbin et al., 2021).

The validation set contains 200 studies manually annotated by 3 board-certified radiologists, and the test set contains 500 studies annotated by consensus of 5 board-certified radiologists, with 3 additional radiologists used for human-performance comparison (Irvin et al., 2019, Garbin et al., 2021). The original CheXpert study evaluates five competition tasks—Atelectasis, Cardiomegaly, Consolidation, Edema, and Pleural Effusion—and shows that the most effective uncertainty strategy depends on the pathology. On validation, the best reported AUCs are 0.858 for Atelectasis under U-Ones, 0.854 for Cardiomegaly under U-MultiClass, 0.939 for Consolidation under U-SelfTrained, 0.941 for Edema under U-Ones, and 0.936 for Pleural Effusion under U-MultiClass (Irvin et al., 2019).

On the 500-study test set, the reported ROC AUCs are 0.85 for Atelectasis, 0.93 for Cardiomegaly, 0.90 for Consolidation, 0.95 for Edema, and 0.97 for Pleural Effusion (Irvin et al., 2019). For Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all three radiologist operating points; for Consolidation, the model outperforms two of three clinicians; for Atelectasis, the clinicians outperform the model (Irvin et al., 2019). These results established CheXpert as a benchmark in which label uncertainty is not peripheral annotation noise but an explicit modeling variable.

A recurring misconception is that CheXpert provides only binary targets. It does not: uncertainty is a first-class label in the source formulation, and much of the benchmark’s methodological literature concerns how to exploit or suppress that state during training (Irvin et al., 2019, Garbin et al., 2021).

4. Modeling paradigms built on CheXpert

The original CheXpert benchmark used DenseNet-121 as its principal image model, with all images resized to $320\times320$, intensity-normalized, and study-level probabilities computed as the maximum over available views when both frontal and lateral images are present (Irvin et al., 2019). Subsequent work broadened the modeling repertoire substantially.

A notable line of work exploits disease dependencies. Pham et al. train ImageNet-pretrained CNNs—including DenseNet-121, DenseNet-169, DenseNet-201, Inception-ResNet-v2, Xception, and NASNet-Large—under a conditional training scheme defined by a clinical hierarchy over the 14 labels (Pham et al., 2019). If $A\rightarrow B\rightarrow C$ is a chain in the hierarchy, inference uses

$p(C)=p(A)\cdot p(B\mid A)\cdot p(C\mid B),$

which guarantees $p(\text{child})\leq p(\text{parent})$ (Pham et al., 2020). The same work introduces label smoothing regularization for uncertain samples, replacing $u$0 with random targets near 1 under U-Ones+LSR or near 0 under U-Zeros+LSR (Pham et al., 2019). Their ensemble reaches a mean AUC of 0.940 on the CheXpert validation set for five selected pathologies and 0.930 on the hidden 500-study test set, ranking first on the leaderboard at the time of writing (Pham et al., 2019).

Architecture-transfer studies on CheXpert complicate common assumptions about pretraining. CheXtransfer compares 16 ImageNet-style CNNs and reports no significant correlation between ImageNet top-1 accuracy and CheXpert AUROC, whether the models are pretrained or randomly initialized. At the same time, ImageNet pretraining provides a statistically significant average boost of $u$1 AUROC across architectures, with larger gains for smaller models, and truncating the final block of pretrained models yields an average $u$2 parameter reduction without a statistically significant drop in performance (Ke et al., 2021). In low-data settings, the CheXpert5000 study finds that ImageNet-21k-pretrained BiT and ViT models outperform a ResNet50 baseline on a fixed 5,000-image regime, and that MixUp and Mean Teacher improve calibration, with MixUp also improving AUC (Ihler et al., 2023).

Collectively, these results show that CheXpert has functioned as a testbed for uncertainty learning, hierarchical multi-label inference, transfer learning, checkpoint ensembling, model compression, calibration, and limited-data adaptation.

5. Multimodal expansion: CheXpert Plus, RadGraph, and report generation

CheXpert Plus transforms CheXpert from a predominantly vision benchmark into a large paired radiology corpus. It contains 36,469,132 text tokens under the BERT-base-uncased tokenizer, including 13,351,758 tokens in Impression sections across 187,575 reports, with up to 11 report sections per study (Chambon et al., 2024). The release also adds eight de-identified patient variables per study—Age, Sex, Race, Ethnicity, Insurance type, Body-mass index, Deceased status, and Interpreter needed—explicitly supporting fairness analyses and stratified modeling (Chambon et al., 2024).

Its de-identification pipeline is itself large-scale. The paper reports 853,878 total true PHI spans in reports; after an automated transformer-based pass with “Hide in Plain Sight” replacement and human review by 25–30 reviewers, 23 spans were fully missed and 841 were partially missed, followed by a second automated pass and radiologist review of true-to-synthetic mappings (Chambon et al., 2024). DICOM metadata were stripped, human-reviewed, and checked at the pixel level so that all pixels remained identical to the approved CheXpert 1.0 release (Chambon et al., 2024).

The added RadGraph annotations create a graph-structured semantic layer over Findings and Impression. In Findings, the release includes 1,581,863 entity annotations and 1,106,659 relation annotations; in Impression, it includes 3,999,559 entities and 2,772,337 relations (Chambon et al., 2024). This enables tasks beyond pathology classification, including structured report extraction and report summarization (Chambon et al., 2024).

These extensions have already been used to benchmark medical report generation. CXPMRG-Bench uses CheXpert Plus with patient-level $u$3 splits of 40,463 training, 5,780 validation, and 11,562 test cases, taking the Findings section as ground truth (Wang et al., 2024). It evaluates 21 mainstream MRG algorithms and 16 LLM/VLM-based variants, using both lexical metrics and a clinical-efficacy metric based on the CheXpert labeler (Wang et al., 2024). In that benchmark, MambaXray-VL-Large achieves BLEU-4 of 0.112 and clinical-efficacy $u$4 of 0.335 on the CheXpert Plus test set, with ROUGE-L 0.276, METEOR 0.157, CIDEr 0.139, 55 minutes test time, and 202M trainable parameters (Wang et al., 2024).

A plausible implication is that CheXpert’s role has shifted from a benchmark for image classification alone to a substrate for report-grounded representation learning, generative modeling, and clinically structured evaluation.

6. Generalization, domain shift, and contemporary failure analyses

CheXpert-trained models have been studied extensively under distribution shift. CheXpedition re-ran the top 10 publicly available CheXpert competition models on three out-of-distribution tasks: tuberculosis detection, pathology detection on smartphone photographs of X-rays, and pathology detection on an external institution’s data (Rajpurkar et al., 2020). Across two public TB datasets, the models achieve mean AUC 0.851 without TB-specific fine-tuning; on smartphone photos of the CheXpert test set, mean AUC is 0.916 versus 0.924 on the original images; and on 420 frontal radiographs from NIH ChestX-ray14, mean AUC across the five competition tasks is 0.897, with model sensitivity at mean radiologist specificity meeting or exceeding average radiologist sensitivity for 4 of 5 tasks (Rajpurkar et al., 2020). The same study reports that predicting TB performance from the model’s average CheXpert AUC yields $u$5, whereas using only the Consolidation-task AUC yields $u$6, suggesting that multitask competence transfers more effectively than narrow single-label performance (Rajpurkar et al., 2020).

At the same time, CheXpert has well-documented limitations. The datasheet lists single-health-system provenance, automated label noise, absence of region-level localization labels, adult-only coverage, and potential demographic and disease-prevalence biases as key caveats (Garbin et al., 2021). Independent evaluation on pediatric data reinforces the domain-shift problem: when CheXpert and CheXbert are applied to 95,008 pediatric chest-radiograph reports using the Impression section, both achieve 56% overall exact-match assertion accuracy against a consensus pseudo-ground truth, below SparkNLP at 76%, Azure at 69%, and Google Healthcare NLP at 63% (Hegde et al., 29 May 2025).

Recent work also uses CheXpert protocols to diagnose failure modes in medical VLMs. CXR-ContraBench defines contrastive multiple-choice probes on CheXpert validation and training splits to measure “negated-option attraction,” where a model selects “No $u$7” even when $u$8 is visibly present (Fang et al., 7 May 2026). On the direct presence probe built from filtered CheXpert validation studies, MedGemma-4B-it reaches 31.49% accuracy and Qwen2.5-VL-7B-Instruct 30.21%, with both choosing the negated option on 153 of 235 records; on a 135,754-record CheXpert training-split protocol, the negated-option rate remains above 62% for both models (Fang et al., 7 May 2026). A deterministic post-hoc rule, QCCV-Neg, raises these direct-presence accuracies to 96.60% and 95.32%, respectively, by overriding the polarity-confused subset without retraining (Fang et al., 7 May 2026). This suggests that standard accuracy on aggregate tasks can obscure clinically meaningful inference-time polarity failures.

CheXpert continues to support narrower experimental pipelines as well. One recent study filters the dataset to 12,716 frontal radiographs, drops all studies containing any uncertain label, derives a 4-bit label vector over $u$9, integrates RadGraph parsing with transfer learning across VGG16, ResNet50, and ConvNeXt Large, and reports overall $\{0,1,u\}$0 and macro-AUROC $\{0,1,u\}$1 (Efimovich et al., 2024). The deliberate removal of all uncertain cases in that pipeline highlights an enduring methodological tension: whether uncertainty should be modeled, marginalized, or discarded.

In sum, CheXpert occupies a distinctive place in radiology AI: it is simultaneously a dataset, a label ontology, a weak-supervision pipeline, a model benchmark, and a stress test for robustness, calibration, multimodality, and clinical error analysis.