Uncertainty-Aware CheXpert Labels

• The paper introduces uncertainty-aware CheXpert labels by systematically extracting and aggregating diagnostic uncertainty from radiology reports using a multi-stage NLP pipeline.
• It evaluates various uncertainty-handling strategies such as U-Ones, U-Zeros, U-SelfTrained, and U-MultiClass along with label smoothing and probabilistic encoding to achieve mean AUC improvements up to 0.940.
• The findings imply that leveraging uncertainty through hierarchy-aware conditional training and soft-label propagation improves clinical realism, statistical robustness, and overall classification performance.

Uncertainty-aware CheXpert-style labels formalize and systematically exploit diagnostic uncertainty present in radiology report–derived labels for chest X-ray (CXR) datasets. In the CheXpert paradigm, each CXR study is automatically annotated with a 14-dimensional vector, where each entry encodes the presence (1), absence (0), or uncertainty (–1/u/U) of a specific thoracic finding based on a multi-stage rule-driven NLP pipeline. Recent research advances have developed numerous strategies for integrating these uncertainty labels into neural network training, leveraging explicit hierarchy structures, probabilistic encoding, and soft-label regularization to improve clinical realism, statistical robustness, and generalizability in multi-label CXR classification.

1. CheXpert Annotation Pipeline and Uncertainty Extraction

The canonical CheXpert pipeline processes free-text radiology reports using three core phases: mention extraction (via large curated phrase lists for 14 findings), mention classification (detecting negation and pre-/post-negation uncertainty using regular expressions and, optionally, dependency parsing), and mention aggregation, assigning each label one of three atomic values: “positive” (1), “negative” (0), or “uncertain” (u or –1) (Irvin et al., 2019). An explicit aggregation rule gives precedence to positive > uncertain > negative, ensuring that “cannot exclude pneumonia” yields ‘uncertain’, while an outright negative phrase yields ‘negative’.

Performance benchmarks for the rule-based labeler reach micro-F1 = 0.969 (extraction), 0.952 (negation), and 0.848 (uncertainty) (Irvin et al., 2019). CheXpert-style uncertain labels have been ported to other languages and clinical contexts, e.g. the German CheXpert adaptation (Wollek et al., 2023), which integrates language-specific phrase repositories and a NegEx-algorithm variant, achieving mention F1 up to 0.995 and robust performance in downstream image classification.

2. Approaches to Uncertainty Label Handling

Several competing strategies have been evaluated for propagating uncertainty labels into neural network training objectives:

• U-Ignore: Discard all uncertain cases from the loss for a given label.
• U-Ones / U-Zeros: Map all uncertain cases to hard positive (1) or negative (0) labels, respectively.
• U-SelfTrained: Initial model is trained with U-Ignore; the resulting output for each uncertain sample (a soft probability) becomes the new “pseudo-label” for further training.
• U-MultiClass: Each observation is modeled as a 3-way problem (negative, positive, uncertain), generally involving a softmax and multiclass cross-entropy loss.

A summary of their characteristics is given below:

Approach	Loss Function	Uncertainty Propagation
U-Ignore	BCE (mask u)	Uncertainty samples omitted
U-Ones / U-Zeros	BCE (u→1/0)	Uncertainty mapped to hard cls
U-SelfTrained	BCE w/ pseudo-lab	Initial outputs for u injected
U-MultiClass	Softmax CE	3-class output per obs

Empirical results indicate significant label/task-specific variability: U-Ones is preferable for findings where clinical experience equates hedged language with presence (e.g. Atelectasis, Edema); U-Zeros is optimal for Consolidation; U-MultiClass is valuable for finding types where borderline semantics (e.g. Cardiomegaly, Effusion) must be differentiated (Irvin et al., 2019). U-SelfTrained yields improvements contingent on initial model accuracy (Irvin et al., 2019).

3. Label Smoothing Regularization and Probabilistic Encoding

Label smoothing regularization (LSR) improves upon hard-mapping by converting each uncertainty (-1) label into a randomly drawn value from an appropriate interval in (0,1), reducing systematic overconfidence and propagating soft supervision (Pham et al., 2019, Pham et al., 2020). Specifically, for each uncertain label element $y_k^{(i)} = -1$, the smoothed target is $\bar y_k^{(i)} = u$, with $u \sim \mathcal{U}(a,b)$. In the “U-Ones+LSR” regime, $u \sim \mathcal{U}(0.55,0.85)$; for “U-Zeros+LSR,” $u \sim \mathcal{U}(0,0.3)$ (Pham et al., 2019). The standard sigmoid binary cross-entropy loss is then computed with these smoothed targets. Empirically, combining LSR with conditional training yields mean AUC = 0.894 (vs. 0.874, U-Ones+LSR only; 0.872, conditional only) on CheXpert validation, and the final ensemble with label smoothing and hierarchy-aware training achieves mean AUC = 0.940 on validation and 0.930 on the hidden test set—outperforming 2.6 of 3 individual board-certified radiologists (Pham et al., 2019, Pham et al., 2020).

Alternative soft-label schemes have recently emerged:

• Randomized Flipping (Agarwal et al., 12 Apr 2025): Each uncertain label is randomly assigned to 0 or 1 with $P(1) = 0.5$ each epoch, thus introducing stochastic regularization while maintaining representation for ambiguous examples. This method improved AUROC by up to 2% on certain pathologies compared to U-Ignore.
• Generalized Label Smoothing (GLS) (Zhang et al., 4 Aug 2025): Explicitly ties the amount of smoothing to a seven-point expert uncertainty scale $u \in \{-3,\ldots,+3\}$, with smoothing rate $r_n = -k|u_n| + r_0$, and flips the label if $u_n<0$. Highly ambiguous cases correspond to maximum regularization; highly confident labels induce “negative smoothing,” i.e., more peaky targets, improving noise robustness and clinical interpretability.

4. Hierarchy-Aware Conditional Training and Disease Dependency Modeling

Clinical knowledge embeds strong dependencies: for example, certain findings (e.g. Pneumonia) are pathologically nested under others (e.g. Consolidation, which is itself under Lung Opacity). Pham et al. (Pham et al., 2019, Pham et al., 2020) devised a two-stage conditional training protocol:

• Stage 1 (Conditional Pretraining): Restrict the training subset for each disease $k$ to those images where all parent labels (in a pre-defined DAG) are positive; train the model to estimate $p(y_k=1\mid \text{parents}(k)=1, x)$.
• Stage 2 (Full-Data Fine-Tuning): Freeze all layers except for the final classification head; reintroduce the whole dataset and train on all samples.

At inference, unconditional probabilities are recovered by recursively multiplying conditional probabilities along the DAG. This ensures local monotonicity (i.e., a child node’s predicted probability never exceeds that of its parent) and endows the classifier with explicitly structured, clinically-grounded reasoning. This approach contributed an absolute mean AUC gain of 0.034 (4% relative) over the next best non-hierarchical LSR baseline (Pham et al., 2019).

Recent extensions exploit multi-relationship graph learning incorporating spatial, semantic, and implicit topologies; expert-uncertainty–aware loss further propagates soft labels based on an uncertainty mapping (e.g., $p_{\text{ex}} \in \{1.0, 0.7, 0.5, 0.3, 0.1, 0.0\}$) to each disease finding (Zhang et al., 2023).

5. Probabilistic, Differentiable Labelers and Uncertainty Quantification

CheXpert++ (McDermott et al., 2020) introduced a transformer-based (BERT) approximation to the CheXpert rule-based labeler, outputting per-finding probabilities $p_i \in (0,1)$. Uncertainty is quantified via predictive entropy $H(p_i) = -[p_i\log p_i + (1-p_i)\log(1-p_i)]$, enabling well-calibrated, differentiable uncertainty-aware labels at scale. CheXpert++ achieves 99.81% fidelity to the original CheXpert outputs and provides utility for active learning—e.g., targeting cases with highest entropy for expert relabeling, yielding an 8% accuracy improvement in proof-of-concept studies (McDermott et al., 2020).

The Pseudo-D approach (Gu et al., 15 Sep 2025) leverages neural network training dynamics (NNTD), calculating instance-level sample difficulty scores ($S_i$; e.g., entropy of epoch-averaged output), calibrating ($g(S_i) \rightarrow U_i$) these into probabilistic uncertainty estimates, and thresholding into CheXpert-style discrete labels. This methodology generalizes automated uncertainty-aware label assignment to settings lacking report-derived labels.

Recent frameworks integrating LLM-guided hedging phrase ranking, probabilistic mapping, and explicit pathway-driven sub-finding expansion (Lunguage++) provide further granularity: replacing “tentative” binary flags with a continuous probability $p \in [0,1]$ reflecting calibrated certainty, thus supporting uncertainty-aware classifier supervision and enabling the computation of uncertainty-aware metrics (Rabaey et al., 6 Nov 2025).

6. Quantitative Impact on Diagnostic Performance

The application of uncertainty-aware CheXpert-style labeling and integration strategies directly correlates with improved classifier robustness and calibration:

• Mean AUC (validation; state-of-the-art system): 0.940, test: 0.930 (Pham et al., 2019).
• LU-ViT + GLS achieves AUROC improvements up to 1–2 points over earlier baselines, outperforming in 13/14 clinical pathologies (Zhang et al., 4 Aug 2025).
• Classifiers trained on expert-soft labels (e.g., $p_{\text{ex}}$), as opposed to 0/1 targets, exhibit mean AUC >0.8 and improved Top-5/10 accuracy (Zhang et al., 2023).
• Randomized flipping, as an uncertainty-handling strategy, increases AUROC for challenging pathologies, especially in low-resolution settings (Agarwal et al., 12 Apr 2025).

In sum, methods that systematically propagate and calibrate diagnostic uncertainty—rather than masking or naively binarizing—demonstrate improved discrimination, reliability, and clinical fidelity in large-scale CXR classification tasks.

7. Best Practices and Ongoing Challenges

Optimal utilization of uncertainty-aware CheXpert-style labels requires:

• Pathology-specific mapping: Different uncertainty-handling strategies (U-Ones, U-Zeros, U-MultiClass) are empirically optimal for different pathologies and should be selected per label (Irvin et al., 2019).
• Hierarchy-aware modeling: Conditional training along a clinical finding DAG outperforms flat classification for structure-dependent observations (Pham et al., 2019).
• Soft label propagation: Probabilistic or soft labels (via LSR, randomized flipping, GLS) are generally superior to hard-mapping or ignoring uncertainty, with the smoothing magnitude tailored to empirical or expert-supplied uncertainty (Pham et al., 2019, Zhang et al., 4 Aug 2025).
• Explicit reporting of calibration: Evaluation of AUROC, calibration error, and attention localization (e.g., Grad-CAM) are crucial to ascertain both the clinical and statistical realism of uncertainty-aware models (Zhang et al., 4 Aug 2025, Pham et al., 2019).
• Adaptation to new languages/domains: Language- and context-specific phrase lists, as well as iterative, human-in-the-loop refinement, are required to maintain performance and transparency in international or novel data settings (Wollek et al., 2023).

Persistent challenges include defining semantic thresholds for binning continuous probabilities into discrete “uncertain” categories, integrating implicit uncertainty (e.g., omitted reasoning) into label pipelines, and aligning probabilistic output with end-use clinical thresholds. Future research directions encompass LLM-based uncertainty calibration, continuous-valued label supervision, and causal reasoning chain expansion for enhanced explainability (Rabaey et al., 6 Nov 2025).