ASRS for CXR: Augmentation-Sensitivity Risk Scoring

• ASRS is a label-free framework that quantifies representation instability under small rotational augmentations to identify error-prone chest radiograph cases.
• It employs a RAD-DINO encoder to compute embedding shifts and stratifies cases into stability quartiles, revealing significant drops in recall despite high confidence and AUROC.
• This method supports selective prediction by deferring high-risk cases for secondary review, thereby addressing hidden failures not captured by standard calibration metrics.

Searching arXiv for the specified paper to ground the article in the source record. Augmentation-Sensitivity Risk Scoring (ASRS) is a label-free framework for identifying error-prone chest radiograph (CXR) cases by quantifying how much an image representation changes under small, clinically plausible rotations. It was introduced to address a specific reliability gap in CXR classifiers: models can exhibit strong aggregate performance, high confidence, and high AUROC while still missing positives in particular within-distribution regimes or patient subgroups. In the reported formulation, ASRS computes embedding shifts with a RAD-DINO encoder under rotations of $\pm 15^\circ$ and $\pm 30^\circ$, and uses the resulting sensitivity score to stratify cases into stability quartiles; the most sensitive quartile exhibits substantially lower recall, often by approximately $-0.2$ to $-0.3$, even when AUROC and confidence remain high (Shu et al., 2 Oct 2025).

1. Problem setting and motivation

ASRS is motivated by the observation that fairness and reliability concerns persist in deep learning models for CXR interpretation despite strong aggregate metrics. In the reported setting, overall AUROC and mean confidence can mask “hidden failures,” particularly when the relevant errors arise from subtle within-distribution instabilities rather than large domain shifts. The paper frames these failures as overconfident failures: cases in which the model assigns high confidence to incorrect predictions, especially missed positives (Shu et al., 2 Oct 2025).

The central claim is not that confidence calibration or out-of-distribution (OOD) detection is universally ineffective, but that these approaches predominantly target global miscalibration or large distribution shifts. In contrast, ASRS is designed for subtle perturbations such as acquisition parameters, patient positioning, and minor rotations. The empirical premise is that such perturbations can degrade sensitivity without producing an obvious deterioration in confidence or AUROC.

In this formulation, the risk signal is representation instability under clinically plausible transforms. A large embedding shift under small rotations is treated as evidence that the image may be diagnostically unstable for downstream models. This suggests a distinct operational perspective: rather than asking whether a sample is globally out of distribution, ASRS asks whether the sample is locally unstable in representation space under perturbations that should be benign.

2. Mathematical formulation and computational procedure

Let $f(\cdot)$ denote the RAD-DINO representation function. For an image $x$, the framework computes an embedding for the original image and embeddings for augmented images:

$$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$

The augmentation set consists of four small rotations,

$$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$

These are described as clinically plausible augmentations for CXR sensitivity analysis (Shu et al., 2 Oct 2025).

Distance in embedding space is measured with the Euclidean metric,

$d(u,v) = \|u - v\|_2.$

The augmentation-sensitivity score is then defined as

$$S(x) = \sum_{k=1}^{K} d\!\left(e_k, e_0\right) \;=\; \sum_{k=1}^{K} \big\| f\!\big(T_k(x)\big) - f(x) \big\|_2.$$

No additional normalization, such as z-scoring, is applied to $\pm 30^\circ$0 in the reported experiments.

Operationally, each score requires $\pm 30^\circ$1 encoder forward passes per image: one for the original image and four for the rotated variants. The implementation guidance is correspondingly simple. For each image, one computes $\pm 30^\circ$2, then $\pm 30^\circ$3, then the four Euclidean distances, and finally their sum. The paper notes that batching and runtime are encoder-dependent, and does not report precise runtimes.

For stratified analysis, ASRS uses validation-anchored quartile thresholds. Given validation scores $\pm 30^\circ$4, thresholds are defined as

$\pm 30^\circ$5

Test samples are assigned to quartiles by

$\pm 30^\circ$6

The specific numeric values of $\pm 30^\circ$7, $\pm 30^\circ$8, and $\pm 30^\circ$9 are not reported, because they depend on the validation distribution.

3. Encoder choice, dataset, and evaluation setting

RAD-DINO is described as a self-supervised, contrastive visual encoder trained on medical images, specifically chest radiographs for this study, to produce stable, semantically meaningful representations under typical image augmentations. Its embeddings lie in a 768-dimensional feature space, consistent with a ViT-like DINO architecture. In the ASRS workflow, RAD-DINO is used to compute embeddings for risk scoring; downstream diagnostic classifiers include RAD-DINO-based, ResNet50-based, and CXR-MAE-based models (Shu et al., 2 Oct 2025).

The experiments use MIMIC-CXR-JPG frontal views (PA/AP), with patient-level 60%/20%/20% train/validation/test splits. The reported counts are 30,238 patients, 117,686 studies, and 131,283 images for training; 10,079 patients, 38,846 studies, and 43,335 images for validation; and 10,081 patients, 38,444 studies, and 42,918 images for testing. Four binary diagnostic tasks are evaluated: Pneumothorax with prevalence 4.9%, Cardiomegaly with prevalence 20.4%, Pleural Effusion with prevalence 23.8%, and Edema with prevalence 12.4%.

Predictions are thresholded at 0.5 for precision and recall unless otherwise specified. Preprocessing is described only partially: the analysis is restricted to frontal views, and the ASRS perturbations are rotations of $-0.2$0 and $-0.2$1. Resolution and normalization specifics are not detailed.

The choice of RAD-DINO is tied directly to the representation-stability hypothesis. A representation trained for augmentation consistency should remain relatively invariant under benign transforms. Accordingly, large embedding shifts under small rotations are interpreted as indications that features, artifacts, or acquisition parameters are unstable for the model. This suggests that ASRS is fundamentally a representation-level consistency method rather than a prediction-level uncertainty score.

4. Empirical performance and the structure of overconfident failure

The core empirical result is that highly sensitive images—those in G4, the most sensitive quartile—have substantially lower recall, while AUROC and confidence remain high. This pattern holds across three architectures and four diagnostic tasks (Shu et al., 2 Oct 2025).

For RAD-DINO, the reported G1 versus G4 comparisons are as follows: for Cardiomegaly, recall decreases from $-0.2$2 to $-0.2$3 while AUROC increases from $-0.2$4 to $-0.2$5; for Edema, recall decreases from $-0.2$6 to $-0.2$7 while AUROC increases from $-0.2$8 to $-0.2$9; for Pneumothorax, recall decreases from $-0.3$0 to $-0.3$1 while AUROC increases from $-0.3$2 to $-0.3$3; and for Pleural Effusion, recall decreases from $-0.3$4 to $-0.3$5 while AUROC increases from $-0.3$6 to $-0.3$7. The associated recall deficits are $-0.3$8, $-0.3$9, $f(\cdot)$0, and $f(\cdot)$1, respectively.

For ResNet50, the same pattern appears with larger deficits in some tasks: Cardiomegaly declines from $f(\cdot)$2 in G1 to $f(\cdot)$3 in G4; Edema from $f(\cdot)$4 to $f(\cdot)$5; Pneumothorax from $f(\cdot)$6 to $f(\cdot)$7; and Pleural Effusion from $f(\cdot)$8 to $f(\cdot)$9. For CXR-MAE, the corresponding declines are from $x$0 to $x$1 for Cardiomegaly, $x$2 to $x$3 for Edema, $x$4 to $x$5 for Pneumothorax, and $x$6 to $x$7 for Pleural Effusion. These values align with the abstract’s summary that the most sensitive quartile shows recall drops of approximately $x$8 to $x$9.

The paper also reports prevalence-adjusted comparisons via resampled recall and AUROC, denoted Rec(R) and AUC(R), to align subgroup prevalence to G4. Even after this resampling, the recall gap persists. A specific example is RAD-DINO Pneumothorax, where Rec(R) is $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$0 for G1, $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$1 for G2, $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$2 for G3, and $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$3 for G4.

A second central observation is that confidence increases rather than decreases in the most sensitive quartile. For RAD-DINO Pleural Effusion, overall confidence rises from $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$4 in G1 to $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$5 in G4. For RAD-DINO Cardiomegaly, it rises from $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$6 to $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$7. For RAD-DINO Pneumothorax, it rises from $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$8 to $$e_0 = f(x) \in \mathbb{R}^{768}, \quad e_k = f(T_k(x)) \in \mathbb{R}^{768}, \quad k = 1,\dots,K.$$9. This is the empirical basis for the term overconfident failures in the study: the cases with the lowest recall can simultaneously be those with the highest confidence.

Explicit calibration metrics such as ECE, NLL, or Brier score are not reported. The paper therefore does not establish formal miscalibration in the narrow metric sense. Instead, it documents a mismatch between confidence and error risk in highly sensitive cases. A plausible implication is that ASRS reveals a failure mode that is not adequately summarized by ranking quality or confidence magnitude alone.

5. Selective prediction, triaging, and deployment logic

ASRS is directly connected to selective prediction through a risk-gating mechanism. Given a target coverage $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$0, a threshold $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$1 is selected from validation data such that

$$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$2

At deployment, the decision rule is

$$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$3

The interpretation is straightforward: low-sensitivity cases are auto-accepted, while high-sensitivity cases are routed for clinician review or secondary checks (Shu et al., 2 Oct 2025).

The quartile construction provides concrete examples of this policy. Choosing $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$4 implements a 25% defer policy that accepts the most stable 75% of cases. Choosing $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$5 defers the top 25% most sensitive cases, namely G4, which the paper shows are disproportionately error-prone. Although explicit risk–coverage curves are not reported, the quartile results demonstrate the principle that deferring G4 can reduce hidden failures because G4 consistently exhibits lower recall despite high AUROC and confidence.

The implementation guidance also specifies monitoring targets. The recommended logging includes coverage $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$6, deferred volume, subgroup distributions across ASRS quartiles, and downstream recall and precision on accepted versus deferred cases. The paper further recommends periodic recalibration of $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$7 as casemix or acquisition parameters evolve.

This deployment framing is notable because it does not require labels at inference time. ASRS therefore functions as a label-free triage score. In the study’s framing, this makes it suitable for safety-oriented review pipelines in which a subset of cases is flagged for additional oversight rather than rejected by a hard abstention rule.

6. Fairness relevance, comparisons, limitations, and possible extensions

ASRS intersects with subgroup fairness because demographic composition shifts across stability quartiles. On the test set, mean age declines from 64.83 years in G1 to 53.90 years in G4, a difference of $$\{T_k\}_{k=1}^K = \{\mathrm{Rot}(-30^\circ), \mathrm{Rot}(-15^\circ), \mathrm{Rot}(+15^\circ), \mathrm{Rot}(+30^\circ)\}, \quad K=4.$$8 years. The percentage of White patients declines from 67.33 to 61.66, while the percentage of Black patients increases from 14.05 to 20.07 and the percentage of Hispanic/Latino patients increases from 4.04 to 8.97 (Shu et al., 2 Oct 2025).

These shifts are important but carefully qualified in the paper. ASRS is said to intersect with, but not be reducible to, demographic or acquisition heterogeneity. In other words, the score does not simply encode demographic membership. Rather, it reveals uneven performance that is not apparent in aggregate metrics and that appears to co-vary with demographic and acquisition structure.

In relation to baseline approaches, the paper acknowledges confidence-based error detection, OOD detection such as ODIN and energy scores, and conventional augmentation-consistency tests. The comparative argument is indirect: ASRS-defined G4 cases retain high confidence and AUROC yet show large recall deficits across all three architectures and four tasks. This is presented as evidence that ASRS is useful for discovering subtle, within-distribution overconfident failures that confidence-based or OOD methods may miss.

Several limitations are explicit. Sensitivity to rotation may not always equate to clinical error; anatomical or device-specific acquisition factors could drive embedding shifts without affecting clinical detectability. ASRS may capture artifacts, projection variability between PA and AP, or device differences. While this may still be useful for flagging risk, it also introduces potential confounders. Generalizability beyond CXR is treated cautiously and said to require careful selection of clinically plausible transforms and validation in other modalities. Formal statistical tests, bootstrap confidence intervals, and p-values are not reported, so the conclusions rely on descriptive stratified performance gaps. Likewise, no ablations are reported for alternative augmentation sets, alternative distance metrics such as cosine distance, alternative aggregators such as mean, max, or median, or alternative encoders beyond the three considered classifiers.

The paper nevertheless outlines a broader extension space. ASRS is described as applicable to other modalities, including CT, MRI, fundus imaging, and dermoscopy, provided that modality-specific, clinically plausible transforms are chosen. Self-supervised domain encoders such as MAE, SimCLR, and DINO variants are identified as natural candidates, and the same representation-sensitivity principle is proposed for tasks beyond binary classification, including segmentation, detection, and multi-label settings. This suggests that ASRS is best understood not as a CXR-specific heuristic, but as a general strategy for converting local representation instability into a deployment-time risk signal.