Robustness Metric for CAMs

• Robustness metrics for CAMs are a set of criteria designed to evaluate the stability and causal fidelity of saliency maps under various input perturbations.
• They quantify structure-aware invariance by measuring alignment with canonical substructures, consistency under corruption, and sensitivity to prediction changes.
• The metrics also incorporate methods to penalize passive fooling and trivial explanations, employing composite scores and spatial drift analyses for comprehensive evaluation.

Searching arXiv for the cited CAM robustness papers to ground the article in current literature. A robustness metric for class activation maps (CAMs) is a quantitative criterion for assessing whether a saliency map remains stable, causally meaningful, and structurally aligned when inputs are perturbed or when explanatory fidelity is probed by controlled interventions. Recent work does not treat robustness as a single universal scalar. In CAMBench-QR, robustness is defined as the stability and invariance of a CAM’s saliency allocation on canonical QR substructures—finder patterns, timing lines, and the QR box—under common input perturbations (Chakraborty et al., 20 Sep 2025). Parallel lines of work define robustness through CAM consistency under corruption (Zhang et al., 8 Jul 2025), through the joint requirements of stability and decision-sensitivity (Sarkar et al., 25 Aug 2025), through resistance to passive fooling (Piland et al., 10 Jun 2025), and through composite faithfulness metrics that penalize trivial or noisy maps (Domeniconi et al., 14 May 2026).

1. Scope and meaning of robustness for CAMs

In the contemporary CAM literature, robustness denotes different but related properties. One formulation emphasizes structure-aware invariance: a robust explanation should keep attribution mass on the parts that make the concept work and avoid background and shortcuts as the image is rotated, warped, blurred, compressed, darkened, or partially occluded (Chakraborty et al., 20 Sep 2025). A second formulation emphasizes clean–corrupted consistency: when the predicted class is preserved, the CAM for the corrupted image should remain close to the CAM for the clean image, especially when training explicitly minimizes an MSE-based alignment loss (Zhang et al., 8 Jul 2025). A third formulation emphasizes consistency and responsiveness simultaneously: a CAM should remain stable under perturbations that do not alter the top-1 prediction, yet change meaningfully when the prediction changes (Sarkar et al., 25 Aug 2025).

A further distinction concerns robustness against adversarial manipulation of explanations. The SHAM benchmark operationalizes passive fooling by fine-tuning a model so that CAMs become misleading while classification accuracy is preserved, making robustness measurable as susceptibility of explanations rather than degradation of task accuracy (Piland et al., 10 Jun 2025). Another line of work evaluates robustness through faithfulness under perturbation protocols, showing that some widely used metrics can be fooled by trivial heatmaps and motivating composite scores such as ARCC that combine perturbation faithfulness, compactness, and self-consistency (Domeniconi et al., 14 May 2026).

These formulations are complementary rather than interchangeable. This suggests that a “robustness metric for CAMs” is best understood as a family of operational criteria, each tied to a specific failure mode: structural drift, corruption inconsistency, insensitivity to decision change, passive fooling, or vulnerability to trivial maps.

2. Structure-aware robustness in CAMBench-QR

CAMBench-QR gives the most explicit structure-aware definition. The image domain is

$\Omega = \{1,\dots,H\}\times\{1,\dots,W\},$

and a CAM produces saliency $C:\Omega\to\mathbb{R}_{\ge 0}$. The benchmark min–max normalizes and mass-normalizes the saliency field: $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$ with $\varepsilon \approx 10^{-6}$ for stability (Chakraborty et al., 20 Sep 2025).

The benchmark then defines binary masks aligned to image coordinates: $M_F$ for finder patterns, $M_T$ for timing lines, $M_B$ for the full QR box, and $\bar{M}_B=\mathbf{1}-M_B$ for background. Using masked inner products $\langle A,B\rangle=\sum_{p\in\Omega}A(p)B(p)$, the core structure-aware mass ratios are

$$\mathrm{FMR}=\frac{\langle \tilde{C},M_F\rangle}{S},\qquad \mathrm{TMR}=\frac{\langle \tilde{C},M_T\rangle}{S},\qquad \mathrm{BL}=\frac{\langle \tilde{C},\bar{M}_B\rangle}{S}.$$

Higher FMR and TMR indicate that saliency mass remains on requisite substructures; lower BL indicates less off-QR leakage. Because these quantities are normalized by total saliency mass, they are scale-invariant and lie in $C:\Omega\to\mathbb{R}_{\ge 0}$0 (Chakraborty et al., 20 Sep 2025).

CAMBench-QR adds threshold-swept coverage statistics. For the superlevel-set indicator

$C:\Omega\to\mathbb{R}_{\ge 0}$1

it computes coverage fractions over finder patterns, timing lines, and background, then averages them over $C:\Omega\to\mathbb{R}_{\ge 0}$2 quantiles of $C:\Omega\to\mathbb{R}_{\ge 0}$3 to obtain coverage AUCs. The benchmark names these aggregated quantities $C:\Omega\to\mathbb{R}_{\ge 0}$4, $C:\Omega\to\mathbb{R}_{\ge 0}$5, and $C:\Omega\to\mathbb{R}_{\ge 0}$6. Quantile-based thresholding is used so that the AUCs are invariant to monotone rescalings of the saliency field (Chakraborty et al., 20 Sep 2025).

Spatial drift is measured by Distance-to-Structure. Let

$C:\Omega\to\mathbb{R}_{\ge 0}$7

where $C:\Omega\to\mathbb{R}_{\ge 0}$8 is the Euclidean distance transform to the nearest structure pixel. The normalized mass-weighted distance is

$C:\Omega\to\mathbb{R}_{\ge 0}$9

Lower DtS indicates tighter spatial alignment of saliency with the union of canonical QR substructures (Chakraborty et al., 20 Sep 2025).

Taken together, these quantities separate three aspects that are often conflated in CAM evaluation: how much attribution falls on required parts, how much leaks into irrelevant regions, and how far the mass drifts spatially even when aggregate mass ratios appear acceptable.

3. Robustness curves, distortions, and evaluation protocol

CAMBench-QR does not treat robustness as a single per-image number. Instead, it tracks structure-aware metrics across controlled distortion sweeps. The distortion suite includes rotation, perspective warp, blur, JPEG compression, low-light, and occlusion, with masks carried forward to transformed coordinates so that ground-truth alignment is preserved under perturbation (Chakraborty et al., 20 Sep 2025).

For a distortion parameterization $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$0 with severity $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$1, and transformation $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$2, the benchmark defines the robustness curve for any metric $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$3 by

$$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$4

It then aggregates the curve with the area-under-robustness-curve

$$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$5

where $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$6 are discrete severity levels. Higher AURC for FMR and TMR means that saliency mass remains on finder patterns and timing lines as severity grows. For leakage, the benchmark also reports the linear-regression slope of BL with respect to severity; lower slope indicates that background leakage grows more slowly under stress (Chakraborty et al., 20 Sep 2025).

The benchmark couples robustness evaluation with two training regimes and one leakage-minimizing variant. In Zero-shot (ZS), the backbone is frozen and only a linear head is trained. In FT-Struct, the last block and head are fine-tuned with cross-entropy,

$$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$7

In FT-LeakMin, the training objective adds a leakage penalty using a unit-mass, differentiable Grad-CAM surrogate $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$8: $$\tilde{C}(p)=\frac{C(p)-\min_{q\in\Omega}C(q)}{\max_{q\in\Omega}C(q)-\min_{q\in\Omega}C(q)+\varepsilon}, \qquad S=\sum_{p\in\Omega}\tilde{C}(p)+\varepsilon,$$9 In practice, the benchmark reports that small $\varepsilon \approx 10^{-6}$0 and $\varepsilon \approx 10^{-6}$1 suffice to suppress leakage without collapsing in-QR coverage (Chakraborty et al., 20 Sep 2025).

The test-time comparison is performed on representative, efficient CAMs—LayerCAM, EigenGrad-CAM, and XGrad-CAM—operating on the same last-block features. Reporting includes per-image and per-distortion values for FMR, TMR, BL, DtS, and coverage AUCs; robustness summaries based on BL slope and FMR/TMR AURCs; means with 95% confidence intervals; and latency in milliseconds per image on a single GPU (Chakraborty et al., 20 Sep 2025).

4. Alternative robustness metrics in recent literature

Different papers operationalize CAM robustness with different observables, aggregation rules, and failure models.

Framework	Robustness notion	Principal score(s)
CAMBench-QR (Chakraborty et al., 20 Sep 2025)	Stability and invariance of saliency allocation on canonical QR substructures under distortion	FMR, TMR, BL, DtS, coverage AUCs, AURC, BL slope
AR2 (Zhang et al., 8 Jul 2025)	CAM alignment between clean and corrupted images, faithful to an MSE-based repair objective	$\varepsilon \approx 10^{-6}$2, optional $\varepsilon \approx 10^{-6}$3
Noise-robustness framework (Sarkar et al., 25 Aug 2025)	Stability when prediction is preserved and change when prediction changes	$\varepsilon \approx 10^{-6}$4
DiffGradCAM / SHAM (Piland et al., 10 Jun 2025)	Resistance of CAMs to entropy-aware passive fooling	Passive Fooling Susceptibility $\varepsilon \approx 10^{-6}$5
ARCC / RefineCAM (Domeniconi et al., 14 May 2026)	Robust faithfulness under perturbation, compactness, and self-consistency	$\varepsilon \approx 10^{-6}$6, with ROAD, Coherency, Complexity

AR2 is centered on classical CAM rather than gradient CAM variants. Its repairing model aligns CAMs for clean and corrupted images to a fixed clean reference CAM from a pretrained model, and the literature derived from that objective proposes a CAM Consistency Score based on normalized MSE between normalized, upsampled CAMs over the union of top-$\varepsilon \approx 10^{-6}$7 predicted classes on the clean and corrupted inputs (Zhang et al., 8 Jul 2025). This metric is explicitly tied to the training loss, so lower CAM MSE implies higher consistency by construction.

The noise-robustness framework introduces a different abstraction. Each image is segmented into QuickShift superpixels, segment saliency is ranked, and similarity between clean and perturbed rankings is measured with Rank-Biased Overlap with $\varepsilon \approx 10^{-6}$8. Consistency is the median RBO over pairs whose top-1 prediction is unchanged. Responsiveness is the AUC of a linear classifier trained on $\varepsilon \approx 10^{-6}$9 pairs. The overall robustness metric is

$M_F$0

with both factors bounded in $M_F$1 (Sarkar et al., 25 Aug 2025).

The SHAM framework addresses a different threat model: explanations can be passively fooled without harming decision performance. Its primary robustness quantity is the average spatial MSE between CAMs from the clean model and the SHAM-fine-tuned model on the same inputs,

$M_F$2

where lower $M_F$3 indicates greater robustness to passive fooling (Piland et al., 10 Jun 2025).

ARCC takes yet another position: robustness must penalize trivial maps and rank-only insensitivity. It defines

$M_F$4

Here ROAD measures perturbation faithfulness, Coherency measures self-consistency under self-masking, and Complexity penalizes overinclusive maps (Domeniconi et al., 14 May 2026).

The comparison reveals a core methodological divide. Some metrics are structure-aware and require masks, some are self-consistency metrics driven by clean–perturbed similarity, some are decision-aware and explicitly reward sensitivity to class changes, and some are faithfulness composites designed to reject trivial or overly broad saliency maps.

5. Empirical patterns and interpretive consequences

In CAMBench-QR, robustness trends are reported after aggregation over rotation, perspective, blur, JPEG, low-light, and occlusion (Chakraborty et al., 20 Sep 2025).

Regime	BL slope ordering (lower is better)	FMR/TMR AURCs (higher is better)
ZS	EigenGrad-CAM $M_F$5 LayerCAM $M_F$6 XGrad-CAM $M_F$7	EigenGrad-CAM $M_F$8; LayerCAM $M_F$9; XGrad-CAM $M_T$0
FT-Struct	EigenGrad-CAM $M_T$1 LayerCAM $M_T$2 XGrad-CAM $M_T$3	EigenGrad-CAM $M_T$4; LayerCAM $M_T$5; XGrad-CAM $M_T$6
FT-LeakMin	EigenGrad-CAM $M_T$7 LayerCAM $M_T$8 XGrad-CAM $M_T$9	EigenGrad-CAM $M_B$0; LayerCAM $M_B$1; XGrad-CAM $M_B$2

These numbers show two regularities. First, leakage suppression during training flattens leakage growth under stress. Second, EigenGrad-CAM consistently provides the best balance between low leakage growth and high structural mass preservation, while LayerCAM becomes close under FT-LeakMin and XGrad-CAM remains relatively leakier at high severity (Chakraborty et al., 20 Sep 2025).

The same benchmark reports that, in zero-shot evaluation, EigenGrad-CAM yields the lowest BL at approximately $M_B$3 and the lowest DtS at approximately $M_B$4, while XGrad-CAM maximizes FMR with higher BL and LayerCAM is balanced. FT-Struct raises structural mass—for example, EigenGrad-CAM reaches FMR of approximately $M_B$5—but can raise BL if unconstrained. FT-LeakMin suppresses BL markedly, with EigenGrad-CAM at approximately $M_B$6, tightens DtS to approximately $M_B$7, and keeps FMR and TMR competitive (Chakraborty et al., 20 Sep 2025).

Causal occlusion tests in the same benchmark reinforce the interpretation of robustness as more than visual stability. After FT-Struct and FT-LeakMin, EigenGrad-CAM exhibits the strongest correlations $M_B$8 and $M_B$9. The reported conclusion is that increasing structural mass is most effective when combined with leakage suppression, that is, with low BL and DtS (Chakraborty et al., 20 Sep 2025).

Across architectures and datasets, the noise-robustness framework finds that Grad-CAM++ consistently achieves the highest RM, while Eigen-CAM and Ablation-CAM typically score lowest; ViT exhibits larger variance in RM across CAMs (Sarkar et al., 25 Aug 2025). A crucial nuance is that under low-severity natural corruptions Eigen-CAM can show high apparent robustness because predictions rarely change and consistency dominates, but its RM drops relative to gradient-based methods as severity increases or under adversarial attacks because responsiveness becomes limiting (Sarkar et al., 25 Aug 2025). This result sharply distinguishes visual invariance from decision-aware robustness.

Under passive fooling, MeanDiffGradCAM attains the lowest susceptibility among the reported methods on ImageNet: GradCAM $\bar{M}_B=\mathbf{1}-M_B$0, EigenCAM $\bar{M}_B=\mathbf{1}-M_B$1, MeanDiffGradCAM $\bar{M}_B=\mathbf{1}-M_B$2, MaxDiffGradCAM $\bar{M}_B=\mathbf{1}-M_B$3, and LSE DiffGradCAM $\bar{M}_B=\mathbf{1}-M_B$4 (Piland et al., 10 Jun 2025). This isolates a failure mode not captured by corruption-only robustness metrics.

6. Misconceptions, limitations, and domain adaptation

A recurring misconception is that a CAM is robust if it is merely stable. The recent literature argues against this reduction. In the noise-robustness framework, robustness requires both high Consistency and high Responsiveness; a method with high Consistency but low Responsiveness can be misleadingly “stable” without reflecting decision changes (Sarkar et al., 25 Aug 2025). In the SHAM setting, even a visually plausible CAM may be non-robust if it can be redirected by passive fooling while accuracy remains intact (Piland et al., 10 Jun 2025).

A second misconception is that commonly used evaluation metrics are automatically robust to trivial maps. The synthetic-ground-truth study explicitly shows that Average Drop and ADCC can be fooled by HalfCAM, whereas ARCC and ROAD correctly assign low scores to trivial CAMs. The same study also shows that ROAD is rank-only and remains unchanged under power transforms that preserve rank, which is why ARCC augments ROAD with magnitude-sensitive Coherency and Complexity (Domeniconi et al., 14 May 2026).

Structure-aware robustness brings its own limitations. CAMBench-QR requires high-quality masks and is sensitive to mask quality and alignment; poor masks can mis-score saliency. Quantile-based thresholding reduces sensitivity to scaling, but extreme noise can still affect coverage AUCs. Synthetic perturbations cover common stressors, yet real-world shifts such as illumination spectra, sensor noise, and multi-object clutter remain open. Adversarial robustness is not explicitly considered in CAMBench-QR, and multi-scale or temporal stability is identified there as natural future work (Chakraborty et al., 20 Sep 2025).

The structure-aware formulation is not restricted to QR codes. CAMBench-QR states that the same design generalizes by replacing QR masks with domain-specific canonical substructures: anatomical regions in medical imaging, title/body/table/figure regions in document layout, or sign borders, pictograms, and text in traffic signs. The corresponding robustness analysis then tracks mass ratios, leakage, DtS, AURCs, and slopes under distortions relevant to the target domain (Chakraborty et al., 20 Sep 2025).

For practice, the recent literature supports a task-dependent choice. When canonical parts and exact masks exist, structure-aware metrics such as FMR, TMR, BL, DtS, and their robustness curves provide direct localization semantics. When only clean–corrupted pairs are available, $\bar{M}_B=\mathbf{1}-M_B$5 or RM offers a model-agnostic consistency view. When passive fooling is a concern, susceptibility $\bar{M}_B=\mathbf{1}-M_B$6 is the direct criterion. When the main risk is trivial or overly broad explanations, ARCC provides a stricter composite test. The broader implication is that CAM robustness is not a single property of a heatmap, but an evaluation design that specifies what should remain invariant, what should change, and what kinds of explanatory failure are being ruled out.