CAMBench-QR: Structure-Aware CAM Benchmarking

• CAMBench-QR is a structure-aware benchmark that rigorously evaluates post-hoc CAM methods by aligning saliency maps with precise QR code substructures.
• It employs synthetic data with exact geometric masks and controlled image distortions to objectively measure structural fidelity in visual explanations.
• The benchmark defines novel metrics, including Finder Mass Ratio and Background Leakage, to ensure that saliency is accurately attributed to canonical QR features.

CAMBench-QR is a structure-aware benchmark for evaluating post-hoc visual explanation methods—specifically Class Activation Mapping (CAM) algorithms—by rigorously testing whether saliency is ascribed to canonical QR code geometry rather than incidental regions. Unlike conventional explanation benchmarks that emphasize only plausible localization, CAMBench-QR demands structural fidelity: the attribution of saliency should coincide with requisite QR substructures (finder patterns, timing lines, module grid) and avoid background leakage. The benchmark synthesizes QR and non-QR image data with exact geometric masks and applies controlled image distortions, providing a reproducible yardstick for structure-constrained interpretability assessment (Chakraborty et al., 20 Sep 2025).

1. Structure-Aware Benchmarking Philosophy

CAMBench-QR is motivated by the observation that visual explanations—especially heatmaps produced by CAM methods—often lack correlation with object-defining geometry, making them unreliable for tasks requiring precise attribution. QR codes provide an ideal domain due to their rigid, parametric layout: three high-contrast finder patterns, linear timing patterns, and a binary module grid. By anchoring evaluation to these ground-truth substructures, CAMBench-QR tests whether explanations truly localize decisive object features. Standard interpretability metrics are insufficient; CAMBench-QR introduces structure-based metrics and masking, utilizing the mathematical properties of the QR code layout for objective scoring.

2. Data Synthesis and Mask Construction

The benchmark employs synthetic data generation, producing QR and non-QR images with exact mask annotations for canonical parts:

• Finder Mask ($M_F$): Marks the three square finder patterns in QR geometry.
• Timing Mask ($M_T$): Selects the timing line modules connecting finders, central to QR decoding.
• Module/Box Mask ($M_B$): Encloses the full QR code, explicitly separating QR foreground from background ($\bar{M}_B$).

Distortions—rotation, perspective transform, blur, compression, lighting variation, occlusion—are applied programmatically, retaining correct mask tracking throughout. This guarantees robust, perturbation-resilient evaluation and prevents "cheating" by methods that exploit superficial visual cues.

3. Structure-Specific Evaluation Metrics

CAMBench-QR introduces a suite of quantitative, structure-aware metrics beyond conventional faithfulness and occlusion scores:

• Finder Mass Ratio (FMR):

$$\mathrm{FMR} = \frac{\langle \tilde{C}, M_F \rangle}{S}$$

• Fraction of normalized CAM mass on finder patterns, where $\tilde{C}(p)$ is min-max normalized saliency and $S = \sum_p \tilde{C}(p) + \varepsilon$.
  • Timing Mass Ratio (TMR): Analogous to FMR, for timing patterns.
  • Background Leakage (BL):

$$\mathrm{BL} = \frac{\langle \tilde{C}, \bar{M}_B \rangle}{S}$$

• Fraction of saliency mass incorrectly attributed to non-QR regions.
  • Structure-Level Coverage AUCs: Area under coverage curves for finder/timing/background masks as a function of CAM threshold.
  • Distance-to-Structure (DtS):

$$\mathrm{DtS} = \frac{\langle \tilde{C}, D \rangle}{S \sqrt{H^2 + W^2}}$$

• Average mass-weighted Euclidean distance from saliency outside the structural regions.

Collectively, these metrics enforce geometric alignment between explanation and object, providing discriminative scores unattainable by generic saliency methods.

4. CAM Algorithm Regimes and Fine-Tuning Procedures

CAMBench-QR benchmarks three efficient CAM methods under realistic deployment scenarios:

• LayerCAM
• EigenGrad-CAM
• XGrad-CAM

Each is evaluated in two regimes:

• Zero-shot: Frozen backbone with linear head.
• Last-block fine-tuning: Fine-tuning only the final residual block, using standard cross-entropy loss ("FT-Struct") and optionally imposing a background leakage penalty ("FT-LeakMin").

Experimental results demonstrate:

• EigenGrad-CAM yields low BL and DtS in zero-shot due to cleaner attribution.
• XGrad-CAM is fastest, sometimes with higher background leakage.
• FT-Struct/FT-LeakMin substantially reduce leakage and spatial error, reallocating CAM mass onto QR substructures, with FT-LeakMin especially effective without added computational burden.

These findings underscore the necessity of structure-aware loss functions and careful tuning for explanation alignment in visual domains.

5. Practical Implications and Litmus Test Conceptualization

CAMBench-QR is proposed as a litmus test: any explanation method passing its structure-aware criteria can be deemed structurally faithful, not merely visually plausible. Applications include:

• Auditable scientific imaging, document forensics, barcode and QR code verification.
• Debugging and model auditing in safety-critical machine vision systems.
• Research environments demanding reproducible, objective interpretability evaluation.

The benchmark's rigorous criteria mitigate risks of misleading explanation via spurious or texture-based saliency, which can undermine trust and safety in automated systems.

6. Conclusions and Future Research Trajectories

CAMBench-QR reframes post-hoc explanation benchmarking around geometric fidelity. By synthesizing structured data, computing exact attribution metrics, and demonstrating tangible improvement via structure-aware training, it establishes a reproducible, objective standard for interpretability. The approach reveals that methods like EigenGrad-CAM, when combined with leakage-penalized training, achieve superior alignment with target geometry.

Future work may include:

• Extension to multi-object and more complex structured domains (barcodes, medical images).
• Adaptation to transformer-based and video explanation architectures.
• Development of richer counterfactual tests and model-edit interventions.

CAMBench-QR advances the interpretability evaluation paradigm by demanding not just visual plausibility, but strict structural compliance of post-hoc explanations, setting a precedent for transparent, reliable deployment of interpretable deep learning in critical applications (Chakraborty et al., 20 Sep 2025).