Effect of Image Alignment on Spatial Shortcut Auditing

Determine how the degree of image alignment across a dataset affects dataset-level inspection, shortcut-group risk-based subset identification, and actionable spatial interventions based on grid-partitioned contribution maps.

Background

The framework partitions contribution maps using a fixed spatial grid and aggregates these maps at dataset or shortcut-group level. This design may work well for approximately aligned images, such as CelebA and CheXpert, but spatial variability in Waterbirds, Camelyon17, and ISIC2019 may alter the resulting patterns. The paper reports meaningful visual patterns and auditing results but does not quantify how alignment affects inspection, risk-based error selection, or intervention effectiveness.

References

The grid-based partitioning used for contribution maps can be affected by spatial alignment across the images, especially when the contribution maps are aggregated across the dataset or when forming a smaller number of shortcut groups. This might be suitable for approximately aligned datasets such as CelebA and CheXpert, but may affect datasets with larger spatial variability such as Waterbirds, Camelyon17, and ISIC2019. While our analyses provide meaningful shortcut group visual patterns, risk-based subset identification and actionable interventions, we do not conduct analyses on how much the image alignment across the dataset can affect the dataset-level inspection and leave this as future work.

— Discovery and Spatial Characterisation of Multiple Shortcut Groups for Auditing Vision Model Bias  (2608.14051 - Achara et al., 14 Aug 2026) in Section 5.1, “Limitations” (label: sec:limitations)

The positive contribution maps contain all regions that support positive conditional rank alignment. This includes both regions that the two relevant models rank more highly than predicted from the conditioning model and regions that both rank less highly than predicted. Consequently, positive contribution should not be interpreted as absolute model importance or as ground-truth causal shortcut or task evidence. Our quadrant analysis (Appendix Section~\ref{supp:sec-quadrants}) shows that these cases can exhibit different spatial structure, but determining which components provide the most meaningful evidence for different auditing objectives is left as future work.

— Discovery and Spatial Characterisation of Multiple Shortcut Groups for Auditing Vision Model Bias  (2608.14051 - Achara et al., 14 Aug 2026) in Section 5.1, “Limitations” (label: sec:limitations)