Spurious Images in Neural Classifiers

• Spurious images are defined by their reliance on non-causal visual cues, such as backgrounds, which mislead classifiers by shifting focus from intrinsic object features.
• Detection methods like Grad-CAM and counterfactual inpainting quantify and localize spurious features, enabling robust performance evaluations across varied benchmarks.
• Mitigation strategies including data augmentation, adversarial training, and post-hoc adjustments significantly improve worst-group accuracy and model generalization.

Spurious images occupy a central position in contemporary research on visual classification robustness, as they expose both the vulnerabilities of deep neural classifiers to non-causal cues and the methodological landscape for their identification, quantification, and mitigation. The critical advances reviewed here establish precise operational definitions, elucidate algorithmic detection and data-generation strategies, and detail robust empirical findings spanning natural and synthetic images, medical domains, and multi-modal systems.

1. Definition and Harm of Spurious Images

Spurious images are those whose predictive power for a designated label derives not from true causal instance features but from non-causal attributes—most commonly backgrounds or co-occurring objects—that are disproportionately present with the label in the training set. Formally, if $A$ is a non-causal attribute and $Y$ is the label, a dataset exhibits spurious correlation if $$P(A = a_1 \land Y = y_1) \gg P(A = a_2 \land Y = y_1)$$, so the classifier can learn $A \rightarrow Y$ instead of modeling intrinsic features of the target object (Li et al., 2024). Spurious images, then, are those that either (1) exhibit a correlated attribute $A$ in the absence of the true object, yet are classified as $Y$ (e.g., “grass” alone triggers "cow"), or (2) combine an object $Y$ with atypical $A$ at test time, resulting in model failure because the shortcut is broken.

This reliance is pernicious: models exhibit high in-distribution and average accuracy, yet their worst-case (worst-group) or out-of-distribution performance degrades severely when the spurious attribute is inverted or missing (Li et al., 2024, Jin et al., 2024, Neuhaus et al., 2022). Real-world consequences include unreliable medical diagnosis, unfair demographic bias, and misclassification under covariate shift.

2. Mechanisms for Detection and Quantification

Multiple detection methodologies have been developed to identify both the presence of spurious features in classifier decision-making and the harmfulness of specific spurious images.

• Saliency and Attention Attribution: Tools such as Grad-CAM, Grad-CAM++, and class activation maps serve to localize classifier attention. For instance, regions with high contribution to incorrect predictions are thresholded to produce masks that highlight spurious regions (Li et al., 2024). The divergence between robust feature saliency (e.g., RF-CAM from surrogates) and the original model’s saliency map quantitatively flags spurious use (Dammu et al., 2023).
• Neural Feature Annotation: By associating neural features (penultimate layer activations) with either core or spurious attributes—using minimal human validation—researchers can assign soft or hard masks over large datasets, as in Salient ImageNet (Singla et al., 2021), which demonstrates that many Imagenet classes are reliably predicted on spurious cues.
• Counterfactual and Attribution-based Scans: Perturbation-based techniques, such as exchanging or masking out localized patches and measuring model confidence changes, automate the identification of spurious regions (Hagos et al., 2022). Counterfactual alignment leverages latent-space manipulation to measure output shifts when specific features are erased, directly probing classifier dependence on the corresponding cues (Cohen et al., 2023).
• Evaluation Metrics: Worst-group accuracy, area under ROC contrasting true versus spurious-only images (mAUC), and the Spurious Correlation Latching Score (SCLS) provide quantitative measures for the degree and impact of spurious reliance (Li et al., 2024, Neuhaus et al., 2022, Kumar et al., 2023).

3. Generation and Benchmarking of Spurious Images

The advent of controllable image-generation pipelines has enabled systematic construction of spurious images for both analysis and augmentation.

• Synthetic Benchmarks: Datasets such as Spawrious are generated using Stable Diffusion, pairing objects (e.g., specific dog breeds) with backgrounds acting as spurious attributes in systematically designed one-to-one and many-to-many spurious correlations (Lynch et al., 2023). Spurious ImageNet is curated by neural PCA to identify features whose presence alone triggers class predictions, confirming their harmfulness via real-world web images (Neuhaus et al., 2022).
• Spurious-Feature Generation with Diffusion Models: Fine-tuning diffusion models with a small set of reliably spurious images and enforcing spurious-feature similarity loss produces new images that consistently fool a variety of classifiers, outperforming web-scraped sets in multi-model spuriousness (MaungMaung et al., 2024). This enables both adversarial auditing and controlled debiasing training data.
• Compositional and Counterfactual Synthesis: Methods such as Decompose-and-Compose, SCGS, and text-to-image inpainting pipelines generate counterfactual images by recombining or re-rendering instance and background regions, either to break existing shortcuts or to increase the coverage of (label, attribute) pairs and thus flatten spurious associations (Li et al., 2024, Noohdani et al., 2024, Parast et al., 21 Mar 2025).
• Medical Domain: Both SpurBreast and MaskMedPaint design splits or augmentations in high-stakes domains (e.g., MRI), where non-clinical image orientation or device artifacts become spurious signals and synthetic inpainting or domain transfer aligns images to break spurious dependencies (Won et al., 2 Oct 2025, Jin et al., 2024).

4. Mitigation Strategies and Theoretical Underpinnings

Mitigating the impact of spurious images or features involves either removing classifier reliance on non-causal cues or constructing data distributions where these cues are no longer predictive.

• Data Augmentation and Synthesis: SCGS guides inpainting of spuriously-attended regions identified by Grad-CAM++, generating “counter-class” images in which the spurious region appears with the true class, thereby flattening $P(A|Y)$ (Li et al., 2024). This method, along with DaC (compositional intervention) and T2I-based methods, augments training data so models do not benefit from spurious shortcuts.
• Adversarial and Invariant Learning: IRSS incorporates adversarial losses for style-invariant features and invariant risk minimization (IRM) across inferred environments to decouple style and spurious features, leading to representations that are robust to both types of distribution shift (Li et al., 2023).
• Feature Suppression via Post-hoc Remediation: SpuFix subtracts the contribution of identified harmful spurious components from classifier logits, raising mAUC with negligible impact on top-1 accuracy (Neuhaus et al., 2022).
• Concept Bottleneck Approaches: Automated pipelines leveraging large language and vision foundation models identify and remove candidate spurious concepts from the intermediate representation, yielding interpretable models with improved worst-group performance at minimal manual effort (Kim et al., 2024).
• Two-Step Synthetic-Real Training: FFR demonstrates that sequential pretraining on perfectly balanced synthetic images, without simultaneous exposure to real images (to avoid combining domain and bias cues), substantially outperforms mixed real-synthetic augmentation under distribution shifts (Qraitem et al., 2023).
• Auxiliary Losses and Explanatory Regularization: Integrating explanation losses—e.g., ensuring GradCAM maps avoid spurious patches—or multi-class counterfactual consistency loss during training, refines model focus towards causal content (Hagos et al., 2022, Kumar et al., 2023).

5. Empirical Evaluation and Findings

The effectiveness of spurious image detection and mitigation is supported by substantial empirical evidence:

• Performance Gains: On MetaShift, Waterbirds, and CelebA, SCGS yielded worst-group accuracy improvements of +13.1 %, +9.1 %, and +24.0 % over ERM, respectively. Results generally show that data-augmentation with spurious-guided synthesis or controlled counterfactual generation closes the gap between traditionally over-optimistic training performance and robust, group-wise generalization (Li et al., 2024).
• Ablation of Image-Generation Protocols: Grad-CAM++-guided inpainting outperforms baseline inpainting and Grad-CAM methods (worst-group accuracy: 75.3 % on MetaShift vs. 71.0 % or 68.8 %) (Li et al., 2024).
• Benchmarking of Robustness: No existing group-robustness method achieves over 70 % accuracy on hardest many-to-many spurious-correlation benchmarks (Spawrious) without explicit countermeasures (Lynch et al., 2023).
• Generalization and Limitations: On very large datasets, including additional synthesized images yields diminishing returns, highlighting the finite corrective impact of augmentation when absolute data scale dominates sampling bias (Li et al., 2024).
• Visualization of Attention Shifts: Models trained with spurious-focused augmentations demonstrate visible attention migration from background or artefact regions toward intrinsic object features, as confirmed by saliency maps (Li et al., 2024, Noohdani et al., 2024).

6. Limitations, Open Challenges, and Outlook

Despite significant progress, several inherent challenges remain:

• Scalability and Coverage: Construction of high-fidelity, context-independent spurious images remains constrained by generator capacity, context interaction (e.g., spurious cues may not generalize across backgrounds), and manual curation bottlenecks. Fully automating spurious detection in arbitrarily complex scenes is an open issue (MaungMaung et al., 2024, Neuhaus et al., 2022).
• Label and Annotation Constraints: Many robust optimization and debiasing pipelines, though effective, presuppose access to group labels or precise segmentation masks, which are expensive or difficult for open-ended datasets (Li et al., 2024, Li et al., 2023). Recent work focuses on unsupervised clustering, language-guided annotation, or minimal human oversight to bridge this gap (Kim et al., 2024, Ghosh et al., 2023).
• Adversarial and Synthetic Artifacts: Models trained on synthetic or inpainted data are susceptible to new forms of spurious shortcuts if synthetic artifacts become discriminative (Qraitem et al., 2023). Addressing these “fake-side” spurious patterns is a topic for method extension (Rajan et al., 11 Feb 2025).
• Multi-modal and Cognitive Biases: In MLLMs, spurious images cause “visual illusion” effects, revealing an instinctive bias to align predictions with salient but incorrect visual cues regardless of text, signifying a multimodal generalization of the spurious cue problem (Han et al., 2024).

Future directions include fully context-aware augmentation, extension to multi-attribute, multi-class and multi-modal cases, automation of mask selection, and robust evaluation pipelines under open-world and real-time distribution shifts.

---

• (Li et al., 2024) Focusing Image Generation to Mitigate Spurious Correlations
• (Li et al., 2023) Invariant Representation via Decoupling Style and Spurious Features from Images
• (Singla et al., 2021) Salient ImageNet: How to discover spurious features in Deep Learning?
• (Qraitem et al., 2023) From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognition
• (Neuhaus et al., 2022) Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet
• (MaungMaung et al., 2024) Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation
• (Dammu et al., 2023) Detecting Spurious Correlations via Robust Visual Concepts in Real and AI-Generated Image Classification
• (Jin et al., 2024) MaskMedPaint: Masked Medical Image Inpainting with Diffusion Models for Mitigation of Spurious Correlations
• (Noohdani et al., 2024) Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation
• (Lynch et al., 2023) Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases
• (Kim et al., 2024) Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort
• (Hagos et al., 2022) Identifying Spurious Correlations and Correcting them with an Explanation-based Learning
• (Rajan et al., 11 Feb 2025) Stay-Positive: A Case for Ignoring Real Image Features in Fake Image Detection
• (Cohen et al., 2023) Identifying Spurious Correlations using Counterfactual Alignment
• (Parast et al., 21 Mar 2025) Leveraging Text-to-Image Generation for Handling Spurious Correlation
• (Han et al., 2024) The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
• (Won et al., 2 Oct 2025) SpurBreast: A Curated Dataset for Investigating Spurious Correlations in Real-world Breast MRI Classification