Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation (2402.18919v3)

Abstract: While standard Empirical Risk Minimization (ERM) training is proven effective for image classification on in-distribution data, it fails to perform well on out-of-distribution samples. One of the main sources of distribution shift for image classification is the compositional nature of images. Specifically, in addition to the main object or component(s) determining the label, some other image components usually exist, which may lead to the shift of input distribution between train and test environments. More importantly, these components may have spurious correlations with the label. To address this issue, we propose Decompose-and-Compose (DaC), which improves robustness to correlation shift by a compositional approach based on combining elements of images. Based on our observations, models trained with ERM usually highly attend to either the causal components or the components having a high spurious correlation with the label (especially in datapoints on which models have a high confidence). In fact, according to the amount of spurious correlation and the easiness of classification based on the causal or non-causal components, the model usually attends to one of these more (on samples with high confidence). Following this, we first try to identify the causal components of images using class activation maps of models trained with ERM. Afterward, we intervene on images by combining them and retraining the model on the augmented data, including the counterfactual ones. Along with its high interpretability, this work proposes a group-balancing method by intervening on images without requiring group labels or information regarding the spurious features during training. The method has an overall better worst group accuracy compared to previous methods with the same amount of supervision on the group labels in correlation shift.

Decompose-and-Compose: Enhancing Robustness to Spurious Correlation in Image Classification

Introduction

Empirical Risk Minimization (ERM) has been the cornerstone of training strategies for image classification tasks, particularly when the goal is to optimize performance on in-distribution (ID) samples. However, the effectiveness of ERM wanes in the face of out-of-distribution (OOD) samples, primarily due to its vulnerability to spurious correlations. This limitation is exacerbated in realistic scenarios where images comprise both causal and non-causal components relative to the target label. The paper undertakes the challenge of spurious correlation through a novel approach dubbed Decompose-and-Compose (DaC), which focuses on balancing groups by intervening on non-causal components of images to generate new, counterfactual examples. This method proposes a solution that does not require explicit knowledge of group labels or spurious features, making it a significant step forward in developing robust models capable of generalizing across varied distributions.

Methodology

The foundation of DaC lies in two critical observations:

• Models trained with ERM tend to focus either on causal parts or on parts that exhibit a high spurious correlation with the target label, especially on samples where the model is confident about its predictions.
• A granular, compositional analysis, facilitated by class activation maps, indicates that the emphasis (causal or spurious) depends on the relative ease of predicting the label based on these components.

This leads to the inception of DaC, which operates by first identifying and decomposing images into causal and non-causal parts. This identification process leverages the class activation maps of models pre-trained with ERM. The pivotal aspect of DaC is the generation of novel data points through the combination of deconstructed image parts from different samples, followed by retraining the model on this augmented dataset, which incorporates these synthetically generated counterfactual instances.

Evaluation

DaC is evaluated against baseline methods such as DFR, MaskTune, and Group DRO on benchmarks including Waterbirds, CelebA, MetaShift, and Dominoes datasets. These benchmarks are selected to cover a range of distribution and spurious correlation challenges.

The contributions are multi-fold, emphasizing:

• A detailed analysis of ERM-trained models' behavior, particularly noting their tendency to focus on either causal or spuriously correlated components based on confidence and loss metrics.
• An innovative method to identify causal parts within images, capitalizing on the insights obtained from the models' attribution maps.
• A group-balancing strategy reliant on the strategic combination of image parts to construct new, balanced training data points without the necessity for group label information.
• Superior performance in worst-group accuracy metrics across a majority of the considered benchmarks, showcasing DaC's efficacy in mitigating the adverse effects of spurious correlation.

Implications and Future Directions

The insights and methodology introduced in this paper have profound implications for both theoretical and practical advancements in AI. Theoretically, DaC provides a nuanced understanding of how models attend to various components of an image and how this attention can be manipulated to foster more robust learning. Practically, the ability to enhance model robustness without explicit reliance on group labels or detailed knowledge of spurious features has far-reaching applications across different domains where robustness and generalizability are paramount.

As for future research avenues, the exploration could extend into:

• Refining the decomposition and composition mechanisms to increase the precision of causal and non-causal part identification.
• Applying DaC's principles to other forms of data beyond images, such as text or audio, where spurious correlation is also a significant challenge.
• Investigating the intersection of DaC with other robustness-enhancing techniques like adversarial training, to further bolster model resilience.

Conclusion

This work stands as a testament to the evolving understanding of how models perceive and process information, moving beyond the monolithic treatment of images towards a more dissected and recombinant strategy. Decompose-and-Compose not only sheds light on the nuanced dynamics of spurious correlation but also provides a practical toolkit to address it, signaling a shift towards developing AI systems that are not only performant but robust and fair by design.