Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations (2203.01517v2)

Abstract: Spurious correlations pose a major challenge for robust machine learning. Models trained with empirical risk minimization (ERM) may learn to rely on correlations between class labels and spurious attributes, leading to poor performance on data groups without these correlations. This is particularly challenging to address when spurious attribute labels are unavailable. To improve worst-group performance on spuriously correlated data without training attribute labels, we propose Correct-N-Contrast (CNC), a contrastive approach to directly learn representations robust to spurious correlations. As ERM models can be good spurious attribute predictors, CNC works by (1) using a trained ERM model's outputs to identify samples with the same class but dissimilar spurious features, and (2) training a robust model with contrastive learning to learn similar representations for same-class samples. To support CNC, we introduce new connections between worst-group error and a representation alignment loss that CNC aims to minimize. We empirically observe that worst-group error closely tracks with alignment loss, and prove that the alignment loss over a class helps upper-bound the class's worst-group vs. average error gap. On popular benchmarks, CNC reduces alignment loss drastically, and achieves state-of-the-art worst-group accuracy by 3.6% average absolute lift. CNC is also competitive with oracle methods that require group labels.

• The paper introduces Correct-n-Contrast, a two-stage approach that employs contrastive learning to mitigate spurious correlations without requiring spurious attribute labels.
• It first utilizes an ERM model to identify sample groups, then applies representation alignment loss to narrow the gap between worst-group and average errors.
• Empirical results demonstrate a 3.6% absolute lift in worst-group accuracy across benchmarks, outperforming state-of-the-art methods that need group labels.

A Contrastive Approach for Improving Robustness to Spurious Correlations

The paper "Correct-n-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations" addresses a significant challenge in machine learning related to spurious correlations. Spurious correlations can significantly degrade the performance of models when they encounter data groups that do not adhere to these correlations. This paper introduces an innovative method named Correct-n-Contrast to improve model robustness by using contrastive learning to mitigate such impacts without relying on spurious attribute labels during training.

At its core, the proposed method builds a robust understanding of data by embarking on a two-stage process. The first stage involves training an empirical risk minimization (ERM) model to identify samples with identical classes but different spurious attributes. Given that ERM models can inadvertently learn spurious correlations effectively, this serves as a foundation for defining groups within the data.

The second stage of the process introduces the Correct-n-Contrast methodology. It utilizes the outputs from the ERM model to engage in a contrastive learning approach that learns similar representations for same-class samples with diverse spurious features. This is done by enforcing representation alignment through contrastive loss, which aims to align features to be consistent within same-class groups while maintaining distinct separation between different classes.

An important aspect of this approach is the connection made between worst-group error and representation alignment loss. The paper both empirically and theoretically demonstrates that minimizing alignment loss can restrict the worst-group versus average error gap within a class. This is substantiated through experimental evaluations on multiple benchmarks, including colored MNIST, Waterbirds, CelebA, and CivilComments-WILDS, where Correct-n-Contrast consistently achieves superior worst-group accuracy.

One of the standout results of this paper is achieving a 3.6% average absolute lift in worst-group accuracy, outperforming state-of-the-art methods that require group labels. It nearly bridges the performance gap with oracle methods that have access to spurious attributes during training.

The implications of this research are profound, as it enhances the robustness of machine learning models against biases introduced by spurious correlations in data. The approach could see widespread applications in fields demanding high fairness and unbiased outcomes, such as medical diagnosis, social media analysis, and algorithmic decision-making systems.

Looking ahead, the principles elucidated in this paper pave the way for further exploration of contrastive learning in refining the granularity of data representation and mitigating biases without extensive label requirements. As AI continues to evolve, such methodologies will be critical in enhancing the reliability and fairness of automated systems across diverse applications.