Do Generated Data Always Help Contrastive Learning?
Abstract: Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet it often depends on intensive manual data augmentations. With the rise of generative models, especially diffusion models, the ability to generate realistic images close to the real data distribution has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, a technique termed ``data inflation''. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data inflation and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these insights, we propose Adaptive Inflation (AdaInf), a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. Notably, without using external data, AdaInf obtains 94.70% linear accuracy on CIFAR-10 with SimCLR, setting a new record that surpasses many sophisticated methods. Code is available at https://github.com/PKU-ML/adainf.
- Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
- Label-efficient semantic segmentation with diffusion models. In ICLR, 2022.
- This dataset does not exist: Training models from generated images. In ICASSP, 2020.
- Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. Transactions on Pattern Analysis and Machine Intelligence, 44(11):7327–7347, 2022.
- Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
- Emerging properties in self-supervised vision transformers. In ICCV, 2021.
- A simple framework for contrastive learning of visual representations. In ICML, 2020a.
- Exploring simple siamese representation learning. In CVPR, 2021.
- Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
- An empirical study of training self-supervised vision transformers. In ICCV, 2021.
- The spectral gap of a random subgraph of a graph. Internet Mathematics, 4(2-3):225–244, 2007.
- Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
- Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
- solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23:56:1–56:6, 2022.
- Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
- On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023.
- Generative adversarial networks. In NeurIPS, 2014.
- Bootstrap your own latent: a new approach to self-supervised learning. In NeurIPS, 2020.
- Learning video representations of human motion from synthetic data. In CVPR, 2022.
- Contranorm: A contrastive learning perspective on oversmoothing and beyond. In ICLR, 2023.
- A theoretical study of inductive biases in contrastive learning. In ICLR, 2023.
- Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
- Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
- Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- Generative models as a data source for multiview representation learning. In ICLR, 2022.
- Learning to see by looking at noise. In NeurIPS, 2021.
- Training generative adversarial networks with limited data. In NeurIPS 2020, 2020.
- Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.
- Auto-encoding variational Bayes. In ICLR, 2014.
- Rethinking the effect of data augmentation in adversarial contrastive learning. In ICLR, 2023.
- Pretrained diffusion models for unified human motion synthesis. arXiv preprint arXiv:2212.02837, 2022.
- Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
- Cross-domain self-supervised multi-task feature learning using synthetic imagery. In CVPR, 2018.
- Learning from synthetic data: Addressing domain shift for semantic segmentation. In CVPR, 2018.
- Understanding contrastive learning requires incorporating inductive biases. In ICML, 2022.
- What makes for good views for contrastive learning? In NeurIPS, 2020.
- Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306.00984, 2023.
- Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
- Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
- Residual relaxation for multi-view representation learning. In NeurIPS, 2021a.
- Reparameterized sampling for generative adversarial networks. In ECML-PKDD, 2021b.
- A unified contrastive energy-based model for understanding the generative ability of adversarial training. In ICLR, 2022a.
- Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022b.
- A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
- Synthetic data can also teach: Synthesizing effective data for unsupervised visual representation learning. In AAAI, 2023.
- Stable target field for reduced variance score estimation in diffusion models. In ICLR, 2023.
- Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
- On the generalization of multi-modal contrastive learning. In ICML, 2023a.
- Identifiable contrastive learning with automatic feature importance discovery. In NeurIPS, 2023b.
- Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In ICLR, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.