2000 character limit reached
Modulate Your Spectrum in Self-Supervised Learning (2305.16789v2)
Published 26 May 2023 in cs.LG, cs.CV, and eess.SP
Abstract: Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.
- Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
- Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.
- Algorithms for the matrix pth root. Numerical Algorithms, 2005.
- Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
- Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
- Emerging properties in self-supervised vision transformers. In ICCV, 2021.
- A simple framework for contrastive learning of visual representations. In ICML, 2020a.
- Exploring simple siamese representation learning. In CVPR, 2021.
- Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
- ImageNet contributors. Imagenet terms of access. 2020. URL https://image-net.org/download.
- solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1–6, 2022. URL http://jmlr.org/papers/v23/21-1155.html.
- ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
- Whitening for self-supervised representation learning. In ICML, 2021.
- Understanding self-supervised and contrastive learning with ”bootstrap your own latent” (byol). Technical Report, 2020.
- Inc. Flickr. Flickr terms and conditions of use. 2020. URL http://aiweb.techfak.uni-bielefeld.de/content/bworld-robot-control-software/.
- On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023.
- Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05808, 2022.
- Bootstrap your own latent - a new approach to self-supervised learning. In NeuraIPS, 2020.
- Exploring the gap between collapsed and whitened features in self-supervised learning. In ICML, 2022.
- Deep residual learning for image recognition. In CVPR, 2016.
- Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
- Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
- On feature decorrelation in self-supervised learning. In ICCV, 2021.
- Decorrelated batch normalization. In CVPR, 2018.
- Iterative normalization: Beyond standardization towards efficient whitening. In CVPR, 2019.
- A survey on contrastive self-supervised learning. arXiv preprint arXiv:2011.00362, 2020.
- Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
- Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
- Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
- Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
- Microsoft coco: Common objects in context. In ECCV, 2014.
- Graph self-supervised learning: A survey. arXiv e-prints, pp. arXiv–2103, 2021.
- Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
- Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
- Self-supervised video transformer. In CVPR, 2022.
- Byol works even without batch statistics. arXiv preprint arXiv:2010.10241, 2020.
- A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
- Whitening and coloring batch transform for gans. In ICLR, 2019.
- Exploring the equivalence of siamese self-supervised learning via A unified gradient framework. In CVPR, 2022.
- Contrastive multiview coding. In European conference on computer vision, 2020a.
- Understanding self-supervised learning with dual deep networks. CoRR, abs/2010.00578, 2020b.
- Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
- Contrastive learning with stronger augmentations. IEEE transactions on pattern analysis and machine intelligence, 45(5):5549–5560, 2022.
- An investigation into whitening loss for self-supervised learning. In NeurIPS, 2022.
- Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
- Network deconvolution. In ICLR, 2020.
- Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
- Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
- How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In ICLR, 2022a.
- From canonical correlation analysis to self-supervised graph neural networks. In NeurIPS, 2021.
- Zero-CL: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022b.
- Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11289–11297, 2023.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.