Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unifying Self-Supervised Clustering and Energy-Based Models (2401.00873v3)

Published 30 Dec 2023 in cs.LG and cs.CV

Abstract: Self-supervised learning excels at learning representations from large amounts of data. At the same time, generative models offer the complementary property of learning information about the underlying data generation process. In this study, we aim at establishing a principled connection between these two paradigms and highlight the benefits of their complementarity. In particular, we perform an analysis of self-supervised learning objectives, elucidating the underlying probabilistic graphical models and presenting a standardized methodology for their derivation from first principles. The analysis suggests a natural means of integrating self-supervised learning with likelihood-based generative models. We instantiate this concept within the realm of cluster-based self-supervised learning and energy models, introducing a lower bound proven to reliably penalize the most important failure modes. Our theoretical findings are substantiated through experiments on synthetic and real-world data, including SVHN, CIFAR10, and CIFAR100, demonstrating that our objective function allows to jointly train a backbone network in a discriminative and generative fashion, consequently outperforming existing self-supervised learning strategies in terms of clustering, generation and out-of-distribution detection performance by a wide margin. We also demonstrate that the solution can be integrated into a neuro-symbolic framework to tackle a simple yet non-trivial instantiation of the symbol grounding problem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. Fixing a Broken ELBO. In ICML, 2018.
  2. Self-Supervised Classification Network. In ECCV, 2022.
  3. Self-Labelling via Simultaneous Clustering and Representation Learning. In ICLR, 2020.
  4. The Hidden Uniform Cluster Prior in Self-Supervised Learning. In ICLR 2023, 2023.
  5. Masked Siamese Networks for Label-Efficient Learning. In ECCV, 2022.
  6. R. Balestriero and Y. LeCun. Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods. In NeurIPS, 2022.
  7. A Cookbook of Self-Supervised Learning. arXiv, 2023.
  8. On the Surrogate Gap between Contrastive and Supervised Losses. In ICML, 2022.
  9. D. Barber and F. Agakov. The IM Algorithm: A Variational Approach to Information Maximization. In NeurIPS, 2004.
  10. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. In ICLR, 2022a.
  11. VICREGL: Self-Supervised Learning of Local Visual Features. In NeurIPS, 2022b.
  12. S. Becker and G. Hinton. Self-Organizing Neural Network that Discovers Surfaces in Random-Dot Stereograms. Nature, 355(6356):161–163, 1992.
  13. Mutual Information Neural Estimation. In ICML, 2018.
  14. Deep Clustering for Unsupervised Learning of Visual Features. In ECCV, 2018.
  15. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NeurIPS, 2020.
  16. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, 2021.
  17. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, 2020.
  18. M. Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In NeurIPS, 2013.
  19. Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR, 23(56):1–6, 2022.
  20. Representation Learning with Contrastive Predictive Coding. In arXiv, 2018.
  21. Y. Du and I. Mordatch. Implicit Generation and Generalization in Energy-Based Models. arXiv, 2019.
  22. Improving Self-Supervised Learning by Characterizing Idealized Representations. In NeurIPS, 2022.
  23. No MCMC for Me: Amortized Samplers for Fast and Stable Training of Energy-Based Models. In ICLR, 2021.
  24. Whitening for Self-Supervised Representation Learning. In ICML, 2021.
  25. On the Duality Between Contrastive and Non-Contrastive Self-Supervised Learning. In ICLR, 2023.
  26. I. Gatopoulos and J. M. Tomczak. Self-Supervised Variational Auto-Encoders. In Symposium on Advances in Approximate Bayesian Inference, 2020.
  27. Your Classifier is Secretly an Energy Based Model and You Should Treat It Like One. In ICLR, 2020.
  28. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. In NeurIPS, 2020.
  29. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. In NeurIPS, 2021.
  30. Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. In NeurIPS, 2019.
  31. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS, 2017.
  32. Learning Deep Representations by Mutual Information Estimation and Maximization. In ICLR, 2018.
  33. Learning Representation for Clustering Via Prototype Scattering and Positive Sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  34. L. Jing and Y. Tian. Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  35. Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-Supervised Learning. In NeurIPS, 2022.
  36. B. Kim and J. C. Ye. Energy-Based Contrastive Learning of Visual Representations. In NeurIPS, 2022.
  37. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. In NeurIPS, 2021.
  38. Y. LeCun. A Path Towards Autonomous Machine Intelligence, 2022.
  39. K. Lee. Prototypical Contrastive Predictive Coding. In ICLR, 2022.
  40. Contrastive Clustering. In AAAI, 2021.
  41. Neural Manifold Clustering and Embedding. In arXiv, 2022.
  42. R. Linsker. Self-Organization in a Perceptual Network. Computer, 21(3):105–117, 1988.
  43. H. Liu and P. Abbeel. Hybrid Discriminative-Generative Training via Contrastive Learning. In arXiv, 2020.
  44. Self-Supervised Learning via Maximum Entropy Coding. In NeurIPS, 2022.
  45. DeepProbLog: Neural Probabilistic Logic Programming. In NeurIPS, 2018.
  46. Neural probabilistic logic programming in DeepProbLog. Artifical Intelligence, 298:103504, 2021.
  47. Neuro-Symbolic Reasoning Shortcuts: Mitigation Strategies and their Limitations. In arXiv, 2023.
  48. D. McAllester and K. Stratos. Formal Limitations on the Measurement of Mutual Information. In AISTATS, 2020.
  49. Representation Learning via Invariant Causal Mechanisms. In ICLR, 2021.
  50. Self-Supervised Learning for Generalizable Out-of-Distribution Detection. In AAAI, 2020.
  51. Representation Uncertainty in Self-Supervised Learning as Variational Inference. In ICCV, 2023.
  52. Do deep generative models know what they don’t know? In ICLR, 2019.
  53. Learning non-convergent non-persistent short-run mcmc toward energy-based model. In NeurIPS, 2019.
  54. On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models. In AAAI, 2020.
  55. O. O. Henaff. Data-Efficient Image Recognition with Contrastive Predictive Coding. In ICML, 2020.
  56. Self-Supervised Learning with an Information Maximization Criterion. In NeurIPS, 2022.
  57. On Variational Bounds of Mutual Information. In ICML, 2019.
  58. E. Sansone. The Triad of Failure Modes and a Possible Way Out. In NeurIPS SSL Theory and Practice Workshop, 2023.
  59. E. Sansone and R. Manhaeve. GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning. arXiv, 2022.
  60. E. Sansone and R. Manhaeve. Learning Symbolic Representations Through Joint GEnerative and DIscriminative Training. In ICLR NeSy-GeMs Workshop, 2023a.
  61. E. Sansone and R. Manhaeve. Learning symbolic representations through joint generative and discriminative training (extended abstract). In IJCAI KBCG Workshop, 2023b.
  62. Classtering: Joint Classification and Clustering with Mixture of Factor Analysers. In ECAI, 2016.
  63. A Theoretical Analysis of Contrastive Unsupervised Representation Learning. In ICML, 2019.
  64. Understanding Contrastive Learning Requires Incorporating Inductive Biases. In ICML, 2022.
  65. Self-Supervised Learning Through Efference Copies. In NeurIPS, 2022.
  66. J. Song and S. Ermon. Understanding the Limitations of Variational Mutual Information Estimators. ICLR, 2020.
  67. Understanding Self-Supervised Learning Dynamics Without Contrastive Pairs. In ICML, 2021.
  68. T. Tieleman. Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient. In ICML, 2008.
  69. Pushing the Limits of Self-Supervised ResNets: Can We Outperform Supervised Learning Without Labels on ImageNet? In ICML Pre-Training Workshop, 2022.
  70. Contrastive Estimation Reveals Topic Posterior Information to Linear Models. JMLR, 2021.
  71. On Mutual Information Maximization for Representation Learning. In ICLR, 2019.
  72. T. Wang and P. Isola. Understanding Contrastive Representation Learning Through Alignment and Uniformity on the Hypersphere. In ICML, 2020.
  73. On the Importance of Asymmetry for Siamese Representation Learning. In CVPR, 2022.
  74. Z. Wen and Y. Li. The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. In NeurIPS, 2022.
  75. An Investigation into Whitening Loss for Self-supervised Learning. In NeurIPS, 2022.
  76. Contrastive Training for Improved Out-of-Distribution Detection. In arXiv, 2020.
  77. Generative-Contrastive Learning for Self-Supervised Latent Representations of 3D Shapes from Multi-Modal Euclidean Input. arXiv, 2023.
  78. A Theory of Generative Convnet. In ICML, 2016.
  79. K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  80. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In ICML, 2018.
  81. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In ICML, 2021.
  82. Zero-CL: Instance and Feature Decorrelation for Negative-Free Symmetric Contrastive Learning. In ICLR, 2021.
  83. S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation. In CVPR, 2020.
  84. Contrastive Learning Inverts the Data Generating Process. In ICML, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Emanuele Sansone (14 papers)
  2. Robin Manhaeve (12 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now