Efficient and scalable learning of sufficient representations

Develop an efficient and scalable self-supervised learning procedure to learn a representation that meets the stated sufficiency and effectiveness criteria, where "efficient" means low computational and memory cost and "scalable" means the optimization can be easily scaled up.

Background

Alongside the question of what representation is sufficient and effective, the authors explicitly raise the question of how to learn such a representation efficiently and at scale. They define efficiency in terms of computational and memory cost, and scalability in terms of ease of scaling the optimization procedure, underscoring the practical need for algorithms that can handle large datasets and models.

This open question drives the algorithmic developments and discussions throughout the paper, including their spectral contrastive, energy-based, latent-variable, and nonlinear formulations.