Proving and Improving the Platonic Representation Hypothesis with JEPAs
This presentation examines a theoretical and empirical investigation of when independently trained encoders for different modalities converge to the same representation of the world. The paper proves that cross-modal alignment is determined not by scale alone, but by whether positive training pairs correspond to multiple observations of the same world state. Through controlled experiments and a novel good-view JEPA architecture, the authors demonstrate that a 109-million-parameter text encoder can exceed the cross-modal alignment of 72-billion-parameter language models while requiring 42 times fewer training samples.Script
Under what conditions do separately trained image and text encoders learn to represent the same world? The Platonic Representation Hypothesis claims they converge with enough scale, but this paper proves the answer depends on something more fundamental: whether your training pairs show multiple views of one scene, or multiple corruptions of one observation.
The authors distinguish good views, which are independent observations of the same world state like two captions of one image, from bad views, which are corruptions of a single observation like two crops of one photo. Next-token prediction turns out to be a special case of bad views, because a prefix and its continuation share the same document rather than describing a shared scene from independent perspectives.
The Platonic Representation Theorem formalizes this intuition. If two independently trained JEPAs use the same positive-pair distribution over world states, and their minimizers are unique up to rotation, then both recover the same world representation up to a modality-specific orthogonal transformation. Cross-modal alignment is not an emergent property of scale; within the stated assumptions, it follows directly from shared positive-pair structure.
On Flickr captions, a good-view text JEPA retrieves the correct image-matched caption first 44 times more often than random chance. The same model trained with next-token prediction retrieves it correctly only 21 times above chance. When scaled to web data, good views reach equivalent alignment with 42 times fewer samples and 435 times fewer unique images than next-token prediction, enabling a 109 million parameter text encoder to surpass the measured cross-modal agreement of language models up to 72 billion parameters.
Because the alignment is geometric, independently trained encoders can be composed at inference time. A text decoder trained only on text-JEPA embeddings can generate captions from frozen image representations using a single orthogonal map fitted on paired data. On COCO, this approach reaches a CIDEr score of 0.838 with no network trained on image-caption pairs and a rotation fitted only on Flickr validation examples.
The learned representations share enough geometric structure that the rotation can even be recovered from unpaired data. Using only cluster matching and optimal transport on independent image and caption clouds, the good-view JEPA retrieves correct captions 16.4 percent of the time without a single labeled image-text pair, while next-token prediction fails in half its runs. The Platonic Representation Hypothesis is no longer a scaling conjecture; it is a theorem about positive pairs. To learn more about this work and create your own research videos, visit EmergentMind.com.
