Learn from your own latents and not from tokens: A sample-complexity theory

Abstract: Generative models, from diffusion models to LLMs, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their \emph{own} latent representations of related views or masked regions, as in data2vec and JEPA -- an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth $L$. For such data, supervised or token-level SSL require a number of samples \emph{exponential} in $L$ to recover the latent tree; we prove that latent prediction achieves this with a number of samples \emph{constant} in $L$, up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant.

• The paper shows that learning via latent prediction achieves sample complexity scaling of m^3, independent of the depth of latent hierarchies.
• It introduces Iterative Latent Clustering (ILC) and Stacked Latent Clustering (SLC) to recover latent structures with rigorous theoretical guarantees.
• Empirical results confirm that latent-based self-supervision outperforms token-level objectives, providing a basis for more data-efficient generative models.

Summary: Learn from your own latents and not from tokens: A sample-complexity theory

Motivation and Context

The paper addresses a fundamental problem in generative modeling and self-supervised learning: modern deep networks (e.g., diffusion models and LLMs) require orders of magnitude more training data than biological learners, despite their capacity for complex generalization. While token-level pretraining (such as next-token prediction or masked language modeling) is widespread, it carries a substantial sample complexity penalty. The authors hypothesize that predicting in latent spaces, rather than raw tokens, may yield exponentially more efficient learning, formalizing this intuition by studying hierarchical compositional data generated by context-free grammars.

Theoretical Framework: Random Hierarchy Model

The paper adopts the Random Hierarchy Model (RHM), a probabilistic context-free grammar with a fixed tree structure and randomly drawn production rules. RHM generates observable sequences by recursively applying production rules along a hierarchy of latent variables. Crucially, the depth $L$ of the latent hierarchy directly impacts the sample complexity of learning: both supervised end-to-end learning and token-level self-supervised learning suffer exponential scaling with $L$ (i.e., $P \sim m^L$ and $P \sim m^{L+1}$ respectively, where $m$ is the number of rules per symbol).

Latent-Based Self-Supervised Learning

The central contribution is a rigorous demonstration that learning via latent prediction—i.e., learning from one's own latent representations—achieves sample complexity $P \sim m^3$ independent of depth $L$, up to logarithmic factors. This is exponentially more efficient than token-level objectives. The results are developed and validated via:

1. Iterative Latent Clustering (ILC): A recursive algorithm that, at each level, clusters grammatical tuples by their empirical context vectors (conditioned on cousin tuples) to recover latent variables and their production rule classes. Theoretical guarantees (under balanced and separated grammar assumptions) are established for full recovery of the latent hierarchy.
2. Stacked Latent Clustering (SLC) Network: A differentiable neural architecture constructed from modules that operate in predictor-clusterer pairs, with each module learning to cluster latent representations at its designated scale. End-to-end gradient descent training achieves the $m^3$ scaling across all hierarchy levels.
3. Data2vec Sample Complexity Analysis: The authors present the first sample complexity analysis of data2vec, showing both empirically and theoretically that this method, which predicts the encoder's own (teacher-network) latent representations, implicitly performs recursive hierarchical latent prediction. Its sample complexity collapses onto $m^3$ for non-root hierarchy recovery.

Empirical Validation

Numerical experiments confirm theoretical scaling predictions. Both the ILC and SLC achieve sharp recoveries of latent hierarchies at sample sizes that scale as $m^3$, verified via probe tasks and clustering scores. The analysis demonstrates that stacked hierarchical architectures or explicit stacking of self-distillation modules (e.g., H-JEPA) are largely redundant, as implicit hierarchical supervision arises naturally from latent prediction.

Figure 1: Recursive clustering recovers the RHM hierarchy at the predicted scale, with clustering and class reconstruction accuracy showing a sharp threshold after rescaling by $L$0.

Figure 2: Data2vec achieves root-classification accuracy on pooled features as a function of sample size, exhibiting scaling collapse at $L$1 and outperforming token-level objectives.

Figure 3: Synonym clustering score in the encoder rises sharply as a function of sample size and collapses after rescaling by $L$2.

Figure 4: Ablations show local learning suffices; stopgradients and removal of teacher network do not degrade sample complexity scaling.

Figure 5: Layer-wise training dynamics; clustering improves sequentially at each layer, propagating improved prediction loss to subsequent layers.

Figure 6: Attempted scaling collapse with token-based scaling ($L$3) shows degradation, validating $L$4 as the correct scaling.

Figure 7: SLC model exhibits depth-independent sample complexity with probe accuracy invariant to hierarchy depth.

Figure 8: Synonym clustering scores and linear probe accuracy per encoder layer and RHM level show latent clustering occurs layer-by-layer.

Figure 9: Offline data2vec pretraining shows linear and MLP probe accuracy collapse by $L$5 for different training set sizes.

Implications and Directions

Practical Implications

The results suggest that own-latent prediction objectives amplify the statistical signal relevant for learning hierarchical structure, enabling correspondence to natural data scaling laws. They raise the possibility that generative models trained with latent-space self-supervision may systematically outperform token-level approaches in data efficiency and scaling regimes. Controlled comparisons with token-level baselines across training-set sizes would empirically validate this prediction.

Biological Plausibility

By demonstrating that local (layer-wise) learning rules and architectural bottlenecks suffice to achieve optimal sample complexity, the paper provides a potential mechanistic explanation for the superior sample efficiency of biological learners. The recursive clustering mechanism is conceptually aligned with predictive coding theories of neocortex and biologically plausible algorithms (e.g., CLAPP, contrastive predictive coding).

Theoretical Limitations

The RHM is intentionally synthetic—fixed tree topology, non-recursive rules, and context-free generativity. Extensions to variable, recursive, and context-dependent grammars (closer to natural language or vision) are necessary to determine generality and limitations. A range of recent hierarchical self-supervised architectures (e.g., Bootleg, V-JEPA) warrant theoretical analysis for their implicit or explicit hierarchical mechanisms.

Future Research Directions

• Extension to grammars with variable topology, recursion, and context dependency to model natural data more realistically.
• Systematic empirical evaluation of latent-based objectives compared to token-level objectives across data scales and modalities.
• Theoretical and practical study of novel hierarchical SSL architectures, including multi-level teacher distillation and multi-scale prediction paths.
• Application of synthetic data models to accelerate architectural search and theoretical understanding prior to scaling.

Conclusion

This work rigorously establishes that self-supervision on latents drastically reduces the sample complexity required to learn hierarchical latent structures, placing the inefficiency of token-level objectives on a quantitative foundation. The findings have immediate implications for the design of scalable, data-efficient generative models and suggest new directions for understanding both machine and biological learning.