Data2vec: Hierarchical Sample Complexity

• The paper introduces a rigorous sample complexity theory for data2vec, demonstrating that hierarchical latent prediction recovers deep structures with nearly depth-independent samples.
• It employs a Recursive Hierarchical Model formalized as a PCFG to show that token-level SSL requires exponentially more samples than latent-prediction approaches.
• The Iterative Latent Clustering algorithm and neural instantiation validate the theoretical scaling, highlighting practical advantages in efficient latent recovery.

Data2vec is a predictive self-supervised learning paradigm in which networks are trained to predict their own latent representations of masked or related inputs, as opposed to predicting only observed (token-level) data. While data2vec and similar latent-prediction methods (such as JEPA) have shown remarkable empirical data efficiency, a rigorous theoretical explanation of their sample complexity advantages remained elusive. Recent work provides the first complete sample-complexity theory for data2vec and related methods, using a tractable probabilistic context-free grammar (PCFG) that formalizes compositional latent structure reminiscent of natural language and images. This theory reveals that data2vec implicitly performs a hierarchical latent prediction, resulting in a sample complexity almost independent of the hierarchy depth, in stark contrast to conventional token-level methods.

1. Formal Model: Recursive Hierarchical Model (RHM) as Context-Free Grammar

The analytical framework employs a Recursive Hierarchical Model (RHM), formalized as a PCFG of depth $L$, branching factor $s$, and symbol vocabularies of size $v$. The hierarchy comprises levels $\ell = 0$ (visible tokens) up to $\ell = L$ (root latent), with $s^{L-\ell}$ symbols $h^{(\ell)}_u\in\mathcal{V}_\ell$ at each level. The observed data is the visible string $x = (x_1,\ldots,x_{s^L})$ with $x_i = h^{(0)}_i$.

Production rules at each level $\ell$ select $s$0 distinct $s$1-tuples from $s$2, partitioned as $s$3 ($s$4) for each $s$5. Each $s$6 uniquely defines its parent via an injective mapping $s$7. Data generation is top-down: $s$8, and each non-leaf node samples its $s$9 children tuples from $v$0 uniformly.

Two learning settings are defined:

• Token-level SSL/Supervised: Predict a visible token or a root label $v$1 from $v$2.
• Latent-prediction SSL: Recursively decode latent variables across hierarchical levels using previously decoded latents as both context and target (Korchinski et al., 26 May 2026).

2. Theoretical Sample Complexity Results

The analysis demonstrates a sharp dichotomy in sample complexity between token-level and latent-prediction objectives for structured data generated by the hierarchical PCFG.

Token-Level SSL

Any method restricted to predicting visible tokens (whether supervised or self-supervised) is shown to require a number of samples exponential in the hierarchy depth $v$3 to recover the full latent tree to level $v$4. Specifically:

$v$5

This exponential lower bound holds under broad regularity (balanced grammar), implying severe inefficiency of token-level objectives for deeply hierarchical data.

Latent-Prediction SSL

An efficient "iterative latent clustering" (ILC) approach is shown to recover all non-root latents $v$6 using a number of samples independent of depth $v$7, up to logarithmic factors:

$v$8

where $v$9 characterizes rule sparsity. For fixed $\ell = 0$0, the sample complexity remains constant as $\ell = 0$1 grows.

3. Proof Strategy and Principal Lemmas

The theoretical results are underpinned by a sequence of invariance, concentration, and clustering arguments:

• Correlation-based invariances: At level $\ell = 0$2, $\ell = 0$3-tuples $\ell = 0$4 are grouped into synonym classes $\ell = 0$5 sharing identical context vectors

$\ell = 0$6

where $\ell = 0$7 is a "cousin" token at level $\ell = 0$8.

• Synonym invariance: $\ell = 0$9.
• Concentration lemma: Empirical estimates $\ell = L$0 from $\ell = L$1 i.i.d. cousins concentrate tightly:

$\ell = L$2

with high probability.

• Stable clustering: If $\ell = L$3 and true centers are $\ell = L$4-separated, a $\ell = L$5-means (or any stable $\ell = L$6-clusterer) recovers the synonym classes exactly.
• Inductive decoding: The observed level $\ell = L$7 anchors the recursion; once $\ell = L$8-level latents are decoded, the estimation problem for level $\ell = L$9 reduces to an isomorphic RHM.

The main proof proceeds by induction over levels, leveraging these lemmas to guarantee exact recovery at every level, given $s^{L-\ell}$0 samples per level.

4. Iterative Latent Clustering (ILC) Algorithm

The ILC algorithm operationalizes the theoretical ideas as a multi-level clustering scheme. At each level $s^{L-\ell}$1:

1. Form all empirical $s^{L-\ell}$2-tuples $s^{L-\ell}$3 from current estimates $s^{L-\ell}$4.
2. Estimate the tuple support $s^{L-\ell}$5 observed in samples.
3. Collect empirical context vectors:

$s^{L-\ell}$6

for a fixed cousin $s^{L-\ell}$7.

1. Cluster $s^{L-\ell}$8 into $s^{L-\ell}$9 groups using a stable $h^{(\ell)}_u\in\mathcal{V}_\ell$0-clusterer.
2. Assign tuples to clusters, yielding next-level latents $h^{(\ell)}_u\in\mathcal{V}_\ell$1.

By union bounding the relevant concentration and stability guarantees, the algorithm achieves exact latent recovery at all non-root levels using $h^{(\ell)}_u\in\mathcal{V}_\ell$2 samples—$h^{(\ell)}_u\in\mathcal{V}_\ell$3 up to logarithmic corrections.

5. Neural Network Instantiation and Scaling Behavior

A neural SLC (Stacked Latent Clustering) architecture is constructed as a stack of $h^{(\ell)}_u\in\mathcal{V}_\ell$4 identical modules, each containing:

• Predictor $h^{(\ell)}_u\in\mathcal{V}_\ell$5: Consumes $h^{(\ell)}_u\in\mathcal{V}_\ell$6-tuples of latents at level $h^{(\ell)}_u\in\mathcal{V}_\ell$7, outputting a distribution for a cousin token via cross-entropy, serving as a neural surrogate for $h^{(\ell)}_u\in\mathcal{V}_\ell$8.
• Clusterer $h^{(\ell)}_u\in\mathcal{V}_\ell$9: Maps context vectors to soft one-hot assignments over $x = (x_1,\ldots,x_{s^L})$0 clusters using a contrastive loss, implementing $x = (x_1,\ldots,x_{s^L})$1.
• Architecture propagation: The soft output at level $x = (x_1,\ldots,x_{s^L})$2 recurses as input tokens to $x = (x_1,\ldots,x_{s^L})$3. Weight-tying or EMA teachers prevent degenerate solutions.

Empirically, root-label classification via a linear probe on the top-level SLC features transitions sharply once $x = (x_1,\ldots,x_{s^L})$4, matching theoretical scaling. Ablations confirm that local learning rules control data efficiency, with or without EMA or stop-gradient mechanisms.

6. Data2vec Mechanism and Hierarchical Prediction

Data2vec trains a student network to regress the teacher's top-$x = (x_1,\ldots,x_{s^L})$5 layer activations at masked input positions, with the teacher providing an EMA of the student parameters. The analysis makes two key assumptions:

• (A1) Target carries learned latents: After $x = (x_1,\ldots,x_{s^L})$6 phases are learned, the teacher's target decomposes as

$x = (x_1,\ldots,x_{s^L})$7

where $x = (x_1,\ldots,x_{s^L})$8 denotes the level-$x = (x_1,\ldots,x_{s^L})$9 ancestor.

• (A2) Gradient-descent learns any detectable correlation.

Learning proceeds by phase induction:

• Phase 0: Reduces to masked-token prediction, learning level-1 latents with $x_i = h^{(0)}_i$0.
• Phase $x_i = h^{(0)}_i$1: The target includes linear functions of $x_i = h^{(0)}_i$2; learning the mapping from decoded $x_i = h^{(0)}_i$3-tuples to teacher activations recasts as the same clustering problem, with identical sample bounds.
• After $x_i = h^{(0)}_i$4 phases, all non-root latents are present in outputs, and the full hierarchy is recovered at $x_i = h^{(0)}_i$5.

Empirical evidence is provided by the synonym-clustering score (at levels $x_i = h^{(0)}_i$6) sharply transitioning from $x_i = h^{(0)}_i$7 to $x_i = h^{(0)}_i$8 as $x_i = h^{(0)}_i$9 crosses a threshold, and by root-classification accuracy exhibiting the same scaling.

7. Consequences for Hierarchical Stacking Strategies

The analysis establishes that data2vec, despite implementing only a single-scale predictor-distiller, executes an effective multi-phase, multi-scale latent prediction. As a result, explicit hierarchical stacking of predictor-clusterer modules across scales, as in approaches such as H-JEPA, is largely redundant: no further improvement in sample efficiency is attainable. Across RHM data of depth $\ell$0 and local fan-out $\ell$1, any token-level method remains exponential $\ell$2 in data requirements, while latent-prediction, including data2vec, achieves $\ell$3 samples independent of $\ell$4. This fully accounts for the extreme data efficiency of latent-prediction methodology and demonstrates the sufficiency of the data2vec strategy for hierarchical latent structure recovery (Korchinski et al., 26 May 2026).