Identifiability in Predictive SSL

• Predictive SSL is defined as learning representations that recover true latent factors up to an affine transformation, ensuring unique data alignment.
• The LDM framework unifies predictive SSL by matching encoded latent distributions with parameterized models (Gaussian or vMF), establishing rigorous identifiability conditions.
• Recent identifiability theorems emphasize encoder invertibility, full-rank noise models, and structured predictors, guiding the design of robust SSL algorithms.

Identifiability in predictive self-supervised learning (SSL) concerns the conditions under which the learned representations correspond to true latent factors of the underlying data-generating process, modulo trivial affine transformations. Recent developments recast major SSL paradigms within the Latent Distribution Matching (LDM) framework, yielding precise identifiability theorems, clarifying necessary assumptions, and establishing rigorous benchmark criteria for predictive SSL methods. These advances situate predictive SSL results between classical Independent Component Analysis (ICA) theory, modern contrastive methods, and emerging theories of representation learning dynamics and sample complexity.

1. Definitions and Context of Identifiability in Predictive SSL

In predictive SSL, identifiability signifies that the mapping from observations to latent representations by an encoder $f$ inverts the (potentially unknown) data-generating process $g$ up to an affine transformation. Explicitly, for data $x_t = g(\tilde z_t)$ generated from latent variables $\tilde z_t$ through an invertible, differentiable map, and encoder $z_t = f(x_t)$, identifiability requires

$z_t = A\,\tilde z_t + c$

for all $t$, with $A$ an invertible matrix and $c$ a vector, ruling out arbitrary nonlinear invertible reparameterizations. This is stronger than the permutation and scaling invariance common in ICA or contrastive setups. In the more rigorous formalism, absolute identifiability is achieved when there exists a group transformation $T$ such that $g$0 for almost all $g$1; relative identifiability refers to two learned encoders differing only by such a transformation (Mikulasch et al., 5 May 2026, Reizinger et al., 17 Apr 2025).

2. The Predictive Latent Distribution Matching (LDM) Framework

The LDM formalism unifies predictive SSL methods by positing a parameterized predictive distribution over latent trajectories, and matching this to the empirical (encoded) latent distribution via forward Kullback-Leibler divergence. Concretely, given:

• Deterministically encoded latent states: $g$2,
• Joint empirical latent trajectory distribution: $g$3,
• Model latent process: $g$4 (with $g$5 usually Gaussian or von Mises–Fisher), the LDM objective is

$g$6

where alignment (expected log-likelihood) matches latent predictive structure, and uniformity (conditional entropy) prevents collapse (Mikulasch et al., 5 May 2026).

3. Identifiability Theorems for Predictive SSL

The central results establishing identifiability in predictive SSL under LDM are as follows:

Theorem 1 (Gaussian Predictor)

If data arise from a latent process $g$7 followed by a diffeomorphism $g$8, and the encoder $g$9 is invertible and differentiable on the data manifold, and the model cover is full-rank (i.e., nondegenerate predictor Jacobian and $x_t = g(\tilde z_t)$0), then at any optimizer with $x_t = g(\tilde z_t)$1,

$x_t = g(\tilde z_t)$2

with $x_t = g(\tilde z_t)$3 invertible and $x_t = g(\tilde z_t)$4 constant (Mikulasch et al., 5 May 2026).

Theorem 2 (von Mises–Fisher Predictor)

Under the corresponding manifold and distributional assumptions for vMF conditionals, the same affine identifiability is achieved.

Proof Sketch:

• At optimum, all conditionals $x_t = g(\tilde z_t)$5.
• Change-of-variables formula links the true and recovered latent densities via $x_t = g(\tilde z_t)$6.
• Differentiating the resulting equation twice with respect to $x_t = g(\tilde z_t)$7 and $x_t = g(\tilde z_t)$8 forces the Jacobian of $x_t = g(\tilde z_t)$9 to be constant, implying $\tilde z_t$0 is affine.

Critical Assumptions:

• Invertibility of $\tilde z_t$1 on the data manifold (enforced by entropy maximization).
• Factorized predictive model with explicit Gaussian or vMF noise (fixes local latent geometry).
• Nondegeneracy (full-rank noise, expressive predictor).

Absence of these assumptions (e.g., missing noise model or non-invertible encoder) results in loss of identifiability, as arbitrary nonlinear reparameterizations become possible.

4. Comparison with Other SSL Paradigms

A variety of SSL methods fit into the LDM formalism, but identifiability guarantees depend on their properties:

Paradigm	Prior/Conditional	Identifiability
ICA	Factorized prior	Permutation, scaling
Contrastive SSL	vMF prior/conditional	Orthonormal, affine
Predictive SSL (LDM)	Gaussian/vMF dynamics	Affine
Non-contrastive SSL	Gaussian (ad-hoc)	No formal guarantee

• ICA: Recovers latents up to permutation and scaling when using a factorized prior.
• Contrastive SSL (SimCLR, InfoNCE): Under temporal structure and the vMF prior, identifiability can be up to an affine map, provided the map is invertible and augmentation kernels are well modeled (Reizinger et al., 17 Apr 2025).
• Non-Contrastive SSL (VICReg, Barlow Twins): Typically lacks formal proofs; regularizers are necessary but not sufficient for full identifiability.
• Predictive SSL: When implemented as LDM with explicit noise model, provides the strongest affine identifiability results (Mikulasch et al., 5 May 2026).

5. Singular Identifiability Theory and Extensions

Classical identifiability theory (IT) for SSL is asymptotic and limited to idealized data-generating processes. Singular Identifiability Theory (SITh) extends this to pathological regimes and the entire SSL pipeline, covering:

• Real-world augmentation (joint/correlated perturbations, non-exponential family noise),
• Encoder/decoder architectures with non-optimal initialization, stop-gradient, predictor heads,
• Loss landscapes with degenerate Hessian spectra,
• Finite-sample and optimization effects.

SITh Framework (Reizinger et al., 17 Apr 2025):

• For loss $\tilde z_t$2 covering realistic pretext tasks and augmentations, the global minima correspond to encoders invertible up to a known symmetry group $\tilde z_t$3. The singularity structure of the Hessian isolates the symmetry directions.
• Typical SITh assumptions (A1–A6): augmentation realism, manifold support, universal function class, sufficient batch/negatives diversity, optimizer regularity, and symmetry isolation.

This suggests practical identifiability can be diagnosed for realistic SSL pipelines through analysis of training dynamics, augmentation structure, and architecture–optimizer regularities.

6. Practical Design and Implications for High-Dimensional Time Series

For predictive SSL applications (e.g., neural population recordings, video data), formal identifiability carries several concrete prescriptions (Mikulasch et al., 5 May 2026):

• Choose a parametric one-step latent model (e.g., $\tilde z_t$4, vMF) with appropriate noise to match data ambiguity.
• Use an expressive encoder; rely on the uniformity term (entropy maximization) to enforce invertibility, rather than ad hoc architectural constraints.
• Methods such as BYOL, SimSiam, CPC, and JEPA are interpretable within LDM; only those consistent with a known noise model exhibit guaranteed identifiability.
• Predictive LDM can be equipped with, for instance, a Kalman-filter backbone to produce both identifiable latents and principled uncertainty estimates.
• Empirically observed phenomena such as the “projector phenomenon,” grokking, or sharpness–batch size tradeoffs point toward further exploration of the SITh regime.

A plausible implication is that techniques ensuring explicit latent stochasticity, and precise noise modeling, are critical for the design of SSL algorithms with provable identifiability.

7. Open Challenges and Research Directions

Key unresolved problems and actionable research directions include (Reizinger et al., 17 Apr 2025):

• Modeling of realistic augmentations and data-generating processes, especially beyond simple noise forms.
• Characterization of partial layer collapse and the identifiability of intermediate representations.
• Incorporation of compositional and causal structure to support out-of-distribution generalization.
• Benchmarking tools utilizing synthetic datasets with ground-truth factors to evaluate identifiability directly.
• Nonasymptotic statistical analysis (finite sample bounds, Fisher information characterization), and the precise role of inductive biases and optimization dynamics.

The ongoing unification through LDM and SITh is expected to sharpen theoretical guarantees across SSL paradigms and clarify the empirical conditions under which semantically meaningful, identifiable representations emerge (Mikulasch et al., 5 May 2026, Reizinger et al., 17 Apr 2025).