Self-Predictive Representation

• Self-predictive representation is a method where latent encodings are trained to predict their own future states, anchoring the representation in system dynamics.
• It employs action-conditional and policy-conditioned predictors with stop-gradient techniques to prevent collapse and ensure robust feature learning.
• Empirical studies show that this approach improves sample efficiency and transfer learning in reinforcement learning, time-series, and spatial-temporal tasks.

Self-predictive representation refers to a class of learning objectives and corresponding algorithmic frameworks that induce representations by requiring them to predict their own future states, typically through the agent’s latent dynamics, either with or without explicit conditioning on actions. This approach pervades modern reinforcement learning (RL), sequence modeling, self-supervised learning, and neuroscience-inspired architectures, with diverse methodological realizations across domains. The unifying principle is to anchor the semantics of a latent representation in its ability to forecast itself under the system’s transition or evolution, thereby discovering features encoding the actionable or predictive structure of the environment, history, or input (Khetarpal et al., 2024, Tang et al., 2022, Bai et al., 2020, Ni et al., 2024).

1. Core Principles and Formal Definitions

A self-predictive representation is a map $\Phi: \mathcal{X} \rightarrow \mathbb{R}^k$ (or more generally, a state/history encoder $\phi: h_t \mapsto z_t$), together with a predictor (often parameterized as $P$, $P_a$ for each action, or a neural dynamics predictor). The key objective is to minimize prediction error in the latent space, measured as mean-squared error or related discrepancies between the predicted future representation and a target (often computed with a “stop-gradient” for stability):

$$L(\Phi, P) = \mathbb{E}_{x, y} \left\| P^\top \Phi^\top x - \text{stopgrad}(\Phi^\top y) \right\|_2^2$$

where $y$ is typically the next state under a transition or dynamics model (e.g., $y \sim T^\pi(\cdot|x)$ in RL) (Khetarpal et al., 2024, Tang et al., 2022, Ni et al., 2024).

This objective can be action-conditional (predicting $\Phi(y)$ given $x$ and $a$), multi-step, and/or recurrent—generalizing to sequence and partially observable settings. In self-predictive RL, the approach is related to bisimulation, belief-state inference, and model-based latent prediction. In self-supervised representation learning, analogous objectives arise in Joint-Embedding Predictive Architectures (JEPA), Predictive Coding, and mutual-information-based forecasting [3D-JEPA: (Hu et al., 2024), Graph-JEPA: (Skenderi et al., 2023, Ororbia et al., 22 Mar 2025, Bai et al., 2020)].

2. Algorithmic Methodologies

2.1. RL-Specific Self-Predictive Objectives

Recent formulations in RL partition self-predictive losses as:

• Policy-conditioned self-prediction (BYOL-$\Pi$):

$$L_{\text{BYOL-}\Pi}(\Phi, P) = \mathbb{E}_{x, y \sim T^\pi} \|P^\top \Phi^\top x - \text{stopgrad}(\Phi^\top y)\|_2^2$$

This encodes the state-transition operator under a fixed policy, leading to representations aligned with the principal eigenspace of $(T^\pi)^2$ (Khetarpal et al., 2024, Tang et al., 2022).

• Action-conditional self-prediction (BYOL-AC):

$$L_{\text{BYOL-AC}}(\Phi, \{P_a\}) = \mathbb{E}_{x, a, y \sim T_a} \|P_a^\top \Phi^\top x - \text{stopgrad}(\Phi^\top y)\|_2^2$$

Each action has its own predictor, enabling the features to represent action-contingent dynamics. This is empirically more robust in complex environments (Khetarpal et al., 2024).

• Variance-like action-conditional objective (BYOL-VAR):

$$L_{\text{BYOL-VAR}} = L_{\text{BYOL-AC}} - L_{\text{BYOL-}\Pi}$$

This isolates the variance of per-action dynamics beyond the policy-average, targeting neural encodings of advantage structure (Khetarpal et al., 2024).

All variants employ a “stop-gradient” on the target to prevent representational collapse, and typically rely on inner- and outer-loop updates or two-timescale optimization (Tang et al., 2022, Ni et al., 2024).

2.2. General Sequence and Predictive Objectives

In sequence and time-series modeling, self-predictive objectives appear as:

• Predictive Information Maximization: Maximizing mutual information between past and future latent windows (e.g., $I(Z^{\mathrm{past}}; Z^{\mathrm{future}})$), under exact or approximate estimation, regularized to maintain input reconstructions (Bai et al., 2020).
• Joint-Embedding Predictive Architectures (JEPA): Predicting the latent embedding of “target” signal from a “context” signal, never reconstructing observed data or using negatives (e.g., 3D-JEPA, Graph-JEPA) (Hu et al., 2024, Skenderi et al., 2023).
• Predictive Coding and Free-Energy Frameworks: Predicting latent representations across parallel streams, as in Meta-Representational Predictive Coding (MPC), without explicit negative samples or generative decoding (Ororbia et al., 22 Mar 2025).

3. Theoretical Properties: Dynamics, Non-collapse, and Spectral Structure

3.1. ODE and Lyapunov Analysis

In the idealized setting, the continuous-time learning dynamics induced by the self-predictive loss (with inner-loop predictor minimization and semi-gradient updates for the encoder) can be characterized by an ODE for the representation parameters:

$\dot{\Phi} = -\nabla_\Phi L(\Phi, P^*)$

with $P^* = \arg\min_P L(\Phi, P)$, leading to dynamical systems whose Lyapunov functionals ensure non-collapse ($\Phi^\top \dot{\Phi} = 0$) and convergence to an invariant subspace corresponding to top eigenvectors or singular components of the environment dynamics (Tang et al., 2022, Khetarpal et al., 2024).

3.2. Subspace Characterization and Value Function Correspondence

• BYOL-$\Pi$ maximizes the trace objective $$f_{\text{BYOL}-\Pi}(\Phi) = \mathrm{Tr}[(\Phi^\top T^\pi \Phi)^2]$$, selecting the principal components of squared policy transition.
• BYOL-AC generalizes this to per-action transitions, capturing the sum over squared $T_a$.
• The variance relation $$|A|^{-1} \sum_a D_a^2 = (|A|^{-1} \sum_a D_a)^2 + \operatorname{Var}_a(D_a)$$ demonstrates that action-conditional self-prediction incorporates both the average squared effect and the variance across actions in the induced representation (Khetarpal et al., 2024).

This spectral perspective underlies why self-predictive loss features are effective for downstream value, Q-value, and advantage fitting (Khetarpal et al., 2024).

4. Empirical Validation and Practical Guidelines

Extensive empirical studies confirm that self-predictive representation learning achieves:

• Robustness against representational collapse, especially with stop-gradient or EMA target encoders.
• Enhanced sample-efficiency and downstream performance in RL, including multitask RL, DeepMind Lab-30, Atari-57, MiniGrid, MuJoCo, and CARLA.
• Superior transfer and generalization, particularly with action-conditional (BYOL-AC) or multi-step predictors (Khetarpal et al., 2024, Kim et al., 5 Jun 2025, Guo et al., 2020).
• State-of-the-art accuracy in time-series, spatial-temporal, and graph SSL tasks (e.g., ST-ReP, 3D-JEPA, Graph-JEPA) without reliance on negative sampling (Zheng et al., 2024, Hu et al., 2024, Skenderi et al., 2023).

A consensus emerges that, in practice, a minimalist RL implementation—auxiliary forward prediction in latent space, stop-gradient target, and joint TD loss—achieves most of the gains attributed to more complex schemes (Ni et al., 2024).

5. Broader Context and Variants

Self-predictive objectives can be construed in relation to wider classes of self-supervised learning:

• Contrastive learning: While contrastive losses (e.g., CPC, InfoNCE) rely on positive and negative pairs, self-predictive approaches avoid negatives and log/exp instabilities, yielding bounded, stable objectives (see Relative Predictive Coding) (Tsai et al., 2021).
• Predictive Coding in Neuroscience: Computational models posit a role for hippocampus as an auxiliary predictive system in supporting flexible, memory-guided behavior; self-predictive neural networks exhibit parallel structure and representational transitions (Fang et al., 2023).
• Bayesian Inference and Belief Representation: In partially observable domains, self-predictive modules enable low-dimensional bottlenecks to recover Bayes-optimal beliefs, enhancing policy generalization and interpretability (Kuo et al., 24 Oct 2025).
• Meta-Representational and Biologically Plausible Coding: Self-predictive learning is extended to Hebbian, local learning rules and cross-stream message-passing for distributed, biologically-inspired inference (Ororbia et al., 22 Mar 2025).

6. Summary and Key Empirical Findings

Self-predictive representation learning is grounded in representation forecasting: requiring latent state encodings to predict successor latents, either in a policy- or action-conditional manner. This principle, realized in both RL and self-supervised sequence models, consistently yields representations with powerful spectral and information-theoretic properties:

• It anchors learned features in system dynamics, ensures non-collapse, and aligns with optimal subspaces for value, Q-value, or advantage prediction (Khetarpal et al., 2024, Tang et al., 2022).
• Empirical performance in RL is best when the auxilary loss is action-conditional (BYOL-AC), which robustly transfers across tasks and improves generalization under distractors (Khetarpal et al., 2024).
• Extensions to non-RL domains, including spatio-temporal forecasting (ST-ReP), 3D vision (3D-JEPA), and graph SSL (Graph-JEPA), confirm that the paradigm of latent-space self-prediction can scale efficiently and outperform contrastive or reconstruction baselines (Zheng et al., 2024, Hu et al., 2024, Skenderi et al., 2023).

Self-predictive methods thus constitute a general, spectrally motivated paradigm for representation learning, with formal guarantees and considerable empirical validation across modalities and domains (Khetarpal et al., 2024, Tang et al., 2022, Ni et al., 2024, Guo et al., 2020).