Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study

Abstract: Joint-embedding predictive (JEPA-style) objectives learn representations by predicting future latents. In doing so they can discard features that are exogenous (uncontrollable by the agent) yet control-relevant, even when those features are trivially encodable. This occurs because the objective optimizes temporal predictability rather than control-relevance. We isolate this failure mode in a controlled 2x2 experimental design that varies feature controllability and relevance independently, using a predictability knob that decouples a feature's temporal predictability from its control-relevance. Comparing six objectives: reconstruction, JEPA, action-conditioned JEPA, controllability-based JEPA, inverse dynamics under a random policy, and reward-grounded JEPA, we observe that all evaluated reward-free predictive objectives leave the exogenous control-relevant feature near chance accuracy, while a reward-grounded variant retains it selectively. The remedy is label-efficient and robust: as little as 2% of reward-labeled transitions recovers the feature, the effect holds across two environments with different surface forms, and it persists across latent dimensions from 16 to 1024. Comparing the learned latent geometry against bisimulation theory's prediction, the JEPA latent realizes only a small fraction of the class separation a supervised reference attains.

• The paper reveals that reward-free predictive objectives systematically omit unpredictable yet reward-relevant exogenous features, leading to a measurable information gap in representations.
• It uses a controlled 2×2 design in synthetic environments to isolate the effects of feature controllability and relevance, clearly showing the divergence from bisimulation theory.
• The study finds that minimal reward supervision (as little as 1–2%) robustly recovers the lost features, suggesting practical benefits for hybrid or semi-supervised models.

Predictive Objectives and the Structural Omission of Exogenous Control-Relevant Features

Problem Setting and Motivation

This work addresses a foundational question in self-supervised representation learning for control: whether predictive objectives such as JEPA (Joint-Embedding Predictive Architecture) structurally preserve control-relevant features that are exogenous (uncontrollable) and temporally unpredictable. The central hypothesis is that, when predictability and control-relevance are orthogonal, reward-free predictive objectives will drop features crucial for optimal policy but unpredictable from previous observations, thereby creating a measurable information gap in the latent representation.

The tension between bisimulation theory and predictive objectives is core to the formulation. Bisimulation dictates that features relevant for reward must be retained irrespective of their predictability, whereas predictive objectives such as JEPA optimize for the retention of only those features that are temporally predictable from the observation stream. The study isolates this divergence with a controlled $2 \times 2$ design, manipulating feature controllability and relevance, and uses an adjustable predictability knob to decouple predictability from reward-signalling. This approach addresses a limitation of prior work, which has targeted exogenous-irrelevant distractors but not exogenous-relevant structure.

Experimental Design

The central empirical system is QuadrantEnv, which generates observations with a local patch encoding the target feature amid a predictable multi-frequency background. Features can be set along two axes — controllability (whether agent actions govern transitions) and relevance (whether the feature determines reward). The exogenous control-relevant quadrant (cell 4) is the primary focus, representing the scenario where a reward-relevant feature is entirely beyond the agent's control and temporally unpredictable ($p_{\text{repeat}}=0.5$, i.i.d. Bernoulli).

Six objectives are considered: pixel reconstruction, JEPA, action-conditioned JEPA, controllability-based JEPA, inverse dynamics under a random policy, and reward-grounded JEPA, alongside a supervised reference. The encoder is byte-identical across objectives, ensuring any observed retention difference is attributable to the objective signal only.

Main Empirical Findings

The objective $\times$ cell retention matrix (Figure 1) provides the central result: all reward-free predictive objectives (JEPA and its variants), as well as inverse dynamics under random actions, fail to retain the exogenous control-relevant feature—linear probe accuracy sits at chance ($\approx 0.51$) and InfoNCE MI is near zero, despite a non-collapsed, high-rank latent (selective dropping, not collapse). Only the reward-grounded variant (jepa_reward), the reconstruction reference, and the supervised reference successfully retain the feature.

Figure 1: Retention matrix shows only reward-grounded and reference objectives retain the exogenous control-relevant feature (cell 4); all self-predictive objectives omit it.

This pattern is robust across two environments (Quadrant and a spectrally different SwitchColor form), persists across batch size, architecture, and latent dimensionality (16 to 1024), and is insensitive to the richness of the observation space. The reward-grounded variant exhibits selectivity: it retains only the relevance axis; controllability-based objectives, by contrast, retain only along the controllable axis, highlighting a misalignment of optimization criterion and control affordance.

Figure 2: Cell-4 probe accuracy and InfoNCE MI for each objective; reward-grounded variant attains ceiling, others stay at chance/zero.

Scalability experiments (Figure 3) demonstrate the failure is objective-structural: increasing latent size does not rescue retention for JEPA; only reward-grounded approaches scale robustly.

Figure 3: Cell-4 retention versus latent dimension; JEPA never crosses the retention threshold at any scale, while reward-grounded JEPA is robust except at the largest, likely budget-constrained, dimension.

Label-efficiency analysis (Figure 4) reveals that as little as 2% of reward-labeled transitions suffice for full retention of the exogenous control-relevant feature, and even 1% is enough to reliably beat chance, indicating substantial practical efficiency for reward supervision under these controlled conditions.

Figure 4: High label-efficiency for reward grounding—2% labeled transitions suffice to surpass retention threshold.

A geometric analysis in the bisimulation framework further sharpens the claim: for the exogenous control-relevant feature, bisimulation theory predicts a separation of 1.0 (analytical), realized at approximately 2.0 by the supervised and reference models, but JEPA achieves only 0.1—an observed separation gap of 1.893.

Figure 5: Mahalanobis distance between latent class centroids for cell-4; JEPA latent exhibits negligible separation relative to theory and reference models.

Implications and Delimitations

The findings have several important implications. First, predictive objectives—despite their robustness to spurious texture and reward-irrelevant distractors—impose an inductive bias that structurally risks omitting information vital for control whenever reward signals flow through unpredictable exogenous factors. This has particular relevance for scenarios where the environment reveals critical, but uncontrollable and unpredictable, aspects (e.g., randomized environment states, sensor drift, or latent task parameters).

Second, the results sharply delineate the differences between reward-free and reward-grounded objectives for representation learning, substantiating, with quantifiable evidence, the theoretical argument advanced by bisimulation literature for reward-informed models [zhang2020dbc].

Third, the label-efficiency of reward grounding suggests that relatively modest additional supervision can suffice to remedy these failures in practice, opening up prospects for hybrid or semi-supervised schemes in environments where reward signals are intermittently available or costly to obtain.

It is important to stress the scope: all evidence is derived from controlled synthetic environments designed to decouple confounds and explicitly reveal the failure mode. The occurrence and practical impact of this risk in large-scale, real-world domains or in pretrained JEPA models remain open questions; empirical transfer is left for future work.

Theoretical and Practical Perspectives

This study reinforces the distinction between statistical predictability and control-theoretic relevance, showing they are often, but not always, aligned. When decoupled by design, as in the controlled synthetic environments of this paper, optimizing only for predictability leads to a measurable information deficit in representations intended to support downstream control.

The results relate intimately to ongoing discourse in representation learning for RL: the limitations of pixel prediction versus reward/transition alignment (DeepMDP), the need for task-aware regularization (bisimulation, action-invariance), and the question of what minimal augmentation to self-supervision—such as sparse reward grounding—is required to guarantee retention of all reward-relevant information.

Looking forward, immediate avenues present themselves: the demonstration of the failure mode in high-capacity, pretrained world models (e.g., V-JEPA 2-AC, DINO-WM), the development of practical schemes for efficient reward-label utilization and adapter modules, and a more thorough theoretical account of why and where in the learning dynamics the omission arises.

Conclusion

This paper provides a rigorous and controlled mechanistic study exposing a structural failure mode of reward-free predictive objectives: the systematic omission of exogenous features that are unpredictable but control-relevant. The omission is not an artifact of architecture, capacity, or implementation, but rather a direct consequence of optimizing for predictability in the absence of reward signal. A reward-grounded variant is shown to recover the lost feature with high selectivity and label-efficiency. The results advance understanding of when and why self-supervised objectives may become misaligned with downstream control requirements, and underscore the importance of reward-informed learning signals for reliable autonomous representation learning.