EPM-JEPA: Operator-Side Experience Modulation in JEPA-Family World Models

Abstract: JEPA-family world models use a static predictor whose weights do not adapt when test-time dynamics diverge from training. We compare two mechanisms for incorporating accumulated experience into a JEPA predictor under distribution shift: operand-side injection, where a compressed experience representation is added as a residual to the predictor's hidden state (EI-JEPA), and operator-side modulation, where the same representation generates low-rank weight deltas via LoRA applied to the predictor's weights (EPM-JEPA). On a pre-registered comparison (Moving MNIST, gravity shift), EPM-JEPA (D_shift^{n=50} = 0.7848 +/- 0.0078, three seeds) differs from EI-JEPA (0.8238) by delta = 4.74% - Outcome C: a null result - by our stated criterion, a valid outcome. As a secondary, non-pre-registered observation, EPM-JEPA improves 1.90% over a no-memory baseline (0.8000), consistently across seeds, while EI-JEPA underperforms the baseline, indicating the benefit is specific to weight-level modulation. Our primary contribution is a mechanism analysis: the D_shift^{n=50} trajectory reflects three independent dynamical processes - buffer cycling, EMA target drift, and an intrinsic LoRA settling transient of +0.021 - rather than convergence to equilibrium. These findings motivate PEM-JEPA, a physics-grounded successor addressing this dynamical-peak limitation.

• The paper introduces operator-side experience modulation through online LoRA-style low-rank updates that adapt predictor weights in JEPA models.
• It details a three-track experimental design comparing static, operand-side, and operator-side approaches, demonstrating a consistent error reduction under test-time shifts.
• The analysis reveals that dynamic buffer cycling, EMA target drift, and low-rank transients are critical mechanisms affecting model performance in nonstationary environments.

EPM-JEPA: Operator-Side Experience Modulation in JEPA-Family World Models

Motivation and Problem Formulation

Static predictors in the JEPA (Joint-Embedding Predictive Architecture) family fail to adapt when test-time dynamics diverge from training, exposing a core limitation for world modeling under distribution shift. Traditional memory mechanisms in sequence models (e.g., Transformer-XL, Neural Turing Machines) perform operand-side augmentation, leaving the predictor's weights unchanged and thus maintaining a fixed inductive bias regardless of input context. In contrast, EPM-JEPA introduces operator-side experience modulation via online LoRA-style low-rank weight updates, enabling direct adaptation of the predictor's weights from compressed, event-driven experience buffers.

Model Architecture

EPM-JEPA is evaluated using a controlled three-track experimental design:

• Track A (Vanilla JEPA): Baseline model with no memory. Predictor receives current latent state as input, operates with static weights.
• Track B (EI-JEPA): Employs operand-side memory. Aggregated experience is injected as a residual into the predictor's hidden state.
• Track C (EPM-JEPA): Operator-side approach. Aggregated experience controls low-rank (LoRA) weight deltas for the predictor, modifying computation directly per input.

All tracks use a four-layer convolutional encoder, an EMA target encoder (BYOL-style), and 2-layer MLP predictors. The experience subsystem, enabled for Tracks B/C, consists of a boundary event detector (surprisal-gated), FIFO experience buffer, a 2-layer transformer experience encoder, and attention-based aggregation. The aggregated experience representation drives either residual injection (Track B) or LoRA modulation (Track C).

Figure 1: Multi-seed at peak step ($\approx$10,000) for all tracks; EPM-JEPA (Track C) yields lower error than Vanilla JEPA, while EI-JEPA (Track B) is inferior to both.

Training Regimen and Metrics

All models are trained end-to-end on a gravity-shift variant of Moving MNIST: training occurs in a zero-gravity regime, and test-time distributional shift introduces a vertical gravitational acceleration. The main evaluation is mean normalized prediction error on the shifted test set, averaged across 3 horizons ($k \in \{5, 10, 20\}$).

Training loss includes a predictive MSE to stop-gradient EMA targets and VICReg-style variance regularization:

• $$\mathcal{L} = \mathbb{E}_t \sum_k \| \bar{E}_\phi(x_{t+k}) - P_\theta(z_t) \|_2^2 + \lambda \mathbb{E}_d \max(0, \gamma - \sigma_d(P_\theta(z_t)))$$

Hyperparameters are tuned using sequential search and ablation protocols, and held fixed for all main comparisons.

Empirical Results

Summary of Main Outcomes

Primary findings for EPM-JEPA (Track C):

• Pre-registered test (EPM-JEPA vs EI-JEPA): Relative error reduction $\delta = 4.74\%$—a null outcome under the predefined threshold ($|\delta|<5\%$).
• Secondary comparison (EPM-JEPA vs Vanilla JEPA): Consistent 1.90% mean reduction across all Track C seeds relative to the best Track A result, with non-overlapping bootstrap CIs.
• EI-JEPA underperforms Vanilla JEPA: Input-side experience injection is ineffective, showing higher error than the baseline.

Figure 2: Full training ranges for all tracks; EPM-JEPA converges to a lower prediction error earlier in training.

When examining the multi-seed distributions, all EPM-JEPA seeds outperform all baselines at peak, establishing reproducibility of the advantage despite the null result versus input-injection.

Mechanism and Dynamics Analysis

Dynamical analysis of EPM-JEPA reveals the real system behavior is governed by three independent mechanisms rather than static convergence:

1. Buffer Cycling: The experience buffer reaches capacity and cycles, shaping the timing of the performance peak as LoRA adapts to a dynamic data prior.
2. EMA Target Drift: The slow-moving EMA target encoder causes post-peak performance divergence by continually shifting the prediction target.
3. LoRA Settling Transient: Freezing both buffer and EMA at convergence isolates an intrinsic +0.021 increase in prediction error—an artifact of low-rank adaptation dynamics, invariant to timing of intervention.

Freeze-based ablations confirm that post-peak divergence is attributable to EMA drift, while the LoRA settling transient is architectural, not schedular.

Figure 3: Mechanism ablation—dual-freeze interventions arrest divergence; intrinsic LoRA transient persists post-freeze.

Representation Quality

All Track C runs show a characteristic contraction in latent diversity (embedding stddev) at the performance peak, with values transiently dropping below the 0.5 safety threshold. However, post-peak, these recover above the risk zone, and SVD analyses find no evidence of permanent representation collapse.

Figure 4: Evolution of per-dimension latent stddev; freeze conditions stabilize diversity above threshold, non-freeze Track C dips beneath 0.5 around peak.

Notably, there is a persistent, irreducible trade-off: lowering regularization $\lambda$ boosts accuracy but contracts the latent manifold; increasing $\lambda$ sacrifices predictive gain for diversity, with no value yielding both optimal accuracy and high diversity at peak.

Theoretical and Practical Implications

EPM-JEPA demonstrates that operator-side experience modulation—applying LoRA-like weight deltas indexed by recent event-driven experience distribution—enables modest but robust adaptation under distributional shift without downstream fine-tuning or architectural expansion. The comparative ineffectiveness of operand-side memory in this setting directly informs the design of world models for on-the-fly adaptation, indicating that computation-level flexibility is essential under nonstationary dynamics.

The mechanism analysis isolates architectural bottlenecks: the LoRA-layer’s adaptation cannot yield a stable equilibrium due to dynamic buffer cycling (non-i.i.d. modulator input) and EMA drift. This motivation leads toward physics-grounded successor models (PEM-JEPA), which promise to constrain the latent state evolution and eliminate post-peak drift by imposing structured state priors.

Future Directions

Future research directions include:

• Explicit latent structure: Introducing kinematics-informed latent spaces to stabilize adaptation.
• Scaling to more complex domains: Extending operator-side modulation to high-dimensional or 3D environments, assessing if buffer cycling, EMA drift, and LoRA transients generalize.
• Alternative modulation schemes: Hypernetwork or FiLM-based modulations to test whether irreducible transients are unique to low-rank adaptation or universal to experience-conditioned operator modulation.
• Robustness to shift types: Evaluating boundary detection and adaptation efficacy under non-physical changes (e.g., object identity, lighting shifts).

Conclusion

EPM-JEPA provides a clear demonstration of operator-side experience modulation in world models. Modulating predictor weights online via LoRA deltas from an event-driven buffer yields a modest, consistent gain under test-time distribution shift—whereas operand-side input injection is at best ineffective and at worst, detrimental. The performance transient in EPM-JEPA is the equilibrium of multiple dynamical processes rather than optimizer convergence, and cannot be removed by scheduling or regularization alone. The results motivate architectural innovation—embedding prior structure or physics-grounded constraints in future model classes to achieve both stability and adaptivity in predictive world modeling.