JEPA-Anything: Learning Predictive Models across Different Worlds

Abstract: World modeling enables intelligence to anticipate consequences, guide interventions, and learn from interaction. Yet predictive models remain domain-specific: can a common learning principle support world modeling across radically different systems? We introduce JEPA-Anything, a domain-agnostic framework based on orthogonal predictive factorization (OPF). Extending joint-embedding predictive architectures, OPF decomposes latent targets into complementary factors, learns them through dedicated pathways, and recombines them within a shared predictive design. We evaluate JEPA-Anything across seven domains: vision, biology, clinical trajectories, control, molecular dynamics, physical fields, and weather. Experiments span representation learning, intervention prediction, out-of-distribution generalization, and long-horizon dynamics, including 10 matched dynamics tasks, forecasting of over 1,000 clinical events, and 100-step molecular rollouts across four systems. Against matched JEPA baselines, JEPA-Anything improves reported metrics on all 10 dynamics tasks and reduces single-intervention prediction error on Interventional Pong by 34.8%. It achieves the lowest one-step and 100-step molecular errors among compared methods in all four systems. Beyond prediction, a factor-nominated biological intervention receives experimental support in cell co-cultures, patient-derived organoids, tumor fragments, and mice; latent orbital modes recover the Keplerian scaling exponent with a fitted slope of -1.4991. These results support a common factorized predictive principle across heterogeneous worlds, connecting world modeling with intervention and experimentally grounded scientific discovery. Code: https://github.com/Gen-Verse/JEPA-Anything

• The paper introduces Orthogonal Predictive Factorization (OPF), a method that improves predictive modeling across diverse domains by organizing latent target representations into active, non-overlapping subspaces.
• OPF achieves significant improvements in various domains, including a 34.8% reduction in single-intervention prediction error on CITRIS Interventional Pong and consistently lower molecular prediction errors across four systems.
• The method supports downstream tasks such as scientific analysis and control, demonstrating robustness and generalizability across heterogeneous domains with stable synthesis and functional factor usage.

Problem setting and central claim

“JEPA-Anything: Learning Predictive Models across Different Worlds” (2609.20800) addresses a structural limitation of conventional JEPA formulations: although joint-embedding prediction can learn useful latent states without pixel- or observation-space reconstruction, the target state is typically represented through a monolithic embedding and predicted by a single pathway. The paper argues that this design creates a capacity-allocation problem when the target contains heterogeneous predictive structure, such as multiple entities, spatial scales, temporal modes, intervention effects, or physical variables with unequal predictability.

The proposed framework, JEPA-Anything, retains the JEPA context–target interface while introducing orthogonal predictive factorization (OPF). A domain adapter converts images, sequences, graphs, fields, molecular states, clinical records, or cellular profiles into tokens and descriptors. A view sampler then specifies which observations constitute context and which constitute targets. After this domain-specific preprocessing, the predictive core is shared: an online encoder represents the context, an EMA target encoder represents target states, and multiple predictor branches estimate complementary components of the target representation.

The paper’s principal claim is not that all domains share an identical representation, encoder, or transition function. Rather, it claims that they can share a factorized predictive learning principle and latent-state interface. Domain-specific observation geometry, tokenization, view sampling, encoder architecture, and downstream readout remain configurable. OPF is intended to organize the resulting latent target into active, non-overlapping subspaces that can be used for terminal readout, recursive dynamics, planning, or scientific analysis.

The evaluation spans visual binding, single-cell biology, clinical forecasting, intervention-conditioned prediction, continuous control, molecular dynamics, physical fields, and weather. The paper reports improvements over matched monolithic JEPA baselines on all ten tasks in its main dynamics benchmark, a 34.8% reduction in single-intervention prediction error on CITRIS Interventional Pong, the lowest one-step and 100-step molecular errors across four systems, and a latent orbital scaling exponent of $-1.4991$, close to the Keplerian value of $-3/2$.

Architecture and orthogonal predictive factorization

Let the online encoder produce a context representation $z_c$ and the EMA target encoder produce a target representation $z_t \in \mathbb{R}^d$. OPF introduces $K$ learned projectors $P_k \in \mathbb{R}^{d \times r}$, with $Kr=d$, and analyzes the stop-gradient target embedding into factor coordinates:

$z_t^{(k)} = P_k^\top z_t.$

Each factor has a dedicated predictor $q_k$ that receives the shared context representation and, when necessary, a target descriptor such as a spatial coordinate, time index, entity identity, action, or intervention label. The predicted factor coordinates are concatenated and synthesized into a complete latent state through the Moore–Penrose pseudoinverse of the combined analysis map:

$$\widehat z_t = (P^\top)^\dagger \left[ \widehat z_t^{(1)};\ldots;\widehat z_t^{(K)} \right].$$

Under exact orthogonality, synthesis reduces to ordinary orthogonal reconstruction. The method therefore retains both factor-specific predictions and a complete latent state suitable for decoding, planning, and autoregressive rollout. This distinction is important: the factors are not independent latent variables in the generative-model sense, nor are they required to correspond to predefined semantic or causal entities. They are predictive subspaces constrained to be complementary.

The training objective adds three regularization terms to the base loss of each domain. The orthogonality loss enforces orthonormality within each factor and orthogonality between factors. A factor-activity loss imposes a lower bound on the empirical standard deviation of projected target coordinates, preventing branches from becoming inactive. An encoder-variance term directly discourages collapse in the online representation. Factor prediction itself uses coordinate-wise squared error between predicted and target factor states.

This formulation differs from redundancy-reduction objectives such as Barlow Twins (Alekseev, 2021) and VICReg (Bardes et al., 2021) in where structure is imposed. OPF applies geometric constraints to learned predictive subspaces and explicitly preserves an analysis–synthesis map. It is consequently designed not only to make representations non-redundant, but also to make the concatenated prediction outputs usable as a numerically stable latent state.

The paper provides an exact geometric argument. If the projectors form an orthogonal basis, then the combined matrix $-3/2$0 is orthogonal, its condition number is one, and errors in factor coordinates are transferred to the synthesized state without amplification. Without cross-factor orthogonality, duplicated directions can make the analysis matrix rank deficient, while nearly duplicated directions can produce arbitrarily large condition numbers. The trained CITRIS audit is consistent with this prediction: orthogonal factorization produces cross-factor overlap of approximately $-3/2$1, minimum singular value $-3/2$2, condition number $-3/2$3, and exact-coordinate synthesis NMSE of approximately $-3/2$4. The unconstrained multi-head alternative has overlap $-3/2$5, minimum singular value $-3/2$6, condition number $-3/2$7, and synthesis NMSE $-3/2$8.

These measurements establish the geometry of the factor interface, but they do not by themselves establish that OPF improves prediction. The remainder of the experiments tests whether this constrained coordinate system yields better representations and dynamics.

Common interface across heterogeneous domains

The framework’s unifying abstraction is a context–target relation between states of the same underlying system. For images, context may be visible patches and targets masked regions. For clinical data, context is patient history and the target is a future patient state. For control, context includes the current state and action, while the target is a future state. For molecular dynamics, the model predicts future atomic configurations. For scientific analysis, the learned factor coordinates are retained for intervention nomination or spectral inspection.

The scenario atlas emphasizes this common structure across three evaluation groups: terminal readout, latent world dynamics, and scientific analysis.

Figure 1: The scenario atlas organizes evaluation around terminal readout, recursive latent dynamics, and factor-level scientific analysis.

This abstraction is deliberately weaker than a claim of architectural uniformity. The paper permits ViTs, Transformers, GNNs, MLPs, and domain-specific adapters. In some tasks, the OPF loss is added to an existing objective rather than replacing it. Consequently, the empirical contribution is best interpreted as a transferable regularization and state-organization mechanism within otherwise specialized pipelines, rather than as a single fully domain-independent model.

Terminal readout: visual, cellular, and clinical evaluation

Controlled visual binding

The visual experiment examines whether factorized predictive pretraining improves a frozen representation’s ability to preserve both the location and operation associated with controlled image changes. DINOv3 and SigLIP2 backbones are compared under frozen-checkpoint, monolithic JEPA, and OPF-JEPA conditions. The downstream readout is held fixed between standard JEPA and JEPA, and evaluation includes image-disjoint splits and leave-one-cell-out generalization.

For DINOv3, JEPA improves injective held-out-cell accuracy from $-3/2$9 to $z_c$0, reduces collapse from $z_c$1 to $z_c$2, and increases grid recovery from $z_c$3 to $z_c$4. For SigLIP2, the corresponding changes are more modest: injective accuracy rises from $z_c$5 to $z_c$6, collapse decreases from $z_c$7 to $z_c$8, and grid recovery rises from $z_c$9 to $0.688.

The gains are consistent but small. They indicate that OPF can improve compositional readout under a matched downstream protocol, although the experiment does not show that factor coordinates themselves acquire identifiable semantic meanings. The visual result therefore supports improved representation quality, not semantic disentanglement.

Single-cell representation and perturbation prediction

The single-cell experiments use an scGPT backbone and compare scGPT, Cell-JEPA, and the proposed factorized JEPA. Models are pretrained on approximately 800,000 human kidney cells and evaluated on PBMC-10K clustering and Adamson and Norman perturbation-response prediction.

JEPA achieves an AvgBIO score of $z_t \in \mathbb{R}^d$0 under PBMC fine-tuning and $z_t \in \mathbb{R}^d$1 in the zero-shot setting, compared with $z_t \in \mathbb{R}^d$2 and $z_t \in \mathbb{R}^d$3 for Cell-JEPA. On perturbation prediction, Pearson correlation reaches $z_t \in \mathbb{R}^d$4 on Norman and $z_t \in \mathbb{R}^d$5 on Adamson, compared with $z_t \in \mathbb{R}^d$6 and $z_t \in \mathbb{R}^d$7 for Cell-JEPA.

Evaluation	scGPT	Cell-JEPA	JEPA
PBMC fine-tuned AvgBIO	0.7531	0.7830	0.8301
PBMC zero-shot AvgBIO	0.5288	0.7194	0.7752
Norman Pearson	0.631	0.787	0.814
Adamson Pearson	0.905	0.937	0.942

The zero-shot improvement is particularly relevant because it suggests that factorized latent prediction can improve transfer across cellular datasets without relying exclusively on task-specific fine-tuning. However, the perturbation setup uses an expression decoder and additional task-specific components, so the observed improvement reflects the complete Cell-JEPA/JEPA pipeline rather than OPF in isolation.

Longitudinal clinical forecasting

The clinical experiment predicts a synthesized future patient state from longitudinal multimodal records and uses a decoder to estimate risks for more than 1,000 future clinical events. Mean PRAUC over the full event vocabulary is the primary metric, with patient-level cohort splitting and five training seeds.

The paper reports that JEPA exceeds the matched monolithic JEPA across the event vocabulary, but the supplied text does not provide the numerical PRAUC values. The result nevertheless tests an important property of factorized prediction: a single future latent state can support a broad event-risk readout rather than a narrow task-specific forecast. The implication is that OPF may improve the information content of a future state when the downstream target comprises many heterogeneous and imbalanced event types.

Figure 2: Mean PRAUC rankings for prediction of more than 1,000 future clinical events.

Intervention-conditioned dynamics and compositional prediction

CITRIS Interventional Pong provides the most direct test of whether factorized states support intervention composition. The model receives a pre-intervention observation and an intervention label, then predicts the next state. Evaluation includes both observed single interventions and a combined intervention withheld as a complete configuration during training.

Relative to standard JEPA, JEPA reduces single-intervention one-step MSE from $z_t \in \mathbb{R}^d$8 to $z_t \in \mathbb{R}^d$9, a 34.83% reduction. For the unseen combined intervention, MSE decreases from $K$0 to $K$1, a 12.90% reduction. Six-step free-rollout MSE decreases from $K$2 to $K$3, an 8.58% reduction.

Figure 3: OPF improves single-intervention, combined-intervention, and six-step free-rollout prediction on Interventional Pong.

The decreasing gain from one-step prediction to free rollout is informative. OPF substantially improves immediate intervention-conditioned prediction, but the advantage attenuates under recursive application. This indicates that factorization improves local state prediction without eliminating long-horizon error accumulation. The combined-intervention result is consistent with compositional reuse, although the withheld configuration remains within a controlled synthetic environment and does not establish causal identification in general.

Out-of-distribution dynamics and rollout stability

The matched dynamics benchmark covers ten tasks spanning CausalWorld, DeepMind Control, PDEBench, WeatherBench2, and related physical-dynamics settings. Standard JEPA and JEPA use the same encoder, transition architecture, training budget, evaluation split, and five seeds. The paper reports improved metrics on all ten tasks, including nine prediction tasks and CausalWorld control.

Figure 4: Relative MSE reductions across the matched ten-task dynamics benchmark and closed-loop CausalWorld returns.

The rollout diagnostics show that OPF reduces both one-step error and error growth. In PDEBench Burgers, standard JEPA increases from MSE $K$4 at step one to $K$5 at step six, whereas JEPA increases from $K$6 to $K$7. In shallow-water prediction, the corresponding sixth-step errors are $K$8 and $K$9. For CausalWorld, JEPA reduces first-step MSE from $P_k \in \mathbb{R}^{d \times r}$0 to $P_k \in \mathbb{R}^{d \times r}$1 and sixth-step MSE from $P_k \in \mathbb{R}^{d \times r}$2 to $0.01079.

The additional APEBench evaluation reinforces the result. On Burgers, held-out late-state MSE decreases from $P_k \in \mathbb{R}^{d \times r}$3 to $P_k \in \mathbb{R}^{d \times r}$4, approximately a 49.5% reduction. Six-step rollout MSE decreases from $P_k \in \mathbb{R}^{d \times r}$5 to $P_k \in \mathbb{R}^{d \times r}$6, approximately a 44.7% reduction. On Kuramoto–Sivashinsky dynamics, held-out late-state MSE decreases by approximately 13.2%, from $P_k \in \mathbb{R}^{d \times r}$7 to $P_k \in \mathbb{R}^{d \times r}$8.

The capacity-matched 50-step Burgers experiment shows smaller but persistent improvements. On in-distribution trajectories, H20/H50 MSE changes from $P_k \in \mathbb{R}^{d \times r}$9 to $Kr=d$0. On high-frequency OOD trajectories, it changes from $Kr=d$1 to $Kr=d$$2. The relative advantage narrows with horizon, from 5.84<h2 class='paper-heading' id='planning-and-functional-factor-usage'>Planning and functional factor usage</h2> <p>In continuous control, a <a href="https://www.emergentmind.com/topics/cross-entropy-masking-cem-loss" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">CEM</a> planner rolls candidate action sequences through the learned latent model and executes the first action of the highest-scoring sequence. Parameter and compute budgets are closely matched: the reported parameter differences remain below 0.3 <p>JEPA improves mean CEM return on Walker2d and HalfCheetah but loses to standard JEPA on Hopper. This environment dependence is an important qualification. The factorized representation improves planning only under some dynamics and reward structures; it does not establish a universal control advantage.</p> <p><img src="https://images.emergentmind.com/paper-images/2609-20800/control_planning_v2.png" alt="Figure 5" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 5: Capacity-matched differences in CEM return, with positive values favoring JEPA.</p></p> <p>The factor-intervention analysis provides a more direct test of whether the learned branches are functionally used. Replacing any one factor with its training-set mean increases 20-step rollout MSE in Hopper, Walker2d, and HalfCheetah. Factor-wise standard deviations remain nontrivial, ranging from$$Kr=d$3 to $Kr=d$4 on Hopper, $Kr=d$5 to $Kr=d$6 on Walker2d, and $Kr=d$7 to $Kr=d$$8 on HalfCheetah. In HalfCheetah, masking F1 or F3 reduces return in all five seeds, while masking F2 or F4 reduces return in four of five seeds.</p> <p><img src="https://images.emergentmind.com/paper-images/2609-20800/factor_intervention_v2.png" alt="Figure 6" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 6: Replacing individual factors with their training-set means increases rollout error and generally reduces planning return.</p></p> <p>These results establish a three-part evidence chain: factors are active, information about motion changes is linearly readable from individual factors, and factor removal affects recursive prediction and control. They do not show that each factor corresponds to a unique physical variable. The paper explicitly notes that a factor may encode several related quantities.</p> <h2 class='paper-heading' id='force-free-molecular-forecasting'>Force-free molecular forecasting</h2> <p>The molecular experiment evaluates autoregressive prediction of atomic positions and velocities in water, quartz, paracetamol, and benzene. OPF is applied to the channel dimension of the$$Kr=d$9 equivariant sector, with projections shared across the three irrep components to preserve $z_t^{(k)} = P_k^\top z_t.$0 equivariance.

JEPA attains the lowest one-step displacement MAE and 100-step final-position RMSD in all four systems.

System	Metric	Scratch	TrajCast-JEPA	JEPA
Water	One-step MAE	0.00387	0.00452	0.00376
Water	100-step RMSD	3.331	2.536	2.459
Quartz	One-step MAE	0.01080	0.01043	0.01011
Quartz	100-step RMSD	2.089	1.912	1.877
Paracetamol	One-step MAE	0.00901	0.00777	0.00705
Paracetamol	100-step RMSD	3.155	1.868	1.776
Benzene	One-step MAE	$z_t^{(k)} = P_k^\top z_t.$1	$z_t^{(k)} = P_k^\top z_t.$2	$z_t^{(k)} = P_k^\top z_t.$3
Benzene	100-step RMSD	0.0958	0.0701	0.0645

The consistent improvement across chemically distinct systems is one of the strongest empirical sections of the paper. It is also methodologically constrained: all models use the same TrajCast-style equivariant backbone, supervised adaptation, and rollout protocol, so the comparison isolates the effect of OPF pretraining more cleanly than the cross-domain experiments. Nonetheless, the molecular evaluation remains a finite benchmark of force-free trajectory emulation and does not establish conservation of energy, symplectic structure, or physical validity outside the data distribution.

Factor coordinates as a scientific analysis interface

The third evaluation group asks whether factor coordinates can support scientific hypothesis generation or diagnosis rather than only prediction.

In the biological application, analysis of OPF coordinates nominates combined IL-18 stimulation and NT5E/CD73 blockade as a candidate intervention. The paper reports supporting evidence in Huh7–PBMC co-cultures, three patient-derived hepatocellular carcinoma organoid specimens, three tumor-fragment specimens, and immunocompetent mice. In the organoid and tumor-fragment experiments, the combined treatment produces the strongest reported tumor-cell killing together with increased immune-cell activation.

Figure 7: The combined IL-18 and CD73-blockade condition shows the strongest reported antitumor activity across organoid and tumor-fragment samples.

This is an externally grounded result, but its evidentiary interpretation requires precision. The model nominates a biologically testable intervention, and experiments provide support for its efficacy in the reported systems. The result does not demonstrate that OPF identifies a causal factor or that the treatment’s mechanism is represented in a uniquely identifiable latent coordinate. The paper itself acknowledges that orthogonal factors do not establish causal mechanisms.

The orbital experiment offers a more controlled diagnostic. From simulated position–velocity trajectories without physical labels, spectral analysis of learned factor modes recovers the relationship between orbital frequency and semimajor axis. The fitted slope is $z_t^{(k)} = P_k^\top z_t.$4 with $z_t^{(k)} = P_k^\top z_t.$5, closely matching the Keplerian exponent $z_t^{(k)} = P_k^\top z_t.$6.

Figure 8: Spectral modes extracted from JEPA recover the Keplerian scaling relation with fitted slope $z_t^{(k)} = P_k^\top z_t.$7 and $z_t^{(k)} = P_k^\top z_t.$8.

The orbital result demonstrates that predictive latent coordinates can expose a known dynamical regularity. Its limitation is equally clear: it is based on one analyzed run and simulated trajectories whose governing law is already known. It validates the diagnostic procedure against an established law, but does not establish that the same procedure reliably discovers unknown laws in uncontrolled scientific data.

Limitations and open questions

The central generality claim is architectural rather than representational. JEPA-Anything does not use identical encoders, tokenizers, losses, or data regimes across domains. Each instantiation adds OPF to a domain-specific base objective, and several experiments include task-specific decoders, probes, or adaptation stages. The evidence therefore supports a reusable factorization mechanism, not a single universal world model.

The empirical comparisons are also uneven in strength. Some experiments are carefully capacity-matched against monolithic JEPA, whereas the broader domain comparisons include additional baseline differences and specialized pipelines. The clinical section reports a qualitative ranking in the supplied manuscript but does not state the numerical PRAUC values. The control results are environment-dependent, with Hopper favoring standard JEPA. Long-horizon improvements diminish in the 50-step Burgers evaluation, and recursive prediction remains subject to compounding error.

The semantics of the learned factors remain underdetermined. Orthogonality guarantees geometric non-overlap and, approximately, stable synthesis; it does not guarantee statistical independence, causal separability, object alignment, or unique semantic interpretation. This distinction is consistent with the broader identifiability limitations of unsupervised disentanglement (Hénaff et al., 2019). The paper’s factor interventions establish functional necessity in selected tasks, but not a general correspondence between individual factors and real-world mechanisms.

Finally, the scientific-analysis results require stronger prospective evaluation. The cancer intervention is supported by external experiments, but the manuscript does not establish whether the nomination process was preregistered, how many alternatives were screened, or how often comparable hypotheses fail. The orbital analysis uses a single analyzed run and a known physical law. Specific open questions are whether OPF retains its advantage under larger-scale distribution shift, whether factor identities remain stable across seeds and domains, how uncertainty should be calibrated for intervention selection, and whether factors can be revised when external evidence contradicts the predicted state.

Conclusion

JEPA-Anything proposes OPF as a mechanism for organizing latent predictive capacity into complementary, active, and synthesizable subspaces. Across representation learning, intervention prediction, autoregressive dynamics, planning, molecular forecasting, and scientific analysis, the reported results generally favor factorized JEPA over matched monolithic alternatives. The strongest evidence concerns CITRIS intervention prediction, the ten-task dynamics benchmark, molecular forecasting across four systems, and the near-unit-conditioned factor geometry.

The paper’s more ambitious claims require a narrower interpretation. OPF provides a common predictive interface across heterogeneous domain adapters, but it does not by itself produce domain-independent representations, causal factors, or universally superior control policies. Its principal contribution is a structured latent-state parameterization that improves predictive capacity allocation and makes factor coordinates available for downstream diagnostics. The remaining question is whether these benefits persist when domains are less controlled, interventions are genuinely open-ended, and scientific hypotheses must be evaluated prospectively rather than against known laws or selected experimental systems.