JEPA-Anything: World Modeling Framework

• JEPA-Anything is a domain-agnostic framework that uses Joint-Embedding Predictive Architectures (JEPA) and orthogonal predictive factorization (OPF) to decompose target latents into predictive, orthogonal factors.
• Its dedicated pathways allocate capacity across complementary components, enhancing numerical stability, intervention generalization, and long-horizon predictions in domains like vision, biology, and control.
• Results have shown that JEPA-Anything improves representation learning and prediction accuracy in multiple domains but does not establish the factors to be causal mechanisms
• GPT-based encoder, projectors for factors and generator UPDATE.

JEPA-Anything is a domain-agnostic framework for latent world modeling based on Joint-Embedding Predictive Architectures (JEPAs) and orthogonal predictive factorization (OPF). It retains the JEPA principle of encoding an observed context and predicting a target representation rather than reconstructing raw observations, while decomposing target latents into complementary, dedicatedly predicted factors. The framework is intended to support representation learning, intervention prediction, out-of-distribution generalization, planning, and long-horizon dynamics across heterogeneous domains, including vision, biology, clinical trajectories, control, molecular dynamics, physical fields, and weather (Cui et al., 17 Sep 2026). “JEPA-Anything” is therefore a general predictive interface rather than a single universal encoder: tokenization, structural descriptors, context-target sampling, encoder architecture, and downstream readouts remain domain-specific, while the factorized predictive core is shared.

1. Conceptual foundations and scope

A conventional JEPA uses an online encoder to process a context, a target encoder—typically updated by exponential moving average (EMA)—to produce a target latent, and a predictor to infer that target from the context. For a context representation $z_c$, target representation $z_t$, and target descriptor $s_t$, the basic structure is

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$

The target branch is stop-gradient, and its parameters follow the online encoder through

$\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$

The target may represent a masked region, future state, intervention outcome, missing modality, or another structured view of the same underlying system. Latent prediction avoids requiring exact reproduction of pixels, waveforms, gene counts, detector measurements, or other potentially noisy observations.

The framework distinguishes between domain-specific and shared components. Domain adapters convert raw observations into content tokens and structural descriptors. A view sampler selects context and target subsets. Domain-specific encoders process those inputs, while OPF supplies factorized target prediction, orthogonality constraints, factor-activity regularization, and latent-state synthesis. Downstream decoders, planners, classifiers, and scientific readouts remain domain dependent.

This scope distinguishes JEPA-Anything from claims of a completely modality-independent network. The framework does not assert that all domains share the same tokenizer, target semantics, physical dynamics, or causal structure. Rather, it proposes a common predictive principle capable of operating over different domain representations.

2. Orthogonal predictive factorization

OPF replaces a monolithic target predictor with $K$ dedicated predictive subspaces. Let the target latent have dimension $d$, let each factor have width $r$, and let $Kr=d$. Learned projectors

$P_k\in\mathbb R^{d\times r}$

map the stop-gradient target latent $z_t$0 into factor coordinates:

$z_t$1

The concatenated factor target is

$z_t$2

where $z_t$3. Each factor has a dedicated predictor,

$z_t$4

The predicted factor vector is recombined into a complete latent state through

$z_t$5

When $z_t$6 is exactly orthogonal,

$z_t$7

For action- or intervention-conditioned dynamics, the predictor receives an exogenous input $z_t$8 and target descriptor $z_t$9:

$s_t$0

Repeated application yields a latent rollout.

OPF is intended to address several problems associated with monolithic latent prediction. Multiple predictive structures may compete for the same latent capacity; highly predictable components may dominate optimization; and a single predictor may represent observed combinations without exposing reusable factors under interventions. Dedicated pathways provide a mechanism for allocating capacity across complementary predictive components.

3. Objective, orthogonality, and stability

The factor-prediction loss is

$s_t$1

OPF constrains the factor projectors through

$s_t$2

The first term enforces orthonormality within each factor, while the second penalizes overlap between factors. In the zero-penalty case,

$s_t$3

and $s_t$4 becomes an orthogonal matrix.

Orthogonality does not ensure that all factors are used. OPF therefore includes factor-activity and encoder anti-collapse losses. For factor-coordinate standard deviations $s_t$5 and encoder-coordinate standard deviations $s_t$6,

$s_t$7

and

$s_t$8

The OPF contribution and total domain objective are

$s_t$9

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$0

Representative coefficients are $$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$1 for most dynamics tasks and $$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$2 for locomotion.

Orthogonal synthesis gives a conditioning advantage. If the factor prediction error is $$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$3 and $$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$4 is exactly orthogonal,

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$5

Thus the recombination map does not amplify factor errors. If

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$6

then

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$7

This can improve numerical stability during rollout, although it does not guarantee accurate transition dynamics, robustness to distribution shift, or long-horizon correctness.

4. Training procedure and operational modes

A JEPA-Anything instantiation follows a six-stage procedure:

1. Raw observations are converted into content tokens and structural descriptors.
2. A view sampler constructs context and target inputs.
3. The online encoder processes the context, while the EMA target encoder processes the target.
4. Target latents are projected into factor subspaces and each factor is predicted.
5. The online encoder, projectors, and factor predictors are optimized using the domain loss plus OPF.
6. The target encoder is updated through EMA.

The framework supports two principal use modes.

Representation mode retains the online encoder and uses a task-specific readout, pooling operation, probe, or decoder:

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$8

The target encoder, projectors, and predictors may be discarded after pretraining.

Operational world-model mode retains the projectors and predictors to synthesize future latent states. These predictions can support intervention analysis, planning, control, or autoregressive rollout. Factor coordinates can also be inspected through

$$z_c=f_\theta(\mathrm{context}),\qquad z_t=f_{\bar{\theta}}(\mathrm{target}),\qquad \widehat z_t=q(z_c,s_t).$$9

The factors are predictive coordinates and are not guaranteed to correspond to unique semantic, mechanistic, or causal variables.

5. Cross-domain applications and evidence

JEPA-Anything is evaluated across seven broad domains: vision, biology, clinical trajectories, control, molecular dynamics, physical fields, and weather.

In vision, controlled visual-binding experiments use DINOv3 ViT-S and SigLIP2 Base backbones. With learned-grid supervision, OPF modestly improves injective held-out-cell accuracy, reduces collapse rate, and improves grid recovery relative to matched monolithic JEPA.

In single-cell biology, scGPT is used with approximately 800,000 human kidney cells for pretraining, PBMC-10K for clustering, and Adamson and Norman Perturb-seq datasets for intervention-response prediction. The reported JEPA model achieves PBMC fine-tuned AvgBIO of $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$0, zero-shot AvgBIO of $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$1, Norman Pearson correlation of $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$2, and Adamson Pearson correlation of $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$3, compared with $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$4, $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$5, $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$6, and $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$7 for Cell-JEPA, respectively. These results concern representation learning and intervention prediction; they do not establish that factors correspond to specific genes or causal mechanisms.

For clinical trajectories, longitudinal multimodal patient histories are encoded with a GPT-2 Small backbone and four factors. The synthesized future state is passed to a shared risk decoder covering more than 1,000 future clinical events. The factorized model achieves higher mean PRAUC than the matched monolithic JEPA, although the supplied results do not provide a numerical PRAUC table.

In Interventional Pong, JEPA reduces single-intervention one-step prediction error from $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$8 to $\bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta.$9, a reduction of $K$0. The improvement is smaller for held-out combinations of interventions and six-step free rollouts, with reductions of $K$1 and $K$2, respectively.

Across ten matched physical-field and weather dynamics tasks, the factorized model improves reported metrics on all tasks. Improvements are particularly pronounced for PDE rollouts. In Burgers dynamics, an additional APEBench evaluation reports a reduction in held-out late-state MSE from $K$3 to $K$4 and six-step MSE from $K$5 to $K$6.

In continuous control, JEPA achieves higher mean return on Walker2d and HalfCheetah, while standard JEPA performs better on Hopper. This makes the control evidence positive but non-uniform. Factor ablations in locomotion show that masking any factor increases 20-step rollout MSE in all three environments, indicating functional use of the factor channels.

In molecular dynamics, an $K$7-equivariant TrajCast-style forecaster is applied to liquid water, crystalline $K$8-quartz, gas-phase paracetamol, and benzene. JEPA obtains the lowest one-step displacement MAE and 100-step RMSD among the compared methods in all four systems. The factor projectors preserve equivariance by acting as $K$9 on the $d$0 sector.

6. Scientific analysis, limitations, and significance

JEPA-Anything is also used as an analysis interface. In biology, factor-guided analysis nominates a combined intervention involving IL-18 and NT5E/CD73 blockade. The paper reports supporting experiments in Huh7–PBMC co-cultures, three patient-derived organoids, three patient-derived tumor fragments, and immunocompetent mice. The results provide experimental support for the nominated intervention, but they do not establish that the latent factors are causal mechanisms.

In a separate orbital analysis, latent modes recover a Keplerian frequency–semimajor-axis relationship with fitted log–log slope $d$1 and $d$2. This is a diagnostic demonstration from one analyzed run rather than evidence that OPF generally discovers physical laws.

The framework has several limitations. Domain adapters remain essential: each application requires an appropriate tokenizer, context-target relation, structural descriptor, encoder, and downstream readout. Orthogonality provides geometric separation but not semantic disentanglement, causal identification, or unique factor meanings. Predictive targets must be statistically related to the context; irreducible noise and unobserved variables remain limiting factors. Performance is not uniformly superior, as shown by the Hopper result. Long-horizon stability remains vulnerable to model bias, chaotic sensitivity, distribution shift, and accumulated transition errors.

OPF also introduces additional hyperparameters, including the number of factors, factor width, projector matrices, orthogonality and activity coefficients, and activity thresholds. The reported matched control experiments indicate parameter increases below $d$3 and prediction-head FLOP increases below $d$4 for tested factor counts, but the computational and optimization costs can increase in larger systems.

The principal contribution of JEPA-Anything is therefore a common factorized predictive interface rather than a universal model with domain-independent semantics. Its strongest supported claim is that orthogonal, active, dedicatedly predicted latent factors can improve predictive capacity allocation, intervention generalization, numerical conditioning, and long-horizon dynamics across heterogeneous domains. The stronger claim that the factors reveal universal causal or scientific variables remains unestablished. The framework’s broader significance lies in treating the organization of predictive information—not only the prediction loss itself—as a central design problem for general-purpose world models (Cui et al., 17 Sep 2026).