SplitJEPA: Invariant-Variant Latent State Learning without Reconstruction

• SplitJEPA is a method for learning latent worlds decomposed into shared/invariant $r_H$ and variant/private $r_L$ subspaces, preserving the relationships between observations and distinguishing task-specific components. It is particularly useful for space simulations and robotic manipulation.
• It's capable of producing multiple executions of robots sharing identical objects, goals and task identity while varying in initial parameters and changes; it can ensure that tasks using $r_H$ are sensitive only to actual perceptual states and not to factors like lighting and camera setup.
• SplitJEPA involves capturing the dynamics within representation space without the need for observation reconstruction; it outperforms systems reconstruction-free in both effectiveness and efficiency

SplitJEPA is a reconstruction-free Joint-Embedding Predictive Architecture introduced for learning latent worlds whose representations are explicitly partitioned into invariant/shared and variant/private subspaces. Given a latent state $z=[z_H,z_L]$, SplitJEPA seeks an encoded representation $r=[r_H,r_L]$ in which $r_H$ preserves factors shared across matched observations and $r_L$ preserves factors that vary between them. Under stationary Gaussian predictive dynamics, centered and whitened representations, and a full-rank variation condition, the method identifies these subspaces up to independent block-wise isometries, without an observation decoder (Hua et al., 8 Oct 2026). The name is distinct from several related systems informally described as “split” JEPAs, including controllability-factorized, distributed, auxiliary-task, and branch-decoupled architectures.

1. Problem formulation and conceptual foundations

SplitJEPA addresses two separate representation-learning problems. First, the representation must be predictively useful: it should retain information about a dynamical world. Second, it must be structurally organized: factors shared across related observations should occupy an invariant block, while factors that vary between observations should occupy a variant block.

The latent state is written as

$$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$

where $z_H\in\mathbb R^{d_H}$ denotes the invariant/shared block and $z_L\in\mathbb R^{d_L}$ denotes the variant/private block. Observations are generated by an unknown smooth function

$h=f(z_H,z_L).$

An encoder $\phi$ produces

$r=\phi(h)=g(z), \qquad g=\phi\circ f.$

The desired identification result is

$r=[r_H,r_L]$0

where $r=[r_H,r_L]$1 and $r=[r_H,r_L]$2 are orthogonal matrices. Thus, SplitJEPA identifies the invariant and variant subspaces without necessarily identifying individual latent coordinates.

A robotic manipulation example illustrates the intended decomposition. Multiple executions may share the object, goal, task identity, and physical task content while differing in robot initialization, approach trajectory, camera viewpoint, lighting, object appearance, and execution timing. Shared task content is assigned to $r=[r_H,r_L]$3, whereas execution- and observation-specific factors are assigned to $r=[r_H,r_L]$4. A policy using $r=[r_H,r_L]$5 should consequently be less sensitive to camera or lighting changes that do not alter the task.

Reconstruction and organization

Reconstruction-based approaches commonly use an encoder-decoder mapping,

$r=[r_H,r_L]$6

with an objective such as

$r=[r_H,r_L]$7

Reconstruction requires a decoder to model the entire observation process, including pixels, texture, lighting, background details, and sensor noise. More fundamentally, perfect reconstruction does not determine how invariant and variant information should be allocated between $r=[r_H,r_L]$8 and $r=[r_H,r_L]$9.

For any smooth $r_H$0, one can define

$r_H$1

and an alternative observation map

$r_H$2

The observations remain unchanged, although

$r_H$3

is generally nonzero. This is the paper’s Nonlinear Latent Ambiguity result: observational equivalence does not imply invariant-variant identifiability.

2. Predictive identification and residual block ambiguity

A JEPA avoids reconstructing observations by predicting a target representation from a context representation. In the theoretical formulation, a generic predictive objective is

$r_H$4

Under the paper’s assumptions, predictive learning identifies the complete latent state up to a global orthogonal transformation:

$r_H$5

This removes the nonlinear ambiguity associated with reconstruction. However, a global orthogonal matrix can still mix the invariant and variant blocks:

$r_H$6

The resulting representation is

$r_H$7

Although the full latent state remains recoverable through $r_H$8, the block organization is not. Predictive learning therefore identifies the world while leaving the invariant-variant partition ambiguous.

SplitJEPA resolves this residual ambiguity using matched invariant correspondences. For an invariant pair,

$r_H$9

so that the invariant component is shared while the variant component may differ. The learner observes only the corresponding observations, not the latent decomposition itself.

The central coverage requirement is full-rank variation in the variant differences:

$r_L$0

with

$r_L$1

Equivalently,

$r_L$2

The condition does not require every individual pair to vary in every direction. It requires the aggregate set of matched-pair differences to span the entire variant subspace.

3. Architecture and learning objectives

SplitJEPA contains context and target encoders, a predictor, an invariant-block consistency objective, and variance-preserving regularization. The predictor operates entirely in representation space; no image or observation decoder is required.

Predictive branch

In the robotic implementation, an encoder produces

$r_L$3

A predictor receives the current representation and action,

$r_L$4

while the target representation is obtained from a stop-gradient target branch,

$r_L$5

The predictive loss is

$r_L$6

The theoretical formulation uses the corresponding pairwise predictive objective. The target-side representation is not reconstructed in observation space.

Invariant-block consistency

For matched invariant pairs, SplitJEPA applies consistency only to the invariant block:

$r_L$7

The variant block is not forced to agree. Applying consistency to the entire representation would discard the variation that SplitJEPA is designed to preserve.

Variance and covariance regularization

Predictive alignment alone admits collapse. SplitJEPA therefore combines predictive and invariant objectives with variance-preserving regularization:

$r_L$8

At the population-theory level, the representation is centered and whitened:

$r_L$9

Whitening restricts the remaining global ambiguity to an orthogonal transformation, after which invariant consistency removes the cross-block components.

In synthetic experiments, variance preservation uses SIGReg. In the robotic implementation, variance and covariance regularization are used. The synthetic algorithm explicitly separates global recovery from block recovery:

1. Train a reconstruction-free predictive encoder.
2. Freeze the encoder.
3. Compute centered ZCA whitening from unlabeled representations.
4. Form matched invariant pairs.
5. Compute representation differences,

$$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$0

1. Estimate the difference covariance,

$$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$1

1. Use the bottom $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$2 eigenspace of $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$3 as the invariant subspace.
2. Use its orthogonal complement as the variant subspace.

If $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$4, then for an invariant pair,

$$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$5

The population difference covariance therefore has a nullspace corresponding to the invariant subspace, while the full-rank variation condition makes the complementary variant covariance nonsingular.

Robotic implementation

The robotic encoder is an ImageNet-initialized ResNet-50. Its output is divided into $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$6 and $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$7. The encoder and predictive components are trained with adjacent temporal transitions and matched pairs. In the reported manipulation experiments, the block dimensions are $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$8 for PickCube and $$z= \begin{bmatrix} z_H\ z_L \end{bmatrix} \in\mathbb R^d, \qquad d=d_H+d_L,$$9 for PushCube.

The formulation also uses a stop-gradient target branch and a predictor conditioned on actions. The precise implementation differs from other JEPA variants that use EMA target encoders, auxiliary reward heads, or separate action and future branches.

4. Identifiability theorem and stability properties

Block Ambiguity of predictive identification

Predictive identification alone is invariant under an additional orthogonal transformation. If

$z_H\in\mathbb R^{d_H}$0

then for any orthogonal $z_H\in\mathbb R^{d_H}$1,

$z_H\in\mathbb R^{d_H}$2

has the same global predictive geometry. Since $z_H\in\mathbb R^{d_H}$3 may contain cross-block terms, predictive learning alone cannot identify the invariant-variant partition.

Reconstruction-free Block Identifiability Theorem

Under the stationary Gaussian predictive model, centered and whitened representations, and the full-rank variation condition, let $z_H\in\mathbb R^{d_H}$4 be a global population minimizer of the predictive objective. If invariant-block consistency holds almost surely,

$z_H\in\mathbb R^{d_H}$5

then

$z_H\in\mathbb R^{d_H}$6

where

$z_H\in\mathbb R^{d_H}$7

Consequently,

$z_H\in\mathbb R^{d_H}$8

The theorem proceeds in two stages.

First, under Gaussian predictive dynamics, a Hermite expansion decomposes representation coordinates by polynomial degree. Degree-$z_H\in\mathbb R^{d_H}$9 components are attenuated by

$z_L\in\mathbb R^{d_L}$0

Because $z_L\in\mathbb R^{d_L}$1,

$z_L\in\mathbb R^{d_L}$2

The globally optimal normalized predictive representation therefore lies in the degree-one subspace, yielding $z_L\in\mathbb R^{d_L}$3. Whitening makes $z_L\in\mathbb R^{d_L}$4 orthogonal.

Second, partition $z_L\in\mathbb R^{d_L}$5 as

$z_L\in\mathbb R^{d_L}$6

For an invariant pair,

$z_L\in\mathbb R^{d_L}$7

Exact invariant consistency implies

$z_L\in\mathbb R^{d_L}$8

Therefore,

$z_L\in\mathbb R^{d_L}$9

Since $h=f(z_H,z_L).$0, it follows that $h=f(z_H,z_L).$1. Orthogonality then forces $h=f(z_H,z_L).$2, leaving only independent transformations within the two blocks.

The result identifies subspaces, not individual coordinates. Rotations and reflections within $h=f(z_H,z_L).$3 and $h=f(z_H,z_L).$4 remain permissible.

Approximate consistency

The theorem has a stability form. Suppose global recovery gives $h=f(z_H,z_L).$5, and invariant consistency has error at most $h=f(z_H,z_L).$6:

$h=f(z_H,z_L).$7

If

$h=f(z_H,z_L).$8

then

$h=f(z_H,z_L).$9

and consequently

$\phi$0

Stronger variation improves the conditioning of block recovery, while greater invariant-consistency error permits more cross-block leakage.

Gradient isolation

If a downstream predictor uses only $\phi$1,

$\phi$2

then block identification gives

$\phi$3

Under perturbations that alter only $\phi$4 while holding the context fixed,

$\phi$5

This is a theorem under the stated assumptions, not a guarantee that every real-world perturbation is harmless.

5. Relation to other SplitJEPA-like architectures

The term SplitJEPA is sometimes used descriptively for JEPA systems that divide information, processing, or objectives into specialized components. These systems are not equivalent to the method defined in (Hua et al., 8 Oct 2026).

Controllability Factorized JEPA

Controllability Factorized JEPA, or CF-JEPA, splits a latent into controllable and uncontrollable components,

$\phi$6

with $\phi$7 and $\phi$8 in the reported configuration. Separate predictors model the two components:

$\phi$9

Inverse dynamics is applied to $r=\phi(h)=g(z), \qquad g=\phi\circ f.$0, while an adversarial action-prediction model with gradient reversal suppresses action information in $r=\phi(h)=g(z), \qquad g=\phi\circ f.$1. Planning uses only $r=\phi(h)=g(z), \qquad g=\phi\circ f.$2. CF-JEPA therefore organizes the latent according to controllability rather than matched invariant-variant correspondences (Byrd et al., 30 Sep 2026).

Its reported standard-distractor success rates were:

Task	LeWM	SMWM	CF-JEPA
Reacher	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$3	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$4	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$5
PushT	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$6	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$7	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$8
OGBench Cube	$r=\phi(h)=g(z), \qquad g=\phi\circ f.$9	$r=[r_H,r_L]$00	$r=[r_H,r_L]$01
TwoRoom	$r=[r_H,r_L]$02	$r=[r_H,r_L]$03	$r=[r_H,r_L]$04

CF-JEPA is consequently relevant as a controllability-based split architecture, but its theoretical target and training constraints differ from SplitJEPA’s invariant-variant block identification.

Auxiliary-task JEPA

P-JEPA augments latent transition prediction with an auxiliary prediction head for a function $r=[r_H,r_L]$05 such as reward. Its No Unhealthy Representation Collapse theorem states that, under deterministic transitions and zero latent-dynamics and auxiliary losses, observations that are non-equivalent under the dynamics-plus-auxiliary bisimulation must map to distinct latent representations (Yu et al., 12 Sep 2025).

This principle concerns which distinctions must be retained, not how invariant and variant factors are partitioned into identifiable blocks. The result is one-sided: non-equivalent observations cannot collapse, but equivalent observations need not be merged. It therefore complements, rather than replaces, SplitJEPA’s block-identification objective.

Distributed JEPA

Distributed JEPA separates temporal and contextual encoders, global representation learning and local decoding, and processing across energy clients and assets. It predicts latent representations of masked temporal segments, uses an EMA target encoder, and applies covariance and temporal-variance regularization (Toderean et al., 15 Sep 2026).

The framework is distributed and modular but is not formally named SplitJEPA, does not implement standard split learning, and does not provide a complete federated optimization protocol. Its separation concerns data ownership, modalities, and global/local decoders rather than invariant and variant latent subspaces.

Shared and decoupled policy branches

JEPA Policy predicts action chunks and future visual representations through a shared Transformer. Its action and future tokens interact through unmasked self-attention, and future-prediction gradients reach the action-producing trunk (Xu et al., 9 Sep 2026).

The paper evaluates Dual-independent and Dual-cross controls, corresponding to decoupled branches and separately parameterized branches with cross-attention. The full shared architecture has a mean simulated success of $r=[r_H,r_L]$06, compared with $r=[r_H,r_L]$07 for Dual-independent and $r=[r_H,r_L]$08 for Dual-cross. These results concern whether action and future prediction should share computation; they do not establish the invariant-variant subspace identifiability addressed by SplitJEPA.

Language-compatible predictive representations

The JEPA Paradox in Language argues that deterministic squared-error latent prediction can map multiple plausible text completions to a single conditional centroid. It motivates multiple prediction spaces, prototypes, mixtures, or distributional objectives rather than a single deterministic latent prediction (Dinh et al., 26 Jul 2026).

This issue is related to SplitJEPA because both question whether one undifferentiated latent prediction space is sufficient. However, the language work does not implement SplitJEPA and addresses multimodal conditional prediction rather than invariant-variant block identification.

Tabular JEPA wiring

A JEPA recipe for tabular foundation models separates a value-prediction path from a latent-prediction path. The value head reads the encoder field, while a predictor receives masked encoder features and predicts an EMA-difference target. This wiring avoids a prior failure in which the value head read a collapsed predictor output (Jeon et al., 22 Sep 2026).

The architecture is a split between value and representation objectives, not between invariant and variant latent factors. Its principal finding is that the latent objective can survive without collapse, although the reported JEPA arm trails the value-only arm across the 147 real datasets.

Theorem-proving JEPA

In Lean theorem proving, a context encoder receives a proof state and tactic, a predictor estimates the successor representation, and an EMA target encoder represents the actual Lean successor (Choudhary, 26 Sep 2026). JEPA achieves $r=[r_H,r_L]$09 same-theorem Top-1 ranking versus $r=[r_H,r_L]$10 for matched InfoNCE, but solves an average of $r=[r_H,r_L]$11 of $r=[r_H,r_L]$12 eligible theorems, compared with $r=[r_H,r_L]$13 for proposer ordering.

This work demonstrates that improved local transition representation does not necessarily yield improved long-horizon search. It is relevant to SplitJEPA insofar as it illustrates the difference between predictive usefulness and downstream organization, but it does not introduce an invariant-variant split.

6. Experiments, applications, and limitations

Synthetic nonlinear systems

The synthetic latent state is eight-dimensional, with

$r=[r_H,r_L]$14

Observations are generated by a four-layer smooth nonlinear coupling map with coupling strength $r=[r_H,r_L]$15. Predictive pairs use an Ornstein-Uhlenbeck-style Gaussian correlation with $r=[r_H,r_L]$16.

The encoder is trained without a decoder or reconstruction loss for 10,000 updates, using batch size $r=[r_H,r_L]$17, learning rate $r=[r_H,r_L]$18, and SIGReg weight $r=[r_H,r_L]$19. After training, it is frozen. Whitening and block recovery use 30,000 samples and 30,000 matched invariant pairs.

The primary results are:

Method	Invariant $r=[r_H,r_L]$20	Variant $r=[r_H,r_L]$21	$r=[r_H,r_L]$22	$r=[r_H,r_L]$23	Purity
LeJEPA	$r=[r_H,r_L]$24	$r=[r_H,r_L]$25	$r=[r_H,r_L]$26	$r=[r_H,r_L]$27	$r=[r_H,r_L]$28
Shuffled pairs	$r=[r_H,r_L]$29	$r=[r_H,r_L]$30	$r=[r_H,r_L]$31	$r=[r_H,r_L]$32	$r=[r_H,r_L]$33
SplitJEPA	$r=[r_H,r_L]$34	$r=[r_H,r_L]$35	$r=[r_H,r_L]$36	$r=[r_H,r_L]$37	$r=[r_H,r_L]$38

All methods achieve approximately the same global recovery, with

$r=[r_H,r_L]$39

The distinction is therefore organizational: SplitJEPA preserves essentially the same total information as predictive learning while substantially reducing cross-block leakage.

Removing predictive alignment damages global recovery. Removing invariant-pair recovery leaves global $r=[r_H,r_L]$40 high but increases cross-block leakage. Removing variance regularization and ZCA normalization increases covariance error and instability. More predictive training improves global recovery, while more matched-pair samples improve block recovery.

Under additional dense orthogonal mixing and an elementwise nonlinear map, SplitJEPA obtains approximately

$r=[r_H,r_L]$41

with

$r=[r_H,r_L]$42

When nonlinear coupling strengths increase from $r=[r_H,r_L]$43 to $r=[r_H,r_L]$44 and $r=[r_H,r_L]$45, global $r=[r_H,r_L]$46 decreases from $r=[r_H,r_L]$47 to $r=[r_H,r_L]$48 and $r=[r_H,r_L]$49. This reflects failure of the global predictive-identification stage under stronger nonlinear coupling.

Efficiency

SplitJEPA is compared with V3, a reconstruction-based content-style method using variance-invariance constraints, vector quantization, and an encoder-decoder pathway. At the target

$r=[r_H,r_L]$50

SplitJEPA requires $r=[r_H,r_L]$51 seconds, compared with $r=[r_H,r_L]$52 seconds for V3. The reported comparison also gives approximately $r=[r_H,r_L]$53 times lower runtime, approximately $r=[r_H,r_L]$54 times fewer observations, and 64 trainable parameters versus 272 for V3. SplitJEPA reaches the target in all five runs, compared with three runs for V3.

These measurements are specific to the controlled synthetic implementation and do not establish a general computational advantage.

Robotic manipulation

Experiments use ManiSkill3 PushCube-v1 and PickCube-v1. Each task contains 800 successful trajectories organized into 100 groups of eight trajectories. Observations are $r=[r_H,r_L]$55 RGB images with four-frame temporal context and eight-dimensional actions.

Within each group, trajectories share object identity, initial object pose, and goal pose while differing in robot initialization, approach, execution trajectory, contact direction, and timing. Correspondences include cross-view pairs from base and side cameras at the same physical state and approximate cross-trajectory pairs aligned by normalized execution progress.

For PushCube, the $r=[r_H,r_L]$56-only policy obtains:

• ID SR@10cm: $r=[r_H,r_L]$57;
• camera OOD: $r=[r_H,r_L]$58;
• lighting OOD: $r=[r_H,r_L]$59;
• combined visual OOD: $r=[r_H,r_L]$60.

For PickCube, it obtains:

• ID SR@10cm: $r=[r_H,r_L]$61;
• camera OOD: $r=[r_H,r_L]$62;
• lighting OOD: $r=[r_H,r_L]$63;
• combined visual OOD: $r=[r_H,r_L]$64.

The full representation $r=[r_H,r_L]$65 is sometimes better when execution-specific information is useful, particularly for PushCube combined shifts, whereas $r=[r_H,r_L]$66 is generally more robust to nuisance changes. On PushCube camera OOD, LeJEPA obtains $r=[r_H,r_L]$67, compared with $r=[r_H,r_L]$68 for the SplitJEPA $r=[r_H,r_L]$69 policy.

Under matched viewpoint changes, the response of $r=[r_H,r_L]$70 is $r=[r_H,r_L]$71 times larger than that of the invariant block for PickCube and $r=[r_H,r_L]$72 times larger for PushCube. Across matched trajectories, the corresponding factors are $r=[r_H,r_L]$73 and $r=[r_H,r_L]$74.

Controlled corruption of $r=[r_H,r_L]$75 further supports selective use of the invariant block. With $r=[r_H,r_L]$76 noise, PushCube achieves $r=[r_H,r_L]$77 with $r=[r_H,r_L]$78 only versus $r=[r_H,r_L]$79 with $r=[r_H,r_L]$80; PickCube achieves $r=[r_H,r_L]$81 versus $$29.6\%.</p> <h3 class='paper-heading' id='limitations'>Limitations</h3> <p>The theoretical guarantees rely on stationary Gaussian predictive dynamics, an exact global population optimum, centered and whitened representations, correct invariant correspondences, known block dimensions, and full-rank variation. Violations of these conditions weaken or invalidate the identifiability result.</p> <p>If$$r=[r_H,r_L]$82 is singular, some variant direction never changes across the available matched pairs and cannot be distinguished from invariant content. Incorrect or approximate correspondences can also introduce false variation in the nominally invariant factors.

The notion of invariance is task-relative. A camera viewpoint may be a nuisance for object pushing but essential for viewpoint estimation. SplitJEPA identifies the structure induced by the supplied correspondences rather than a universally correct semantic decomposition.

The experiments use controlled visual shifts and relatively small manipulation tasks. Robustness to texture variation, lighting beyond the evaluated conditions, shadows, complex objects, camera motion, semantic distractors, partial task occlusion, and interacting agents is not established. The method also assumes that variant factors do not alter controllable dynamics in the way excluded by its factorization assumptions.

Finally, the theorem identifies subspaces rather than individual semantic coordinates. Rotations and reflections within each block remain unresolved, and the framework does not prove complete disentanglement inside the invariant or variant components. Its central contribution is narrower and more precise: predictive JEPA learning recovers a latent world up to a global orthogonal transformation, while matched invariant correspondences remove cross-block mixing and identify invariant and variant subspaces without reconstruction.