Papers
Topics
Authors
Recent
Search
2000 character limit reached

SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction

Published 8 Oct 2026 in cs.LG | (2610.12349v1)

Abstract: Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them. For example, a robot pushing a cube to a goal should take the same action when the camera shifts or the lights dim, since nothing in the scene has moved. Existing approaches to this decomposition commonly obtain it through reconstruction, so the latent variables must first explain the entire observational world before their organization can be trusted. Joint embedding predictive architectures (JEPAs) model the latent state directly and never reconstruct, yet no existing result recovers the invariant and variant parts of the state they learn. How to learn the invariant-variant structure of the latent world without paying for its reconstruction therefore remains open. To close this gap, we introduce SplitJEPA, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction. We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries, without introducing an observation decoder. Since the guarantee needs no decoder, the result extends reconstruction-free latent recovery to invariant-variant block identification. Experiments on synthetic nonlinear systems and robotic manipulation tasks support the theoretical results and show their practical value for both robustness and efficiency.

Summary

  • The paper develops SplitJEPA, which jointly resolves predictive identification in temporal dynamics and invariant correspondence by solving the block recovery problem without reconstructing observations.
  • For a synthetic system, SplitJEPA achieves high block separation with near perfect invariant and variant block R² of 0.9979 and robust performance in robotic manipulation tasks.
  • Robotic experiments demonstrate improved performance in tasks like PushCube under shifted visual factors, with stronger performance on the task with invariant blocks because the variant information disturbs task-specific content.

Problem formulation and contribution

SplitJEPA addresses a narrower and more demanding objective than generic representation learning: recovering a latent state together with its decomposition into invariant and variant subspaces, without reconstructing observations. The distinction is operationally important in dynamical control. A manipulation policy should preserve task-relevant state while suppressing changes induced by viewpoint, illumination, execution trajectory, or other nuisance factors when those changes do not alter the control objective.

The paper models observations as h=f(z)h=f(z), where the latent state is partitioned as z=[zH,zL]z=[z_H,z_L]. The block zHz_H is shared across a specified correspondence, whereas zLz_L may vary. An encoder produces r=ϕ(h)r=\phi(h), partitioned analogously into rHr_H and rLr_L. The target is not coordinate-wise disentanglement: each recovered block may undergo an independent orthogonal transformation. The desired identification statement is therefore

rH=A11zH,rL=A22zL,r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,

with A11A_{11} and A22A_{22} orthogonal. This is a subspace-level claim, excluding cross-block mixing while permitting arbitrary rotations within each block.

The central methodological claim is that two relational signals resolve two distinct ambiguities. Predictive learning identifies the complete latent state up to a global orthogonal transformation; invariant-pair consistency then determines which directions belong to the invariant block. SplitJEPA implements this decomposition using latent prediction, consistency of z=[zH,zL]z=[z_H,z_L]0 across matched pairs, and variance/covariance regularization. It does not introduce an observation decoder or a pixel-level reconstruction loss. The paper’s principal theoretical result is consequently stronger than merely demonstrating that reconstruction is unnecessary for useful representations: under its assumptions, reconstruction is also unnecessary for identifying the invariant–variant partition (2610.12349).

Why reconstruction and prediction alone are insufficient

The paper first establishes that the block structure cannot be recovered from observations alone. Given any smooth function z=[zH,zL]z=[z_H,z_L]1 of the variant variables, the transformation

z=[zH,zL]z=[z_H,z_L]2

can be absorbed into a redefined observation map. The resulting latent parameterization generates exactly the same observations while making the nominal invariant block depend on z=[zH,zL]z=[z_H,z_L]3. Thus, information preservation and observational equivalence do not imply recovery of the intended block decomposition. This is a standard nonlinear latent-variable ambiguity, but the paper uses it to distinguish global state recovery from structured state recovery.

The predictive component reduces this ambiguity substantially. Under the assumed stationary Gaussian predictive dynamics,

z=[zH,zL]z=[z_H,z_L]4

with conditional correlation z=[zH,zL]z=[z_H,z_L]5, a centered and whitened population minimizer of the predictive objective is shown to have the form z=[zH,zL]z=[z_H,z_L]6, where z=[zH,zL]z=[z_H,z_L]7 is orthogonal. The proof uses the Hermite expansion of the representation: degree-z=[zH,zL]z=[z_H,z_L]8 components receive correlation factor z=[zH,zL]z=[z_H,z_L]9, which is strictly smaller for zHz_H0 than for degree one. Predictive alignment therefore suppresses nonlinear components at the population optimum.

This result leaves a consequential residual ambiguity. A global orthogonal matrix can mix zHz_H1 and zHz_H2 while preserving whitening, distances, and the predictive objective. Hence, predictive identification alone recovers the complete state but not its organization. The paper makes this distinction explicit: a representation can be globally linearly equivalent to the latent state while still assigning variant directions to the invariant block. This is the ambiguity that invariant correspondences are designed to resolve.

Block identifiability from sufficient variation

The second identifying signal consists of pairs sharing zHz_H3 but differing in zHz_H4. Let zHz_H5 for such pairs. The key condition is

zHz_H6

This full-rank variation condition requires that no nonzero linear direction in the variant subspace remain unchanged across all invariant correspondences. It does not require every individual pair to vary in every coordinate, nor does it require independent variation. Complementary pair families can collectively span the entire variant subspace.

After predictive identification, the invariant representation has the form

zHz_H7

Because matched pairs share zHz_H8, invariant consistency implies

zHz_H9

If this difference vanishes almost surely and zLz_L0 is positive definite, then zLz_L1. Orthogonality of zLz_L2 subsequently forces zLz_L3, yielding a block-diagonal transformation. Theorem 2 therefore identifies both subspaces up to independent within-block orthogonal transformations, without access to latent variables and without an observation decoder.

The assumption is substantive rather than technical bookkeeping. If viewpoint and lighting always change in one fixed combination, another variant direction may never be exposed and can remain indistinguishable from invariant content. The theorem does not identify variation that the correspondence distribution fails to reveal. In finite samples, the paper gives a stability version: if zLz_L4 and the invariant consistency error is at most zLz_L5, then

zLz_L6

Thus, stronger variation coverage improves robustness to approximate consistency, while weakly varying directions are intrinsically difficult to separate.

Estimation procedure

The practical objective combines three components:

  1. Predictive alignment, which preserves information useful for temporal prediction.
  2. Invariant-block consistency, applied only to zLz_L7 for matched pairs.
  3. Variance and covariance regularization, which prevents collapse and approximates the centered, whitened geometry required by the theory.

The synthetic implementation separates global representation learning from block recovery. A reconstruction-free encoder is first trained and frozen. Its representations are centered and ZCA-whitened, after which the invariant subspace is estimated from the bottom eigenspace of the matched-pair difference covariance. Under exact block structure, invariant directions have zero difference energy, while variant directions contribute positive energy. The robotic implementation jointly optimizes the analogous components using an action-conditioned predictor, matched cross-view and cross-trajectory pairs, and variance/covariance penalties.

This separation clarifies the causal role of each component. Prediction is responsible for global linear recovery; correspondence information selects the invariant orientation within the globally recovered state; normalization supplies the metric structure needed to eliminate the complementary cross-block leakage. The paper’s ablations support this decomposition rather than treating the complete objective as an undifferentiated regularized loss.

Synthetic identification results

The synthetic system uses an eight-dimensional latent state with four invariant and four variant dimensions, a nonlinear observation map, Gaussian OU predictive pairs, and no decoder or reconstruction objective. All main methods reuse the same frozen predictive encoder, isolating block organization from encoder quality.

The numerical separation between global recovery and block recovery is strong:

Method Invariant zLz_L8 Variant zLz_L9 r=ϕ(h)r=\phi(h)0 r=ϕ(h)r=\phi(h)1 Block purity
LeJEPA r=ϕ(h)r=\phi(h)2 r=ϕ(h)r=\phi(h)3 r=ϕ(h)r=\phi(h)4 r=ϕ(h)r=\phi(h)5 r=ϕ(h)r=\phi(h)6
Shuffled pairs r=ϕ(h)r=\phi(h)7 r=ϕ(h)r=\phi(h)8 r=ϕ(h)r=\phi(h)9 rHr_H0 rHr_H1
SplitJEPA rHr_H2 rHr_H3 rHr_H4 rHr_H5 $0.99993

Global $r_H$6 is approximately $r_H$7 for all methods, showing that the comparison is not primarily about retaining more latent information. LeJEPA preserves the state but leaves substantial cross-block mixing. Shuffling the correspondences destroys the partition while retaining access to the same marginal observations. SplitJEPA instead produces near-perfect block recovery. The result directly supports the paper’s central claim that meaningful correspondence information resolves an ambiguity that predictive recovery alone cannot resolve.

The scaling experiments further separate the data requirements of the two stages. More predictive training improves global recovery, with a transition around $r_H$8 predictive pairs and limited gains beyond $r_H$9. With the encoder fixed, increasing the block-recovery budget from 250 to 30,000 samples reduces normalized $r_L$0 from $r_L$1 to $r_L$2. Increasing latent dimension has little effect on global $r_L$3, but makes estimation of the complementary block more sensitive; normalized $r_L$4 rises from $r_L$5 at dimension four to $r_L$6 at dimension 64.

The efficiency comparison is also notable. At a common target of average invariant and variant $r_L$7 at least $r_L$8, SplitJEPA reaches the target in all five runs, compared with three for the reconstruction-based variance-invariance method V3. Under the capped protocol, SplitJEPA requires $r_L$9 seconds versus $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$0 seconds, a $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$1 reduction, and $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$2 million observations versus $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$3 million, a $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$4 reduction. It also uses 64 trainable parameters versus 272 in the comparison. These figures are specific to the controlled implementation and protocol, but they substantiate the computational motivation for avoiding reconstruction.

The nonlinear robustness experiments qualify the result. Under joint nonlinear MLP mixing, SplitJEPA achieves invariant and variant $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$5 values of $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$6 and $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$7, with cross-block leakages of $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$8 and $r_H=A_{11}z_H,\qquad r_L=A_{22}z_L,$9, while LeJEPA remains near $A_{11}$0 for both block scores. However, increasing the coupling strength from $A_{11}$1 to $A_{11}$2 lowers global $A_{11}$3 from $A_{11}$4 to $0.8765 and raises normalized leakage. The block theorem presupposes that predictive training reaches the globally identifiable linear class; finite optimization and stronger nonlinear coupling violate this condition approximately, and the empirical degradation tracks that violation.

Robustness and robotic manipulation

The paper evaluates PickCube and PushCube in ManiSkill using 800 successful trajectories per task. Matched data include cross-view pairs and approximate cross-trajectory correspondences. Policies are trained by behavior cloning on in-distribution demonstrations and evaluated under camera, lighting, and combined visual shifts.

The principal result is improved out-of-distribution control, especially for PushCube:

Method PushCube ID SR PushCube camera PushCube visual PickCube ID SR PickCube camera PickCube visual
Raw 98.0 12.8 12.7 14.0 10.3 10.7
R3M 97.7 48.0 25.8 32.0 14.3 10.0
LeJEPA 67.3 25.0 28.7 16.3 10.0 10.0
SplitJEPA A11A_{11}5 99.3 71.5 64.5 30.3 42.7 28.3
SplitJEPA A11A_{11}6 99.3 75.3 70.2 27.7 43.3 30.3

The invariant block substantially improves PushCube camera-shift performance over R3M, from A11A_{11}7 to A11A_{11}8 SR@10cm, and combined visual-shift performance from A11A_{11}9 to A22A_{22}0r_HA22A_{22}1r_H+r_Listask−andshift−dependent.Variantinformationcanbeusefulwhenexecution−specificdetailsmatter,soinvariant−onlyinputisnotuniformlyoptimal.</p><p>Representation−leveldiagnosticssupportthebehavioralresults.Underviewpointchanges,thevariantblockresponds is task- and shift-dependent. Variant information can be useful when execution-specific details matter, so invariant-only input is not uniformly optimal.</p> <p>Representation-level diagnostics support the behavioral results. Under viewpoint changes, the variant block responds A_{22}$2 more strongly than the invariant block on PickCube and $A_{22}$3 more strongly on PushCube. Across matched trajectories, the corresponding ratios are $A_{22}$4 and $A_{22}$5. These measurements provide evidence that the learned blocks differ in sensitivity rather than merely producing better downstream action scores.

The controlled intervention experiment directly tests the gradient-isolation claim. When perturbations are injected into $A_{22}$6, policies using only $A_{22}$7 retain approximately $A_{22}$8 of clean performance. With both blocks exposed, performance degrades as the perturbation magnitude increases. On PickCube, two-standard-deviation noise reduces performance from the $A_{22}$9 baseline by $z=[z_H,z_L]$00 percentage points for the combined representation, whereas the invariant-only policy remains at approximately $z=[z_H,z_L]$01. This is consistent with the theorem’s restricted guarantee: a downstream predictor that depends only on $z=[z_H,z_L]$02 is insensitive to perturbations confined to $z=[z_H,z_L]$03 or its recovered counterpart. It does not imply robustness to perturbations that alter task-relevant invariant content.

The ablations reinforce the necessity of the proposed structure. Removing predictive learning reduces visual-shift performance, while removing invariance causes the largest PushCube degradation: camera SR falls from $z=[z_H,z_L]$04 to $z=[z_H,z_L]$05 for the invariant-only interface, and combined visual SR falls to $z=[z_H,z_L]$06. The diversity ablation is less uniform. In some PushCube conditions it improves performance, whereas on PickCube it reduces camera and combined-shift results. This is important because practical cross-trajectory pairings trade alignment precision against variation coverage, whereas the population theorem assumes exact invariant correspondences with full-rank variant changes.

Limitations and open questions

The theoretical result depends on a restrictive population model. Stationary Gaussian predictive dynamics, exact centering and whitening, a square representation of the correct dimension, global optimization, and positive-definite variant-change covariance are all central to the proof. The result does not establish identifiability for non-Gaussian, nonstationary, partially observed, or misspecified dynamical systems. Nor does it address unknown block dimensions. In particular, a variant direction absent from the correspondence distribution is formally indistinguishable from invariant content.

The empirical robotics setting only approximates the correspondence assumptions. Cross-trajectory pairs are aligned by normalized execution progress and group-level task identity rather than exact latent-state equality. Consequently, practical performance reflects a trade-off between correspondence accuracy and diversity that is not captured by the exact theorem. The robotic experiments are also limited to two simulated manipulation tasks, 800 trajectories per task, and controlled visual interventions. The conclusion explicitly acknowledges scale as the principal limitation. A specific unresolved question is whether sufficiently informative correspondences can be constructed in large, real-world datasets where invariant content is only approximately shared and the sources of variation are neither labeled nor controlled.

The paper also reports degradation when the predictive encoder fails to reach the required global geometry. At coupling strength z=[zH,zL]z=[z_H,z_L]07, global z=[zH,zL]z=[z_H,z_L]08 drops to z=[zH,zL]z=[z_H,z_L]09, with substantial orthogonality error and increased leakage. Thus, block consistency cannot compensate indefinitely for failures in the preceding predictive-identification stage. The framework’s empirical robustness is therefore conditional on maintaining a sufficiently accurate global latent representation.

Conclusion

SplitJEPA separates two identification problems that are often conflated. Predictive latent learning recovers the complete state up to a global orthogonal transformation; invariant correspondences, provided their variant changes span the relevant subspace, remove the residual cross-block ambiguity. The resulting representation is identifiable up to independent within-block isometries without observation reconstruction.

The synthetic experiments show near-perfect block recovery, lower computational cost than the reconstruction-based comparison, and clear failure when correspondences are shuffled. Robotic experiments show improved performance under visual shifts and strong resistance to interventions confined to the variant block, although the guarantees remain conditional on the predictive model, the correspondence structure, and finite-sample estimation. The paper’s main contribution is therefore a formal and empirical account of how reconstruction-free predictive learning can recover not only latent information, but also a specific subspace-level organization of that information (2610.12349).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

The paper introduces a machine-learning method called SplitJEPA. Its goal is to help computers understand the important parts of a changing world, especially when the same situation can look different.

For example, imagine a robot pushing a cube. The camera might move, the lighting might change, or the robot might approach from a different angle. These details change what the robot sees, but they may not change the real task: push the cube to the goal.

The paper tries to separate information into two groups:

  • Invariant information: things that stay the same, such as the task or the cube’s goal.
  • Variant information: things that can change, such as lighting, camera angle, or the robot’s exact movement.

The main idea is to learn this separation without rebuilding the original images or observations.

2. What questions are the researchers asking?

The paper focuses on several main questions:

  1. Can a computer learn a useful hidden description of the world without reconstructing the original images?
  2. Can it separate stable information from changing information?
  3. What kind of training data is needed to make this separation reliable?
  4. Will this separation help robots perform better when their surroundings look different from the training examples?

The researchers are especially interested in whether the computer can identify the correct groups of information, rather than simply remembering all the information in a mixed-up way.

3. How does the method work?

Learning a hidden description

A computer does not directly understand an image in the same way a person does. Instead, a neural network changes each observation into a list of numbers called a representation or latent state.

You can think of this as making a short summary of a picture:

  • some numbers describe the important task,
  • other numbers describe temporary details,
  • and together they describe the whole situation.

The true world information is called the latent state, while the camera image or other input is called the observation.

Why reconstruction is not enough

Many systems learn by trying to recreate the original image. This is called reconstruction. It is similar to looking at a photograph, compressing it into a set of notes, and then trying to redraw the photograph from those notes.

Reconstruction can preserve many details, including details that are not useful. It also does not guarantee that the notes are organized into “stable” and “changing” information.

For example, a system might store the lighting information together with the task information. It could still redraw the picture correctly, but its internal organization would be messy.

Prediction in representation space

SplitJEPA is based on a method called a Joint-Embedding Predictive Architecture, or JEPA.

Instead of rebuilding images, JEPA tries to predict one hidden representation from another related observation. For instance, it may learn that two views of the same robot task should have related internal descriptions.

This is like comparing two summaries rather than trying to redraw both photographs.

The paper uses two types of relationships:

  • Predictive pairs: observations that follow one another or are related in time. These help the system learn the complete hidden state.
  • Invariant pairs: observations that share the same important content but differ in appearance. For example, two views of the same cube and task, taken with different cameras or lighting.

SplitJEPA is trained so that:

  1. Its whole representation can predict related observations.
  2. The part labeled “invariant” is similar for invariant pairs.
  3. The representation keeps enough variation and does not collapse into the same answer for everything.

What does “identifiable” mean?

A central technical idea in the paper is identifiability. In simple terms, this means that the method can recover the intended structure instead of producing many equally possible but confusing versions.

Imagine rotating a box of colored blocks. The blocks are still there, but their positions in the box have changed. A normal prediction method may learn the right information while mixing the stable and changing blocks together.

SplitJEPA uses the invariant pairs to determine which directions should belong to the stable group and which should belong to the changing group.

Why varied examples matter

The paired examples must show enough different kinds of change. This is called the sufficient variation condition.

For example, if the camera always moves upward and never sideways, the system may not learn everything about camera-related variation. But if the examples include many different camera movements and lighting changes, the system gets stronger evidence about which information is truly variable.

The paper’s mathematical result says that, if the variation covers all directions of the changing information, SplitJEPA can separate the two groups, except for harmless mixing within each group.

4. What did the researchers find?

The researchers tested SplitJEPA in two main settings.

Synthetic experiments

First, they created a controlled artificial world where they already knew which factors were invariant and which were variant. This allowed them to check whether the method recovered the correct answer.

The results showed:

  • SplitJEPA recovered the invariant and variant groups very accurately.
  • It had almost no mixing between the two groups.
  • A standard predictive method, called LeJEPA, learned most of the information but mixed the two groups together.
  • Shuffling the paired examples made the separation much worse.

For example, SplitJEPA’s block-purity score was about 0.9999, meaning its learned groups were almost perfectly organized in the artificial test.

This supports an important point: learning the complete hidden state is not the same as organizing that state correctly.

Robot experiments

The researchers then tested the method on simulated robot tasks called PickCube and PushCube. The robot had to pick up or push a cube using visual observations.

The robot was tested in situations with:

  • the usual camera view,
  • different camera positions,
  • different lighting,
  • other visual changes.

SplitJEPA generally performed better than the comparison methods, especially when the camera or appearance changed.

For the PushCube task, using only the invariant representation achieved about:

  • 71.5% success under camera changes,
  • 99.7% success under lighting changes,
  • 64.5% success under broader visual changes.

Using both the invariant and variant representations performed even better in some cases, reaching about:

  • 75.3% success under camera changes,
  • 98.3% success under lighting changes,
  • 70.2% success under visual changes.

The PickCube task was harder, but SplitJEPA still usually outperformed the comparison methods.

Why these findings matter

These results suggest that separating stable information from temporary appearance changes can make robots more reliable.

A robot that focuses on the invariant part may continue to perform the correct action even if:

  • the camera moves,
  • the lights become brighter or darker,
  • the scene looks somewhat different.

The experiments also showed that the two parts are not always equally useful. Sometimes the robot benefits from using only the invariant part. In other situations, the changing information is also useful, so using both parts works better.

5. What are the broader implications?

SplitJEPA suggests that artificial intelligence can learn not only what information exists, but also how that information is organized.

This could be useful for robots and other systems that must work in changing environments. A robot could use the invariant part to make decisions that remain stable when appearance changes, while still using the variant part when details such as position or movement are important.

The method also avoids reconstructing images, which may make it:

  • more efficient,
  • less focused on unimportant visual details,
  • easier to use for tasks that care about actions rather than perfect image reproduction.

However, the method has important limits. It needs paired examples that preserve the important information while showing enough different kinds of variation. If the training data does not include certain changes, the system may incorrectly treat them as invariant. The experiments were also relatively small, so it is not yet certain how well the approach will work in very large and complicated real-world environments.

Overall, the paper’s main message is:

A computer may understand a changing world better when it learns both the complete hidden state and the difference between what stays the same and what changes. SplitJEPA attempts to do this directly, without rebuilding the original observations.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper establishes reconstruction-free block identifiability under strong assumptions, but leaves the following issues unresolved:

  • Generality beyond stationary Gaussian dynamics: The theoretical guarantee depends on isotropic, stationary Gaussian predictive dynamics with a context-dependent scalar correlation ρX\rho_X. It is unclear whether block identifiability holds for non-Gaussian, nonstationary, nonlinear, non-isotropic, or higher-order dynamical processes.
  • Validity of the predictive-identification premise: The results assume that the predictive objective already recovers the complete latent state up to a global orthogonal transformation. The paper does not establish when this premise holds for finite neural networks, pixel observations, imperfect optimization, or realistic data distributions.
  • Population-optimum versus trainable solutions: The main theorem applies to global population minimizers. It remains unknown whether gradient-based training reliably reaches solutions with the claimed block structure, particularly in the presence of nonconvexity, finite data, overparameterization, and competing loss terms.
  • Finite-sample block-recovery guarantees: The paper provides a qualitative approximate-consistency bound, but does not derive sample-complexity or confidence guarantees for estimating the invariant and variant subspaces from finite paired data.
  • Quantification of correspondence quality: The theory assumes exact invariant pairs sharing precisely the same zHz_H. The practical experiments use approximate cross-trajectory correspondences, but the paper does not provide a formal identifiability or error bound for mismatched pairs.
  • Robustness to false invariances: The method may force genuinely task-relevant factors into the invariant block when the pairing procedure incorrectly labels them as shared. The consequences of systematic pairing errors are not theoretically characterized.
  • Known latent and block dimensions: The analysis assumes that the total dimension dd and both block dimensions dHd_H and dLd_L are known. Methods for selecting these dimensions, detecting unequal or changing dimensions, or handling overcomplete and undercomplete representations remain unexplored.
  • Strength and interpretability of the sufficient-variation condition: Positive definiteness of ΣΔL\Sigma_{\Delta L} is necessary for the stated argument, but the paper does not determine how much variation is sufficient in finite samples or how the smallest eigenvalue affects practical recovery beyond the basic error bound.
  • Unobserved and weakly varying variant directions: Directions with zero or very small variation across available correspondences are theoretically indistinguishable from invariant factors, but the paper does not propose a way to identify, report, or safely handle such partially covered directions.
  • Dependence between invariant and variant factors: The analysis does not clarify whether zHz_H and zLz_L may be statistically dependent, how such dependence affects predictive recovery, or whether dependence can cause leakage despite full-rank variant differences.
  • Changing invariant–variant semantics across contexts: A factor may be invariant for one task or correspondence type but variant for another. The framework does not address context-dependent partitions or representations that must support multiple, potentially incompatible decompositions.
  • Limits of the observation model: The theory assumes a smooth embedding ff and a continuously differentiable encoder. It does not cover non-injective observations, occlusion, discontinuities, sensor quantization, missing modalities, stochastic observation processes, or observations with irrecoverable information loss.
  • Role of representation normalization in practice: Exact whitening is required by the theory, whereas the implementation uses finite-weight variance regularization and optional ZCA normalization. The relationship between covariance error and block-recovery error is not fully established.
  • Interaction among objective terms: The paper does not characterize how the choice of λinv\lambda_{\mathrm{inv}}, λvar\lambda_{\mathrm{var}}, batch size, predictor architecture, or normalization method changes identifiability and optimization stability.
  • Potential collapse or information loss in the invariant block: Agreement losses encourage invariance but may discard task-relevant information that varies within the chosen correspondences. The paper does not give a criterion for balancing invariance against preserving all useful shared information.
  • Block identification versus semantic factor identification: The guarantee recovers subspaces only up to independent within-block orthogonal transformations. It does not identify individual semantic factors, causal variables, or interpretable coordinates within either block.
  • Scope of the robustness theorem: Theorem 3 concerns predictors that use only rHr_H and perturbations that remain within the modeled support. It does not quantify robustness for finite distribution shifts, off-support perturbations, policy feedback loops, or predictors that use both rHr_H and rLr_L.
  • Relationship between representation robustness and closed-loop control: The experiments show improved success rates, but the paper does not establish that block identifiability causes the control gains, nor does it analyze how representation errors propagate through long-horizon closed-loop dynamics.
  • Limited empirical scale and diversity: Experiments use controlled synthetic systems and two ManiSkill manipulation tasks with 800 successful trajectories per task. Generalization to large-scale, real-world, multimodal, long-horizon, and failure-rich datasets remains untested.
  • Limited environmental shifts: The robotic evaluation focuses mainly on camera, lighting, and visual changes. It does not test substantial changes in object geometry, dynamics, friction, distractors, robot morphology, action latency, camera calibration, or task goals.
  • Dependence on successful-trajectory data: Because the robotic data consist of successful trajectories, it is unclear whether SplitJEPA remains effective with demonstrations containing failures, sparse rewards, exploratory behavior, or highly imbalanced correspondence classes.
  • Approximate pairing and diversity trade-offs: The experiments suggest that broader variation can conflict with correspondence accuracy, but the paper does not provide an algorithm for optimizing this trade-off or a principled way to select useful pairings.
  • Insufficient comparison with reconstruction-based and alternative disentanglement methods: The empirical evaluation primarily compares against Raw, R3M, LeJEPA, and shuffled pairs. It does not determine whether the proposed reconstruction-free approach outperforms strong reconstruction-based, contrastive, multiview, causal, or content–style baselines under matched compute and data budgets.
  • Computational and scaling costs: Although reconstruction is avoided, the paper does not report detailed training time, memory usage, pair-construction cost, or scaling behavior relative to reconstruction-based methods and large JEPA models.
  • Sensitivity to encoder pretraining and shared-encoder evaluation: The synthetic block-recovery experiment fixes a pretrained encoder for all methods, which isolates the spectral recovery stage but does not show whether end-to-end SplitJEPA consistently learns superior encoders from raw observations.
  • Statistical significance and reproducibility at larger scale: Most experiments use five seeds, and the paper does not report broader uncertainty analyses, hyperparameter sensitivity, or independent replications across datasets and architectures.
  • Use of ground-truth latents for evaluation: Synthetic metrics such as invariant and variant R2R^2, cross-block leakage, and block purity require access to ground-truth latent variables. Practical diagnostics for determining whether learned blocks are correct when such latents are unavailable remain undeveloped.
  • Multiple invariant correspondence structures: The framework considers one designated invariant block and one correspondence distribution. It remains open how to recover hierarchical, overlapping, multimodal, or more than two factor groups from several correspondence types.
  • Causal interpretation of invariant factors: The paper uses “invariant” in a correspondence-relative statistical sense, but does not establish that the recovered block corresponds to causal mechanisms or remains stable under interventions outside the observed pairing distribution.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Robust visual robot control — robotics and manufacturing. Deploy SplitJEPA-style encoders in behavior-cloning or model-predictive-control pipelines, using the invariant block r_H for task-relevant decisions and optionally adding the variant block r_L when execution-specific information is useful. This can improve manipulation under camera, lighting, appearance, and viewpoint changes, as demonstrated for PickCube and PushCube. Dependencies: reliable approximate correspondences between trajectories or views; sufficient variation across variant factors; calibration of the split dimensions; validation that a change is genuinely task-irrelevant rather than safety-critical.
  • Simulation-to-real transfer for embodied AI. Train representations on simulated trajectories with matched task states but varied rendering conditions, then transfer policies to real robots. The reconstruction-free objective can reduce dependence on simulator-specific pixels and preserve the physical task content in r_H. Dependencies: simulation must vary the same factors that change in deployment; simulated correspondences must preserve invariant task state; real-world dynamics and sensing must remain within the learned support.
  • Camera- and illumination-robust perception modules. Use r_H as an input to object localization, grasp selection, contact prediction, or navigation components that should remain stable under nuisance visual changes. Use r_L separately for viewpoint estimation, appearance modeling, or sensor adaptation. Dependencies: the definition of “invariant” is task-relative. A camera change that is irrelevant for grasping may be essential for viewpoint estimation, so the same representation should not automatically be used for every downstream task.
  • Efficient self-supervised representation learning — computer vision and video. Replace or supplement autoencoder-based pipelines with reconstruction-free predictive training. The method can reduce decoder computation and avoid allocating representation capacity to pixel-level details that are not needed for prediction or control. A practical workflow is:

    1. collect predictive temporal pairs;
    2. collect matched invariant pairs;
    3. train a JEPA-style encoder with predictive, invariant-consistency, and variance-preserving losses;
    4. whiten or normalize the resulting representation;
    5. expose r_H and r_L as separate features. Dependencies: the paper’s formal guarantee assumes stationary Gaussian predictive dynamics, centered and whitened representations, and population-level optimization. Real deployments require empirical monitoring of approximation error and representation collapse.

  • Robust offline behavior cloning. Train separate policies using r_H and r_H + r_L, selecting between them according to the expected deployment shift. The experiments indicate that r_H can retain much of the information needed for action prediction, while including r_L may help in conditions where execution-specific details matter. Dependencies: invariant-only policies may discard information needed for grasping, fine manipulation, or collision avoidance. The split should therefore be evaluated under both in-distribution and anticipated out-of-distribution conditions.

  • Data-efficient representation diagnostics for machine-learning researchers. Use cross-block leakage metrics such as ∥A12∥F\lVert A_{12}\rVert_F and ∥A21∥F\lVert A_{21}\rVert_F, invariant and variant R2R^2, block purity, and covariance error to evaluate whether a learned representation is genuinely organized rather than merely predictive. This provides a practical evaluation suite for world models and self-supervised encoders. Dependencies: ground-truth latent variables are not normally available, so real applications need proxy tests, intervention-based evaluations, or held-out correspondence data.
  • Dataset and augmentation design tools. Construct matched pairs that preserve task content while varying appearance, camera pose, lighting, robot configuration, or execution trajectory. The paper shows that shuffled or insufficiently diverse pairs undermine block recovery, whereas complementary changes can collectively span the variant subspace. Dependencies: pairing quality is crucial. Incorrectly matched observations can force genuinely task-relevant information into the invariant block; narrow augmentations can leave some variant directions indistinguishable from invariant content.
  • Policy and perception robustness testing. Treat the learned variant factors as a structured perturbation space for stress testing. Generate changes in lighting, viewpoint, appearance, or execution style while holding the invariant state fixed, then measure whether predictions based on r_H remain stable. Dependencies: perturbations must stay within the modeled latent support; the guarantee does not cover changes that also alter the shared task state.
  • Academic research infrastructure for reconstruction-free identifiability. Implement SplitJEPA as a baseline for research on nonlinear representation learning, multimodal learning, temporal prediction, and world models. It offers a controlled way to compare three sources of information: predictive relations, invariant correspondences, and normalization. Dependencies: the theoretical result concerns subspace recovery up to independent within-block orthogonal transformations, not recovery of individual semantic coordinates.
  • Early-stage policy and safety assessment for invariant AI systems. Use the framework to distinguish nuisance variation from operationally important variation in embodied systems. Regulators, auditors, or safety teams could require evaluations showing that models are insensitive to specified nuisance factors while remaining sensitive to hazards and task-state changes. Dependencies: the invariant/variant designation must be explicitly documented for each application; incorrect designations can produce unsafe invariances.

Long-Term Applications

  • Large-scale visual world models with structured latent states. Extend SplitJEPA to long videos, multi-camera streams, language-conditioned tasks, and heterogeneous robot fleets. A world model could maintain a stable task or object-state block while separately representing camera, style, embodiment, and execution variation. This may support planning across environments without reconstructing every observation. Dependencies: scaling the theory beyond the paper’s stationary Gaussian setting; handling nonstationarity, partial observability, discrete events, long-horizon dependencies, and imperfect correspondences.
  • Cross-robot transfer and embodiment-invariant control. Learn an invariant block representing object state, task goals, and contact relationships while isolating robot-specific morphology, joint configuration, and actuator behavior in the variant block. Policies could then transfer across robot arms or mobile manipulators. Dependencies: correspondence construction must preserve the same physical task state across embodiments; robot morphology may sometimes be causally relevant and cannot always be treated as a nuisance factor.
  • Multi-sensor and multimodal fusion. Apply the method to synchronized camera, depth, tactile, audio, force, and proprioceptive observations. Shared physical or semantic content could be represented in r_H, while modality-specific and sensor-condition information could be represented in r_L. Dependencies: modalities must be temporally aligned or paired through a reliable correspondence relation; modality-specific signals may contain essential information and should not be removed from all downstream computations.
  • Healthcare monitoring under domain and device shifts. In principle, use matched measurements from different hospitals, devices, acquisition protocols, or patient views to separate clinically relevant patient-state factors from scanner, site, and measurement-style variation. The invariant block could support risk prediction, while the variant block could retain information for calibration or device correction. Dependencies: patient identity or clinical state must be aligned across pairs; privacy, consent, distribution shift, and clinical validation are substantial requirements. Device variation must not be incorrectly labeled irrelevant when it affects diagnosis.
  • Financial forecasting and risk modeling. Separate stable economic or borrower-state factors from market-regime, institution, geographic, or reporting-style variation. A predictor based on an invariant block could be more robust to changes in data collection or market presentation, while r_L could support regime-sensitive strategies. Dependencies: financial environments are strongly nonstationary and rarely satisfy the paper’s stationary assumptions; invariant pairs must be defined carefully because regime changes may alter the causal relationship itself.
  • Personalized and domain-robust education systems. Represent relatively stable learning concepts, misconceptions, or mastery states separately from student-specific style, interface, language, and presentation variation. Educational assessments could use the invariant block for knowledge estimation and the variant block for personalization. Dependencies: constructing pairs with identical underlying knowledge but different presentation conditions is difficult; fairness requires checking that demographic or accessibility-related factors are not inadvertently discarded.
  • Energy and industrial monitoring. Learn invariant equipment-health representations across sensors, weather, operating loads, and sites, while retaining variant information for forecasting demand or diagnosing environmental effects. This could support predictive maintenance and transfer of models between plants. Dependencies: operating conditions can be causally related to failures, so “variant” does not necessarily mean irrelevant. Coverage of all meaningful change directions and reliable temporal correspondence are required.
  • Human–robot collaboration and adaptive interfaces. Use r_H for shared task intent and object state, and r_L for user-specific motion style, viewpoint, speed, or embodiment. Robots could maintain consistent task behavior while adapting interaction style to different users and environments. Dependencies: human behavior is highly nonstationary; invariant correspondences must not suppress safety-critical cues such as hesitation, fatigue, or unexpected motion.
  • Counterfactual and intervention-based simulation. Once invariant and variant subspaces are estimated, systems could vary r_L while holding r_H fixed to generate structured counterfactual tests—for example, testing whether a policy changes when only lighting or viewpoint changes. This could support causal debugging and robustness certification. Dependencies: the learned representation is not automatically a causal model. Counterfactual validity requires that interventions in the latent blocks correspond to physically meaningful changes and remain within the data-generating support.
  • Structured foundation-model interfaces. Future multimodal foundation models could expose separate latent channels for stable world state and context-dependent realization. Planning, memory, retrieval, and control modules could selectively consume r_H, r_L, or both, reducing unwanted sensitivity to superficial changes. Dependencies: large-scale training may violate the assumed fixed dimensionality, Gaussianity, and exact whitening; semantic interpretation of the blocks requires additional supervision or intervention-based validation.
  • Formal robustness and certification workflows. The approximate-consistency bound suggests a route toward quantitative certification: if the invariant-pair error is ε\varepsilon and the variation covariance has minimum eigenvalue μ\mu, residual cross-block leakage is bounded by approximately ε/μ\sqrt{\varepsilon/\mu}. This could become a practical audit metric for robustness claims. Dependencies: estimating μ\mu reliably requires sufficiently diverse correspondence data; finite-sample error, encoder optimization failures, and distribution shift can make the population bound optimistic.
  • Automated correspondence discovery. A future system could discover or refine invariant pairs from demonstrations, video, simulator metadata, language instructions, or known interventions instead of requiring manually paired trajectories. This would make SplitJEPA applicable to large uncurated datasets. Dependencies: automatically inferred pairs may encode spurious invariances. Pair discovery should be coupled with uncertainty estimation, human review, or downstream intervention tests before deployment.
View Paper Prompt  View All Prompts 

Glossary

  • Ablation: An experiment that removes or modifies a component to measure its contribution. “Figure 1 distinguishes the roles of prediction, correspondence and normalization.”
  • Block identifiability: The ability to recover groups of latent variables, while allowing transformations within each group. “We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries”
  • Block purity: A measure of how exclusively learned representation dimensions correspond to their intended latent blocks. “We report Global R2R^2 , Invariant and Variant R2R^2 , cross-block leakage ∥A12∥F\| A_{12} \|_F and ∥A21∥F\| A_{21} \|_F , and block purity over five seeds.”
  • Block-wise isometry: A distance-preserving transformation applied independently within separate representation blocks. “Our work thereby recovers both invariant and variant subspaces within the complete predictive representation, up to independent block-wise isometries and without an observation decoder.”
  • Centered representation: A representation whose expected value is zero. “For centered and whitened population solutions, this takes the form g(z)=Qzg(z)=Qz, where Q⊤Q=IdQ^{\top}Q=I_{d}.”
  • Collapse: A degenerate representation in which inputs map to identical or insufficiently varying outputs. “The variance regularizer discourages collapse, while $\mathcal{L}_{\text{inv}$ encourages agreement of the designated invariant block across matched pairs.”
  • Contrastive learning: Representation learning that uses relationships between related and unrelated examples to shape the learned features. “Contrastive objectives can reduce nonlinear latent ambiguities by exploiting relationships between observations under asymmetry assumptions”
  • Cross-block leakage: Unintended information from one latent block appearing in another learned block. “Removing variance regularization and ZCA normalization increases covariance error and makes complementary-block recovery less stable.”
  • Diffeomorphism: A smooth, invertible mapping whose inverse is also smooth. “so $V_{\text{leak}$ is a global smooth diffeomorphism.”
  • Disentanglement: The separation of distinct underlying factors into distinct representation components. “This objective has inspired invariant representation learning from multiview and temporal robotic data”
  • Downstream predictor: A model that uses a learned representation for a subsequent task. “Under the block identifiability established in Theorem 2, suppose a differentiable downstream predictor depends solely on the invariant representation rHr_{H} .”
  • Full-rank variation: Variation whose second-moment matrix has no zero eigenvalues, meaning every direction changes sufficiently. “We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries”
  • Gaussian predictive dynamics: A temporal or relational model in which latent states and their predictive noise follow Gaussian distributions. “Under stationary Gaussian predictive dynamics and appropriate representation normalization, it identifies the complete latent process”
  • Global orthogonal transformation: A rotation or reflection applied to the entire representation that preserves Euclidean geometry. “prediction first reduces the ambiguity to a global orthogonal transformation, and variation across correspondences identifies its block partition.”
  • Gradient isolation: The property that changes in one latent block produce no gradient effect on a computation based on another block. “Theorem 3 (Gradient Isolation).”
  • Identifiable generative model: A generative model whose latent factors can be recovered, subject to specified allowable transformations. “Identifiable generative models address this ambiguity by introducing additional supervision”
  • Invariant correspondence: A paired observation relationship in which selected latent factors are shared while others may vary. “The invariance is specific to this correspondence: zHz_{H} is shared within an invariant pair but may change across predictive pairs.”
  • Invariant representation: A representation component that remains unchanged under specified variations. “When a downstream computation uses only the invariant representation, changes confined to the variant latent factor have no path through its representation.”
  • Isotropic: Having equal variance in every direction, typically represented by an identity covariance matrix. “Orthogonal transformations preserve distances, covariance, and the isotropic predictive structure.”
  • JEPA (Joint-Embedding Predictive Architecture): A self-supervised architecture that predicts one embedding from another without reconstructing the original observation. “Joint-Embedding Predictive Architectures (JEPAs) learn informative representations by making predictions directly in representation space without reconstruction”
  • Latent ambiguity: Uncertainty about which latent parameterization or factor organization generated the observations. “This result excludes the nonlinear mixing in Theorem 1, but a global rotation may still mix the two latent blocks.”
  • Latent factor: An unobserved variable representing an underlying source of variation in observed data. “The ground-truth latent state consists of invariant and variant blocks that jointly generate the observations.”
  • Latent subspace: A vector space spanned by a group of latent variables or representation directions. “Hence, the invariant and variant latent subspaces are identified up to independent within-block orthogonal transformations without reconstruction.”
  • LeJEPA: A reconstruction-free predictive representation-learning method with theoretical latent-recovery guarantees under specified conditions. “LeJEPA provides a complementary result in this direction.”
  • Nonlinear ICA: Independent component analysis for recovering latent sources when the observation-generating process is nonlinear. “This perspective also appears in nonlinear ICA, where contrastive and predictive objectives can achieve component-wise identifiability”
  • Nonlinear reparameterization: A nonlinear change of coordinates in latent space that can preserve observations while altering the apparent factor structure. “Indeed, an invertible nonlinear reparameterization may preserve the complete observation distribution while mixing the underlying latent factors”
  • Observation decoder: A function that maps a latent representation back into the observation space. “Since the guarantee needs no decoder, the result extends reconstruction-free latent recovery to invariant-variant block identification.”
  • Orthogonal transformation: A linear transformation whose transpose is its inverse and that preserves lengths and angles. “For any orthogonal matrix R∈Rd×dR\in\mathbb{R}^{d\times d}, the representation g~(z)=Rg(z)\tilde{g}(z)=Rg(z) belongs to the same equivalence class.”
  • Population minimizer: A representation or model that achieves the lowest possible expected loss under the data distribution. “let g⋆g^{\star} be a global population minimizer of $\mathcal{L}_{\mathrm{pred}$”
  • Predictive alignment: Learning representations so that related observations have appropriately related embeddings. “The variance regularizer discourages collapse, while $\mathcal{L}_{\text{inv}$ encourages agreement of the designated invariant block across matched pairs.”
  • Pushforward distribution: A probability distribution obtained by transforming random variables through a specified mapping. “the pairs (f,Pz)(f,P_{z}) and $(\widetilde{f},P_{\widetilde{z})$ induce the same observed marginal distribution p(h)p(h).”
  • Representation normalization: Constraints imposed on a representation’s mean and covariance to fix its geometry. “Under stationary Gaussian predictive dynamics and appropriate representation normalization, it identifies the complete latent process”
  • Reconstruction-free learning: Learning representations without decoding them to reproduce the original observations. “Reconstruction-free learning offers a promising foundation.”
  • Smooth embedding: A differentiable mapping that preserves the structure of a latent space within the observation space. “we take ff to be a smooth embedding on the connected latent support”
  • Stationary dynamics: Dynamics whose statistical properties do not change over time. “We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition”
  • Sufficient variation: A condition ensuring that available paired changes cover every direction of the variant latent space. “Assumption 1 (Sufficient Variation).”
  • Whitening: Transforming variables so that they have zero mean and identity covariance. “Finite-weight variance regularization encourages this normalized geometry but does not by itself impose exact whitening.”
  • ZCA normalization: A whitening procedure that preserves the orientation of the original features as much as possible. “Removing variance regularization and ZCA normalization increases the representation covariance error.”
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Tweets

Sign up for free to view the 5 tweets with 309 likes about this paper.

Sign Up for Free