---
title: 'RoboJEPA: Action-Conditioned Latent Model'
url: https://www.emergentmind.com/topics/robojepa
type: topic
---

# RoboJEPA: Action-Conditioned Latent Model

RoboJEPA is an action-conditioned latent world model for robotics that predicts future visual representations rather than reconstructing future RGB frames. It uses a frozen V-JEPA 2.1 visual encoder, a trainable multimodal transformer predictor, and cross-entropy-method (CEM) planning in latent space. Trained on approximately 2.8713 million episodes spanning 12 robotic platforms and 26 action-space or embodiment identifiers, RoboJEPA studies how latent imagination quality and downstream planning scale with model size and training compute. Its predictors range from approximately 22 million to 8 billion parameters, and its principal empirical result is a second-order power law relating multi-step latent prediction error to compute. At deployment, the model can plan toward a single goal image without task-specific policy fine-tuning [2610.10515].

## 1. Conceptual basis and development context

RoboJEPA belongs to the family of Joint-Embedding Predictive Architectures (JEPAs), in which a model predicts a target representation in latent space rather than reconstructing an observation in pixel space. For an observation $x_t$, action $a_t$, and visual encoder $f$, RoboJEPA forms a latent representation

$$
z_t=f(x_t),
$$

and trains a predictor $g_\theta$ to estimate a future representation,

$$
\hat z_{t+1}=g_\theta([z_{\leq t}],a_{\leq t},s_{\leq t}),
$$

where $s_t$ denotes proprioceptive state. During rollout, predicted features are recursively fed back into the predictor.

This design follows the broader JEPA principle that task-relevant, predictable structure can be modeled without reproducing texture, lighting, background appearance, and other pixel-level variation. The EB-JEPA library illustrates the progression from image representations to video prediction and action-conditioned latent world models. Its Two Rooms experiment uses latent prediction and model-predictive planning, reporting $97\pm2\%$ planning success with cumulative latent cost, although its experiments are restricted to CIFAR-10, Moving MNIST, and simulated navigation [2602.03604].

Subsequent robotics-oriented systems emphasize different aspects of this design space:

- SkyJEPA applies latent prediction to high-frequency quadrotor control and combines a latent dynamics model with a physics-inspired prober and MPPI [2606.23444].
- AquaJEPA combines RGB, forward-looking sonar, proprioception, explicit validity masks, EMA target encoding, modality dropout, and action-margin supervision for underwater navigation [2607.29393].
- JEPLO uses proprioceptive and LiDAR observations for mapping-free quadruped locomotion, with SIGReg and a concurrent JEPA teacher–student pipeline [2609.15770].
- H-JEPA introduces multiple action-conditioned latent spaces for hierarchical long-horizon planning [2610.06805].
- CF-JEPA factorizes latent states into controllable and uncontrollable subspaces to improve robustness against visual distractors [2610.00727].
- JEPA-WAM systems integrate latent future or stage representations with video-action models for manipulation, either through shared action–future predictors [2608.09381], stage-level semantic prediction [2608.10780], or generated visual goal references [2609.20277].

RoboJEPA differs from these systems primarily in scale, dataset breadth, explicit compute scaling analysis, and the use of a frozen V-JEPA 2.1 representation across multiple embodiments.

## 2. Architecture and latent prediction

RoboJEPA consists of a frozen V-JEPA 2.1 visual encoder, a trainable action-conditioned predictor, multimodal token projections, and a latent-space planner. The visual encoder is fixed throughout the main scaling study. It is not trained by gradient descent and is not updated using an exponential-moving-average target mechanism.

For each camera view $v$, V-JEPA 2.1 maps a short frame sequence to a spatial feature grid:

$$
z_t^{(v)}=f(x_t^{(v)}).
$$

At the standard $240\times320$ input resolution, the encoder produces a $15\times20$ spatial grid, with 300 visual tokens per view. Each token has feature dimension $D_e=1664$. The predictor projects these tokens into its internal width, processes them jointly with action and state tokens, and projects predicted visual tokens back to the encoder feature dimension.

At each time step, the predictor receives an interleaved sequence of visual, action, and proprioceptive tokens:

$$
\big(z_1^{(1:V)},a_1,s_1\big),\ldots,
\big(z_T^{(1:V)},a_T,s_T\big).
$$

Actions and proprioceptive states are separately projected into the transformer width. The model also receives an embodiment identifier and semantic camera identifiers. These conditioning signals are necessary because the training corpus combines heterogeneous actuator spaces, state conventions, camera configurations, and control rates.

Actions are padded to a shared maximum dimensionality of $d_a^{\max}=64$:

$$
\tilde a_t=\left[a_t;\mathbf 0_{d_a^{\max}-d_a}\right].
$$

An embodiment-specific projection then maps the padded action into the predictor representation:

$$
\operatorname{ActTok}(a_t,e)
=(W_a^{(e)})^\top\tilde a_t+b_a^{(e)}+\operatorname{PE}_t^a.
$$

The dataset contains 12 physical platforms but 26 learned embodiment or action-space identifiers because distinct controller and state conventions for the same platform are treated as separate interfaces.

The predictor uses a ViT-style transformer with pre-normalization, RMSNorm, GELU feed-forward blocks, grouped-query attention, QK normalization, factorized four-axis rotary positional embeddings, frame-level causal attention, and a local temporal window of eight frames. Rotary embeddings encode time, image row, image column, and camera view. A learned per-camera bias further distinguishes visual roles such as wrist, external, head-mounted, or left and right cameras.

The causal mask permits attention to current and previous frames but prevents access to future frames. Within the permitted temporal window, visual patches, camera views, action tokens, and state tokens can interact. The local window makes recurrent rollout cacheable and limits extrapolation to unbounded context lengths.

Unlike conventional masked-image JEPAs, RoboJEPA predicts future representations conditioned on actions and proprioception. Unlike pixel-space video models, it does not use RGB reconstruction as its world-model objective. A separately trained diffusion decoder can visualize latent rollouts, but it is not connected to the world model by gradients and is not used for planning.

## 3. Training corpus and multimodal normalization

RoboJEPA is trained on 23 public datasets comprising approximately 2.8713 million episodes, 15,022 hours of video, 6,692 hours with synchronized actions, 12 robotic platforms, and 26 embodiment or action-space identifiers. The mixture includes single-arm, dual-arm, humanoid, mobile-base, and simulated systems.

| Dataset group | Episodes | Video hours | Action hours |
|---|---:|---:|---:|
| DROID | 92k | 414 | 138 |
| Roboset | 73k | 1,244 | 380 |
| RoboMind | 35.3k | 276 | 98 |
| LeRobot | 21.5k | 182 | 93 |
| RoboCasa365 | 32k | 482 | 482 |
| 1X | 23.5k | 90 | 90 |
| AgiBot World | 167.5k | 8,036 | 2,734 |
| Open-X Embodiment | 2.4265M | 4,298 | 2,677 |

Open-X Embodiment contributes nine datasets, including RT-1, Bridge, Bridge V2, QT-Opt, MT-Opt, RoboNet, Language-Table, BC-Z, and FMB. The data include one-, two-, and three-camera recordings, source rates from approximately 1 Hz to 60 Hz, and action formats including Cartesian, joint-space, velocity, state-delta, and command representations.

Most action vectors describe state differences. A typical single-arm action contains three-dimensional translation, three-dimensional orientation change, and gripper change, producing a seven-dimensional vector. Dual-arm and humanoid actions concatenate corresponding components. High-rate controller streams may be concatenated into action chunks; an eight-command DROID chunk can produce up to $8\times8=64$ action dimensions.

Preprocessing standardizes the heterogeneous corpus. Videos are converted to a common on-disk representation, temporally decimated to dataset-specific world-model frequencies, and sampled into 16-frame clips. Short clips are zero-padded with a validity mask. Images are resized and cropped using the V-JEPA 2.1 recipe, generally at $240\times320$ resolution. ImageNet normalization is applied, while horizontal flips, auto-augmentation, motion shifts, and random erasing are disabled because they could distort robot geometry.

Quaternions are converted to Euler orientation representations, gripper values are normalized to a common $[0,1]$ closedness convention, and joint-only data may be converted to end-effector states through forward kinematics. Camera names are mapped into seven semantic camera categories. RoboJEPA does not force all platforms into a common physical action space; instead, it uses shared transformer dynamics together with embodiment-specific projections and explicit embodiment identifiers.

The supplied material does not provide a conventional unified train–validation–test episode table for every source dataset. The scaling evaluations use held-out DROID scenes and held-out RoboCasa target scenes, but exact dataset-level split counts are not specified.

## 4. Training objectives and rollout regimes

RoboJEPA uses both teacher-forced prediction and autoregressive rollout training. In teacher forcing, the predictor receives ground-truth encoded context features and predicts the next frozen V-JEPA feature. For view $v$,

$$
\mathcal L_{\mathrm{TF}}
=
\sum_{t,v}
\left\|
\hat z_{t+1}^{(v)}-z_{t+1}^{(v)}
\right\|_1.
$$

The implementation additionally applies frame-validity masks and normalization.

The autoregressive loss begins from a shorter observed prefix and requires the model to use its own predictions as subsequent inputs:

$$
\mathcal L_{\mathrm{AR}}
=
\sum_{t,v}
\left\|
\tilde z_{t+1}^{(v)}-z_{t+1}^{(v)}
\right\|_1.
$$

The combined objective is

$$
\mathcal L
=
\frac{1}{2}\mathcal L_{\mathrm{TF}}
+
\frac{1}{2}\mathcal L_{\mathrm{AR}}.
$$

The standard pretraining recipe uses an autoregressive rollout horizon of $K=2$. A long-horizon cooldown increases this to $K=10$ and adds higher-resolution inputs at both $240\times320$ and $480\times640$. Recurrent rollout inputs are detached from the autograd graph according to the reported training configuration.

This distinction is important because teacher-forced prediction and autonomous imagination measure different capabilities. Teacher forcing evaluates prediction from correct histories, whereas planning requires recursively generated histories. The long-horizon cooldown lowers multi-step imagination error and improves planning on obstacle-avoidance and pushing tasks.

Optimization uses AdamW with weight decay $0.1$, gradient-norm clipping at $1.0$, a peak learning rate of $5\times10^{-5}$, linear warmup, a long flat learning-rate phase, and linear cooldown. The AdamW coefficients are $(\beta_1,\beta_2)=(0.9,0.95)$.

RoboJEPA’s target representations are generated by the same frozen V-JEPA 2.1 encoder used for context representations. There is no separate online–target encoder pair, no EMA update, and no trainable target branch in the described recipe. This differs from systems such as AquaJEPA, which uses an EMA target encoder with momentum $0.99$ [2607.29393], and from SJEPA, which formalizes online and target encoders with stop-gradient and EMA updates [2608.04060].

The predictor sweep includes models of approximately 22M, 50M, 100M, 300M, 1B, 2B, 4B, and 8B parameters. Appendix configurations correspond approximately to 0.02B, 0.05B, 0.1B, 0.3B, 1.0B, 2.4B, 4.4B, and 8.4B parameters. The largest configuration has width 3072 and 72 layers.

## 5. Compute scaling and imagination error

RoboJEPA’s central empirical contribution is an analysis of how latent imagination error changes with training compute. Training compute spans approximately

$$
2\times10^{19}
\quad\text{to}\quad
9.5\times10^{22}
$$

FLOPs. Compute is estimated from measured per-step FLOPs multiplied by optimizer steps rather than solely through a rough parameter–token approximation.

The reported scaling relation is a second-order power law:

$$
\boxed{
L(C)=E+A\,C^{\alpha-\gamma\ln C}
}
$$

where $L(C)$ is imagination error, $C$ is training compute, $E$ is an asymptotic or irreducible error, and $A$, $\alpha$, and $\gamma$ are fitted parameters. In logarithmic form,

$$
\ln(L-E)=\ln A+\alpha\ln C-\gamma(\ln C)^2.
$$

Unlike a conventional power law with a constant log–log slope, the effective exponent varies with compute.

For the DROID holdout, the fitted parameters are

$$
E=0.218,\qquad
A=7.96\times10^{-19},\qquad
\alpha=1.91,\qquad
\gamma=0.02308.
$$

For the RoboCasa holdout, they are

$$
E=0.1709,\qquad
A=1.66\times10^{-17},\qquad
\alpha=1.761,\qquad
\gamma=0.02105.
$$

The fit uses 47 checkpoints from the 22M–2B models. The 4B and 8B models are held out for extrapolation. Mean absolute extrapolation errors, expressed in units of $10^{-3}$, are:

| Scaling curve | DROID | RoboCasa |
|---|---:|---:|
| Standard power law | 2.0 | 3.7 |
| Second-order power law | **0.6** | **1.4** |
| BNSL | 1.0 | 1.8 |
| UNSL | 1.4 | 5.7 |

The second-order law extrapolates approximately two to three times better than the ordinary power law on the reported held-out models. The fitted floors suggest near-saturation under the current representation, dataset mixture, and training recipe, but they do not establish a universal physical-world error floor.

Imagination error is the per-token latent $\ell_1$ distance between predicted and encoder features over a multi-step rollout:

$$
L=
\frac{1}{\text{number of evaluated tokens}}
\sum
\left\|
\hat z-z_{\mathrm{GT}}
\right\|_1.
$$

For scaling evaluation, an initial observation is encoded, the predictor rolls forward nine future steps using recorded actions, and corresponding future frames are encoded for comparison. The evaluations use unseen DROID scenes and held-out RoboCasa scenes with external and wrist-camera views.

The paper reports that imagination error decreases with more compute, larger compute-efficient models, longer rollout training, and higher-resolution training. Larger models preserve object identity, contact, gripper–object interaction, deformation, and cross-view consistency for longer periods. The supplied material does not provide a complete numerical horizon-by-horizon error table.

The results support a strong but bounded interpretation: within the fixed V-JEPA 2.1 representation and tested multi-embodiment training regime, offline multi-step latent error is a useful proxy for downstream planning quality. They do not establish a universal scaling law for all robotic world models, jointly scaled visual encoders, other datasets, or substantially different embodiments.

## 6. Latent-space planning and robotic deployment

At test time, RoboJEPA receives a current observation and a goal image. The goal is encoded once:

$$
z_g=f(x_g).
$$

For a candidate action sequence, the predictor generates a latent rollout, and CEM searches for a sequence minimizing final latent distance:

$$
a^\star_{t:t+H-1}
=
\arg\min_{a_{t:t+H-1}}
\left\|
\hat z_{t+H}(a_{t:t+H-1})-z_g
\right\|_1.
$$

The planner samples candidate action sequences, rolls them through the latent world model, retains elite candidates, refits the action distribution, executes a short action prefix, and replans. A rolling key–value cache avoids recomputing the observation history for every candidate.

For real Franka deployment, the reported configuration uses horizon $H=5$, two encoder steps of context, two executed actions per replan, 20 environment steps per episode, 4,096 samples per CEM iteration, 15 cold-start iterations, 10 warm iterations, and 20 elites. Translation and gripper are optimized while orientation is held fixed. The general planning description gives a five-centimeter translation bound, whereas the detailed DROID table reports a $0.15$ m planning bound.

RoboCasa uses horizon $H=1$ for Reach and ObjectReach and $H=5$ for ObstacleReach and Push, with 1,024 samples per iteration, 15 cold-start iterations, 10 warm iterations, and 20 elites. Closed-loop replanning occurs after every executed action.

The real-world platform is a Franka Panda with a parallel-jaw gripper, external left camera, and wrist-mounted camera. The reduced action space contains three-dimensional end-effector translation and one gripper command, with orientation fixed. Each task is specified by a single final goal image. No text instruction, intermediate subgoal, task-specific policy, or task-specific fine-tuning is used.

| Model | Grasp | Object lift | Pick and place |
|---|---:|---:|---:|
| RoboJEPA-4B | 60% | 30% | 21% |
| RoboJEPA-8B | **67%** | **50%** | **27%** |

Each model–task combination is evaluated over 30 episodes. Across the full model-size sweep, success is not perfectly monotonic for every task, but overall behavior improves with scale. The 22M model obtains 30% grasp, 2% lift, and 17% pick-and-place success; the 8B model obtains 67%, 50%, and 27%, respectively.

RoboCasa evaluation uses held-out kitchen scenes and a simulated mobile-base PandaOmron robot. Tasks include free-space reaching, obstacle avoidance, carrying a grasped banana, can, or mug, and pushing an object across a counter. ObjectReach begins with a scripted grasp controller, while ObjectPush keeps the gripper closed and requires non-prehensile contact manipulation.

The planning results suggest capability thresholds associated with increasing compute: free-space reaching emerges around $10^{20}$ FLOPs, carrying an object around $3\times10^{20}$ FLOPs, obstacle avoidance around $10^{21}$ FLOPs, and pushing around $10^{22}$ FLOPs. The supplied material does not provide exact correlation coefficients or confidence intervals linking imagination error to planning success.

The real-robot comparison with VLA systems is contextual rather than controlled. Reported results are:

| Model | Grasp | Lift | Pick and place |
|---|---:|---:|---:|
| $\pi_0$-FAST | 22% | 12% | 40% |
| $\pi_{0.5}$ | 5% | 0% | 53% |
| RoboJEPA-8B | **67%** | **50%** | 27% |

The systems differ in goal modality, training data, action interface, control frequency, and fine-tuning. RoboJEPA receives a goal image and uses a pretrained multi-embodiment world model, whereas the VLA systems receive text instructions and were fine-tuned on DROID. These results therefore do not isolate architectural superiority.

RoboJEPA’s principal limitations arise from fixed visual representations, finite data diversity, deterministic latent dynamics, CEM inference cost, dependence on goal images, and planner-specific hyperparameters. It does not explicitly model multiple possible futures, calibrated uncertainty, safety constraints, language-conditioned goals, or learned intermediate subgoals. Long-horizon predictions can still accumulate error, and latent $\ell_1$ accuracy does not guarantee physically correct contact dynamics.

The broader RoboJEPA research program is consequently defined by several unresolved directions: joint scaling of visual encoders and predictors, larger and more diverse interaction datasets, improved contact representations, language and multimodal goal specifications, uncertainty-aware planning, policy proposals to reduce CEM cost, transfer to unseen embodiments, and scaling laws that vary data quality, embodiment diversity, horizon, representation size, and test-time planning computation.

Source: https://www.emergentmind.com/topics/robojepa