RoboJEPA: Scaling Robotic Latent World Models

Abstract: Latent world models have shown a remarkable ability to predict future states and to plan in the real world. In practice, however, we lack a principled way to estimate how their capabilities scale with model size, data, and compute, an open problem that slows progress in the field. In this work we present RoboJEPA, a world model based on the Joint Embedding Predictive Architecture (JEPA) and trained on a large-scale dataset spanning 12 robotic embodiments. We show that RoboJEPA's imagination error, the error of its latent rollouts, follows a second-order power law in compute, allowing us to predict model quality well beyond the scale at which the law is fit. We further show that downstream robotic planning performance improves predictably with compute, and that imagination error is strongly correlated with it, making it a reliable proxy for real-robot evaluation. Finally, we demonstrate that latent world models can be deployed zero-shot as robotic agents, planning toward a single goal image to solve tasks requiring long-horizon planning on real hardware. We release all model checkpoints together with our training and robot deployment code. To our knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data, and RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date.

• The paper introduces RoboJEPA, a family of latent world models ranging from 22M to 8B parameters, revealing that held-out latent imagination error follows a second-order power law in training compute, and significantly improving real-world robotic tasks like Grasp, Object Lift, and Pick and Place.
• RoboJEPA models conditions the prediction on observations, proprioception, and actions while supporting receding-horizon planning to a single goal image without needing task-specific policy training, and the trained models show low failure rates during multi-step manipulation tasks, demonstrating that a single goal image is sufficient for various deployment scenarios.
• The study concludes that training longer horizons in addition to increasing model size improves non-greedy tasks involving dynamic manipulation, but mostly unobjective.
• follow-up-questions

Research objective and contribution

“RoboJEPA: Scaling Robotic Latent World Models” (2610.10515) investigates whether action-conditioned robotic world models exhibit predictable scaling with model size and training compute. The paper addresses a methodological problem: although scaling laws are established for language, vision, and generative models, it has remained unclear whether they apply to robotic dynamics models trained on heterogeneous, real-world interaction data. This uncertainty makes it difficult to determine whether additional resources should be allocated to predictor capacity, training duration, data diversity, or longer-horizon supervision.

The paper introduces RoboJEPA, a family of latent world models ranging from 22M to 8B predictor parameters. RoboJEPA predicts future V-JEPA 2.1 representations rather than pixels, conditions on observations, proprioception, and actions, and supports receding-horizon planning toward a single goal image. The central empirical claims are:

• held-out latent imagination error follows a second-order power law in training compute;
• this offline error predicts downstream planning performance;
• long-horizon training changes the scaling frontier for non-greedy manipulation tasks;
• model scale improves zero-shot planning on a real Franka robot;
• a single goal image is sufficient for solving multi-step manipulation tasks without task-specific policy training.

The experiments span more than 50,000 simulated and real-robot evaluation episodes. The training corpus contains approximately 2.87M trajectories, 15,022 hours of video, 6,692 hours with synchronized action data, 12 robotic platforms, and 26 action-space or embodiment identifiers.

RoboJEPA architecture and training objective

RoboJEPA uses a frozen V-JEPA 2.1-G visual encoder with a 1,664-dimensional latent representation. The predictor is a causal ViT-style transformer that receives a sequence of visual tokens, action tokens, and proprioceptive-state tokens. It predicts the next latent frame in a single forward pass. The predictor is autoregressive only across time: predicted features are fed back into the model during rollout, while spatial patches and camera views are jointly processed within each timestep.

The architecture is designed to absorb substantial heterogeneity in the training corpus. It supports variable numbers of camera views, different resolutions, heterogeneous action spaces, and multiple robot embodiments. Factorized four-axis RoPE encodes time, spatial coordinates, and camera view. Per-view biases distinguish semantic camera roles such as external, wrist-mounted, head-mounted, or top-down cameras. Per-embodiment input projections map different action and state conventions into a shared token space. A local causal attention window of eight timesteps bounds the computational cost of autoregressive prediction and permits a rolling KV cache during planning.

Training combines teacher-forced next-frame prediction with autoregressive rollout prediction. The teacher-forced term trains the predictor against encoder features given the true preceding frames. The rollout term supplies only a prefix of encoded observations and requires the model to generate subsequent latent states from its own predictions. Both terms use a per-token $\ell_1$ loss. This combination is intended to reduce exposure bias and to align training more closely with deployment, where the planner repeatedly consumes model-generated states.

The key design choice is to hold the visual encoder fixed while scaling only the predictor. Consequently, the reported scaling laws characterize the dynamics-model component conditional on a particular representation space. They do not establish joint scaling laws for the encoder and predictor.

The data mixture is unusually broad for a robotic world-model study. It includes DROID, Roboset, RoboMind, LeRobot, RoboCasa365, AgiBot World, 1X data, and nine datasets from Open-X Embodiment. The mixture combines single-arm, dual-arm, humanoid, simulated, and multiple action-space configurations. This breadth is important because the study aims to measure scaling of a shared, multi-embodiment predictor rather than a model specialized to one robot and one controller.

Latent imagination and qualitative scaling

RoboJEPA predicts in a frozen representation space, but the authors train a separate diffusion decoder to visualize latent rollouts in pixel space. The decoder is not used for planning and receives no gradient from the world model. Its role is diagnostic: decoded sequences reveal whether improvements in latent prediction correspond to more faithful physical and visual imagination.

At larger scales, decoded rollouts preserve object identity, gripper-object contact, structured deformation, and cross-view consistency for longer horizons. These improvements are not limited to static appearance. They concern the temporal details that determine whether a model can support planning through manipulation events.

Figure 1: Decoded AgiBot imagination improves with predictor scale, with larger models preserving dynamics, contact structure, and object detail more accurately.

The two-view experiments are particularly relevant because view inconsistency can undermine image-goal planning even when each individual view appears plausible. The larger predictors maintain mutual consistency between external and wrist-mounted cameras deeper into the rollout. This indicates that multi-view conditioning is not merely a perceptual convenience: it supplies a constraint on the inferred scene dynamics.

Figure 2: Two-view RoboCasa imagination remains mutually consistent for longer horizons as predictor scale increases.

The authors also demonstrate a high-resolution variant by fine-tuning the 8B predictor on three DROID camera views at $720 \times 1280$ resolution. This experiment is primarily qualitative and does not establish a separate high-resolution scaling law, but it shows that the latent-prediction formulation remains computationally practical at substantially larger token counts than the main scaling experiments.

Figure 3: High-resolution 720p imagination from the 8B RoboJEPA model across multiple DROID camera views.

Scaling laws for latent imagination

The main quantitative contribution is the characterization of held-out forward-prediction error as a function of total training compute. The evaluation metric is the same latent $\ell_1$ objective used for training and planning: the mean error over a nine-step rollout on unseen scenes. The authors evaluate both a real-robot DROID holdout and held-out RoboCasa scenes.

They compare four functional forms and fit them using the smaller predictors, from 22M through 2B parameters. The 4B and 8B models are held out for extrapolation. The best-performing form is a second-order power law:

$L(C) = E + A C^{\alpha-\gamma \ln C},$

where $C$ is training compute and $E$ is the fitted irreducible error. Unlike a conventional power law, the effective exponent changes with compute, allowing the log-log slope to vary over the observed range.

The extrapolation results are strong relative to the competing parameterizations:

Scaling form	DROID extrapolation error	RoboCasa extrapolation error
Standard power law	$2.0 \times 10^{-3}$	$3.7 \times 10^{-3}$
Second-order power law	$0.6 \times 10^{-3}$	$1.4 \times 10^{-3}$
BNSL	$720 \times 1280$0	$720 \times 1280$1
UNSL	$720 \times 1280$2	$720 \times 1280$3

The fits are trained on the 22M–2B regime but evaluated on 4B and 8B checkpoints. The second-order law therefore provides evidence of extrapolation rather than merely interpolation. Its advantage over the standard power law is approximately two- to threefold in the reported held-out extrapolation error.

Figure 4: Held-out latent rollout error on DROID and RoboCasa follows a second-order compute scaling law across model sizes and training durations.

The fitted irreducible errors are approximately $720 \times 1280$4 on DROID and $720 \times 1280$5 on RoboCasa, with similar floors estimated by the second-order and broken neural scaling laws. This result has a direct interpretation: under the current fixed encoder, data mixture, training recipe, and evaluation protocol, the predictor is approaching saturation. Increasing predictor size alone is unlikely to remove the remaining error. The paper therefore treats the apparent floor as evidence that additional and more diverse interaction data may become more valuable than further parameter scaling.

An important methodological detail is the use of the compute-optimal frontier. For each compute budget, the authors select the best-performing model size rather than fitting all checkpoints indiscriminately. This recognizes that no single model size is compute-optimal throughout training: small models can be preferable at low budgets, whereas larger models become advantageous at higher budgets.

From offline prediction to online planning

The paper tests whether latent imagination error is a useful proxy for robotic control rather than merely an offline representation metric. Planning uses the cross-entropy method over action sequences. A goal image is encoded once, candidate actions are rolled out through the world model, and candidate sequences are ranked according to the final latent distance to the goal. The planner executes actions in receding-horizon fashion and replans from new observations.

This formulation is task-agnostic at deployment. There is no task-specific reward model, language instruction, subgoal sequence, or learned policy proposal. Candidate actions are sampled from a parameterized distribution and prioritized using the predictor. The same planning hyperparameters are largely retained across tasks and model sizes.

For greedy RoboCasa tasks, downstream performance scales predictably with compute. Free-space end-effector reaching begins to emerge at approximately $720 \times 1280$6 FLOPs, while reaching while holding an object emerges around $720 \times 1280$7 FLOPs. These capabilities saturate relatively early, reflecting the limited complexity of the tasks.

Figure 5: RoboCasa pose-reaching performance improves with training compute on unseen kitchens, with held-object reaching requiring a higher compute threshold.

The implication is operationally important: held-out latent prediction loss can serve as a lower-cost model-selection metric for capabilities that would otherwise require extensive online robotic evaluation. This claim is strongest for the greedy tasks, where the planner can make progress through short-horizon actions and does not require detailed long-range counterfactual reasoning.

Long-horizon training and non-greedy planning

The initial short-horizon recipe does not produce a clean scaling frontier on non-greedy tasks. In obstacle-reach experiments, smaller models sometimes outperform larger ones, and downstream reward correlates only weakly with compute. The authors attribute this failure to a train–test mismatch: standard training uses only a two-step autoregressive rollout, whereas obstacle avoidance and object pushing require accurate multi-step prediction.

To address this problem, the authors introduce a long-horizon cooldown with ten-step autoregressive rollouts and mixed low- and high-resolution inputs. Rather than retraining every model from scratch, they resume from flat-learning-rate checkpoints and allocate the additional compute to the final cooldown phase. This design isolates the value of long-horizon supervision while avoiding the cost of repeating the entire pretraining run.

Figure 6: Long-horizon prediction training becomes more compute-efficient at higher budgets and reaches a lower imagination-error floor than short-horizon training.

The long-horizon recipe overtakes the short-horizon recipe at approximately $720 \times 1280$8 FLOPs and reaches a substantially lower fitted floor. The improvement is not simply a constant offset: the scaling frontier itself changes. This supports the paper’s claim that training objective and rollout horizon are part of the scaling problem, rather than fixed implementation details.

The corresponding obstacle-reach results show a much cleaner relationship between compute and performance. Models below approximately 300M parameters remain largely unable to route around the obstacle. Obstacle avoidance begins to improve around $720 \times 1280$9 FLOPs and saturates near $\ell_1$0 FLOPs. Object pushing is more demanding: positive success appears only around $\ell_1$1 FLOPs.

The authors identify an ordered emergence of capabilities:

1. free-space end-effector control;
2. moving while maintaining a grasp;
3. reasoning about static scene geometry;
4. modeling object motion under contact.

This ordering is consistent with increasing requirements for spatial representation, gripper-state control, collision avoidance, and contact-rich dynamics. The result also qualifies the broader offline-to-online claim: short-horizon latent error predicts simple planning performance, but long-horizon capabilities require training explicitly matched to deployment horizons.

Figure 7: On the DROID holdout, long-horizon training eventually surpasses short-horizon training and attains a lower irreducible error.

Figure 8: The same crossover between short- and long-horizon training appears on RoboCasa.

Real-robot deployment

RoboJEPA is deployed on a DROID Franka platform with an external side camera and a wrist-mounted camera. The planner receives a single final goal image and must solve Grasp, Object Lift, and Pick and Place tasks. The models are pretrained on the heterogeneous mixture and receive no task-specific fine-tuning.

The real-robot results show a generally increasing success trend with model size. For the largest models, RoboJEPA-4B achieves 60% success on Grasp, 30% on Object Lift, and 21% on Pick and Place. RoboJEPA-8B achieves 67%, 50%, and 27%, respectively. Progress scores also increase from 4B to 8B on Grasp and Object Lift.

Model	Grasp	Object Lift	Pick and Place
RoboJEPA-22M	30%	2%	17%
RoboJEPA-300M	37%	18%	17%
RoboJEPA-1B	50%	24%	10%
RoboJEPA-2B	43%	40%	31%
RoboJEPA-4B	60%	30%	21%
RoboJEPA-8B	67%	50%	27%

The monotonic relationship is not exact for every task and model size. For example, Pick and Place performance fluctuates, with RoboJEPA-2B outperforming the 4B model. Nevertheless, the aggregate trend is positive, and the 8B model provides the strongest Grasp and Object Lift results.

Figure 9: Real-world Franka Object Lift is specified by a single goal image observed from external and wrist-mounted cameras.

The comparison with $\ell_1$2-FAST and $\ell_1$3 should be interpreted cautiously. The baselines use text instructions, different controller interfaces, and DROID fine-tuning, whereas RoboJEPA uses a single image goal and pretraining on a broader mixture that includes DROID. The results are therefore contextual rather than controlled comparisons. The paper itself notes that $\ell_1$4 performs well on Pick and Place but achieves 0% on Object Lift, possibly because the DROID data contains many more pick-and-place examples than lift examples. RoboJEPA’s image-goal formulation avoids dependence on task-word frequency, but this does not establish superiority under matched training and evaluation conditions.

The qualitative analysis identifies a behavioral transition with scale. Smaller predictors can produce smooth approaches but often fail to maintain gripper closure or recover from an unfavorable grasp. Larger predictors more often preserve contact, correct mistakes, and complete the manipulation sequence. This observation links the latent scaling results to concrete control behavior, although the qualitative conclusions are not independently quantified beyond the task success rates.

Limitations and open questions

The most important limitation is that the scaling study holds the V-JEPA 2.1 encoder fixed. The reported laws therefore describe predictor scaling conditional on a particular latent representation. They do not show whether jointly scaling representation learning and dynamics prediction would yield the same functional form, a lower error floor, or different compute-optimal allocations.

The data regime also limits interpretation of the asymptotic behavior. The models are trained for multiple epochs over a fixed corpus, and the fits approach apparent saturation. The observed irreducible floor may therefore reflect data coverage, encoder information loss, action-space heterogeneity, or optimization limitations rather than an intrinsic limit of latent world modeling.

The evaluation distribution is broad in embodiment but narrower in task structure than the raw dataset mixture might suggest. The principal quantitative holdouts are DROID and RoboCasa, and the real-world experiments use a single Franka platform. Generalization to unseen robot embodiments, camera configurations, action conventions, and physical environments is not established by the reported results.

The planning objective is also constrained. RoboJEPA does not use language, learned action proposals, a task-specific value function, or intermediate subgoals. Uniform or Gaussian action sampling combined with CEM is computationally expensive, even though KV caching and AOT compilation make several-second planning feasible for the 8B model. The paper leaves open whether a learned proposal distribution would improve long-horizon planning or alter the observed scaling thresholds.

Finally, the real-robot baseline comparison is confounded by goal modality, data mixture, controller interface, and fine-tuning status. The results support a scaling trend within RoboJEPA, but they do not isolate the causal contribution of latent world modeling relative to alternative policy architectures.

Conclusion

RoboJEPA establishes an empirical scaling framework for multi-embodiment latent robotic world models. Across DROID and RoboCasa, held-out imagination error is well described by a second-order power law in training compute, with held-out extrapolation errors of $\ell_1$5 and $\ell_1$6 on 4B and 8B models. For short-horizon tasks, this offline metric correlates strongly with downstream planning performance and can reduce the need for expensive online evaluation.

The paper further shows that long-horizon planning requires long-horizon training: a ten-step autoregressive cooldown produces a lower error floor and restores predictable scaling on obstacle avoidance and object pushing. Real-robot deployment demonstrates increasing success with model size, including 67% Grasp and 50% Object Lift success for the 8B model under single-image-goal planning. The main unresolved issue is whether these laws persist when the visual representation, data distribution, and planning mechanism are scaled jointly rather than treating the frozen encoder and fixed corpus as constants.