Papers
Topics
Authors
Recent
Search
2000 character limit reached

XPACE: Joint World and Action Modeling from Heterogeneous Experience

Published 15 Sep 2026 in cs.RO | (2609.17372v1)

Abstract: A general-purpose robot needs to draw on diverse experience, choose actions, and anticipate how those actions will change the world. We introduce XPACE, a unified embodied world model that serves as both a world action model, jointly predicting executable robot actions and future video, and a world simulator, predicting the visual consequences of prescribed actions. Our key insight is that video prediction can both connect heterogeneous experience to action learning and generate new experience for policy improvement. With a shared video backbone between the policy and simulator, we use action-unlabeled video to learn visual dynamics and action-labeled human and robot demonstrations to jointly learn video and action prediction. Building on this architecture, a coarse-to-fine training curriculum progressively emphasizes robot control while retaining human experience, allowing the policy to learn behaviors beyond those covered by robot demonstrations. Beyond learning from recorded experience, XPACE uses its simulator to create additional recovery supervision for the policy. Specifically, we adapt the simulator to its own generated context, synthesize deviation-recovery trajectories around expert demonstrations, and fine-tune the policy on filtered recovery examples. Experiments on XPENG's IRON humanoid robot show that heterogeneous training improves robustness and enables transfer of human-observed skills to tasks absent from robot demonstrations, while recovery data generated by the model's own simulator further improves real-world task completion. Together, these results demonstrate how joint world and action modeling connects learning from heterogeneous experience with simulation-driven policy self-improvement.

Summary

  • The paper proposes XPACE, a unified model that leverages heterogeneous video and action data for embodied learning, offering a framework for joint learning of visual dynamics and executable actions.
  • Key findings include an 25 percentage point increase in task completion success rate, showcasing XPACE's effectiveness in leveraging diverse data sources for robot control tasks.
  • XPACE's self-improvement mechanism uses synthesized data to enhance policy refinement, demonstrating significant real-robot performance improvements in various manipulation tasks.

XPACE addresses a central problem in embodied learning: robot policies require executable action supervision, whereas the largest and most diverse sources of interaction experience are typically human videos without robot actions. The paper proposes a unified model that uses video prediction as the common interface between heterogeneous experience, policy learning, and simulation-based policy improvement. Its central claim is that a single shared predictive representation can support both action generation and action-conditioned visual simulation, provided that data composition, cross-embodiment action representations, and autoregressive training are designed jointly.

Problem formulation and central contribution

The framework has three coupled objectives. First, it learns visual dynamics from action-unlabeled egocentric video and action-conditioned behavior from human and robot demonstrations. Second, it predicts executable action chunks while simultaneously predicting future video. Third, it uses the learned simulator to synthesize off-trajectory recovery examples for policy fine-tuning.

This positioning distinguishes XPACE from systems that use video solely as a representation-learning source or expose a world model only as a planning interface. The paper instead treats prediction and control as mutually reinforcing: visual prediction supplies a training signal for broad experience, action supervision grounds the shared representation in embodiment-specific control, and the simulator generates corrective supervision that is absent from nominal demonstrations. The work is situated among unified video-action models such as UWM [(Wan et al., 26 Mar 2025)?] and UVA [(Wan et al., 26 Mar 2025)?], but its distinctive emphasis is the complete data-to-simulation-to-policy loop rather than joint prediction alone. The primary paper is “XPACE: Joint World and Action Modeling from Heterogeneous Experience” (2609.17372).

Heterogeneous experience and the data pyramid

XPACE organizes approximately 5,000 hours of embodied video into four layers, supplemented by a failure and recovery set used only for simulator training.

Figure 1

Figure 1: The XPACE data pyramid combines broad egocentric video, human action demonstrations, aligned bridge data, and target-robot teleoperation; failure and recovery data are reserved for simulation.

The first layer consists of in-the-wild egocentric video without action labels. Its role is to expose the video backbone to diverse scenes, object interactions, and visual transitions. The second layer contains human videos paired with hand and wrist poses, supplying action supervision across manipulation behaviors that may not occur in robot teleoperation. The third layer is bridge data selected for task or visual alignment with the target robot domain. Task-aligned clips share manipulation families with the robot corpus, whereas appearance-aligned clips approximate the deployment viewpoint, background, and object arrangement. The fourth layer consists of IRON humanoid-robot teleoperation, which provides native executable actions and therefore grounds the policy in the target embodiment.

The division is technically consequential. Human video cannot simply be treated as robot data because its action representation, viewpoint, kinematics, and task coverage differ from those of the deployment platform. XPACE therefore uses human experience for both video prediction and action supervision where kinematic correspondence permits it, while retaining robot demonstrations as the direct source of target-embodiment control. The paper does not introduce an explicit feature-alignment loss between human and robot data; instead, cross-embodiment alignment is induced through overlapping task and appearance distributions, a shared action objective, and continued mixture training.

The failure set contains unsuccessful, low-progress, and recovery trajectories. These data are excluded from policy imitation but used to teach the simulator that prescribed motions can produce unsuccessful visual outcomes. This design avoids training the simulator exclusively on successful behavior, which would bias it toward predicting nominal outcomes even under incorrect controls.

Language conditioning is also multi-level. Each clip may be associated with an episode-level instruction, a dense description, an atomic action command, and paraphrased task instructions.

Figure 2

Figure 2

Figure 2: Caption sampling shifts from dense clip descriptions toward deployment-like task instructions during training.

This hierarchy addresses the mismatch between clip-level supervision and episode-level deployment commands. Dense descriptions are useful during video-model adaptation, while original and paraphrased task instructions become more prevalent during policy-oriented training. The paper therefore treats language not merely as an input modality but as a mechanism for reconciling different temporal scopes of supervision.

Shared world-action architecture

XPACE uses an asymmetric mixture-of-transformers architecture. A causal video Transformer predicts future video latents, and an action Transformer reads multi-level visual features from the video backbone to predict executable action chunks.

Figure 3

Figure 3: The shared causal video backbone supports both task-conditioned action prediction and skeleton-conditioned visual simulation; the simulator subsequently supplies recovery data for policy refinement.

The architecture has two operating modes. In simulation mode, the video Transformer receives visual history, prescribed 2D skeleton controls, and camera poses, then predicts future video. In video-action mode, it receives visual history and a language instruction; the action Transformer additionally receives the current robot state and predicts a 16-step action chunk. The action Transformer is asymmetric in that it reads spatially dense features from multiple video blocks without feeding action features back into the video stream during the forward pass. Nevertheless, action-loss gradients propagate through the visual readout into the video backbone, allowing executable-control supervision to refine the representation used by simulation.

The model represents each video clip using an initial anchor latent followed by causal chunks. Each chunk corresponds to 16 video frames and is paired with a 16-step action horizon. Bidirectional attention is used within a chunk, while attention across chunks is restricted to preceding context. This factorization permits autoregressive generation while preserving local temporal modeling.

Human and robot actions share a kinematic schema containing end-effector poses, hand articulation, and torso or camera poses. Embodiment-specific input and output projections accommodate differences in dimensionality and normalization, while the action Transformer remains shared. This is particularly appropriate for the humanoid platform because articulated hands provide a closer correspondence between human and robot motion than gripper-only representations. However, the approach depends on the existence of a sufficiently meaningful cross-embodiment kinematic correspondence; the paper does not establish that the same construction would transfer to substantially different robot morphologies.

Skeleton controls are injected by token addition. Rasterized skeleton sequences are encoded through a separate patch embedding and added elementwise to video tokens at matching spatial and temporal positions. The embedding is zero-initialized so that the pretrained video representation is preserved at initialization. In the reported conditioning ablation, token addition achieves the strongest overall visual quality without increasing inference cost.

Conditioning method ID PSNR ID SSIM OOD PSNR OOD SSIM
AdaLN 17.12 0.5818 16.88 0.5818
Cross-attention 16.29 0.5631 16.31 0.5703
Channel concatenation 17.62 0.5902 17.47 0.5865
Token addition 18.07 0.6030 17.99 0.6109

Token addition improves PSNR over AdaLN by 0.95 dB on in-distribution data and 1.11 dB on out-of-distribution data. Its OOD optical-flow similarity is also highest, although AdaLN is marginally better on ID optical-flow similarity. The implication is specific: aligned additive conditioning provides a favorable quality-cost tradeoff for this architecture, but the result does not demonstrate that token addition is universally optimal across video backbones or control modalities.

Coarse-to-fine training curriculum

The training schedule separates broad visual adaptation from action grounding.

Figure 4

Figure 4: XPACE first adapts video prediction, then jointly trains simulation and action prediction, and finally separates simulator adaptation from policy fine-tuning.

Stage I trains future-video prediction on human and robot video while treating all sources as video-only. Stage II introduces the action branch and alternates equally between video-action and simulation modes. Its data mixture changes across three phases:

  • Phase II-a emphasizes broad human and action-unlabeled video.
  • Phase II-b increases bridge and robot supervision and introduces failure data for simulation.
  • Phase II-c makes target-robot data dominant while retaining replay from human, bridge, and action-unlabeled sources.

Figure 5

Figure 5: The curriculum reduces broad L1/L2 sampling while increasing bridge and robot supervision, without eliminating human replay.

The fixed 50/50 sampling of policy and simulation modes is distinct from the changing source mixture. This separation allows the optimization objective to remain stable while the supervision progressively shifts toward target-robot control. Human replay prevents the final stage from becoming robot-only fine-tuning.

The paper’s ablations support both aspects of this curriculum. In a controlled video-pretraining sweep, the first eighth of Stage I exposure produces an 8.1% reduction in held-out action loss relative to no Stage I adaptation, compared with a 12.5% reduction after full exposure. The benefit saturates partially: quarter and half exposure each produce a 9.0% reduction, while full exposure gives the best result.

Continued human action supervision is more important than initialization alone. Relative to robot-only adaptation, human mid-training followed by robot-only fine-tuning reduces benchmark action loss by 8.5%, whereas human-robot co-training through the final stage reduces it by 14.0%. Because both human-data conditions share the same mid-trained initialization and final-stage budget, the 5.5-percentage-point difference supports the claim that human supervision acts as active rehearsal during robot adaptation rather than merely providing useful initialization.

The coverage-stratified analysis gives this result a more specific interpretation. Co-training reduces action loss by 10.0% for behaviors densely covered by both robot and human data, and by 22.7% for behaviors absent from robot teleoperation but densely represented in human data. The gain is therefore not limited to robot-unseen tasks, but it is larger where human data supplies behavioral coverage unavailable from robot demonstrations. The experiment does not isolate bridge data or prove that explicit feature alignment is unnecessary; it establishes the aggregate value of the complete mixture and curriculum.

Self-gradient forcing and simulator fidelity

A major technical obstacle is the teacher-forcing mismatch. During ordinary training, each predicted video chunk is conditioned on recorded preceding frames. During autoregressive simulation, later chunks are conditioned on the model’s own generated frames, so errors can accumulate.

XPACE addresses this mismatch with a Stage III simulator copy trained using self-gradient forcing. The simulator generates an entire autoregressive prefix without gradient tracking, detaches the generated latent values, re-encodes the context with gradients, and applies context-aware flow matching against recorded target chunks. The simulator remains conditioned on externally supplied skeleton controls and camera poses, while the visual history becomes self-generated.

The method improves both prediction quality and efficiency. On 193-frame rollouts, SGF raises ID PSNR from 16.94 to 17.83 dB and OOD PSNR from 17.74 to 18.75 dB. It improves all reported quality metrics at both short and long horizons. At the long horizon, inference is approximately 3.00 times faster on ID data and 3.30 times faster on OOD data, with essentially unchanged memory consumption.

The speed improvement follows from reducing the ODE sampling budget from 50 steps per chunk to 8 steps, while training the simulator under the same self-conditioned regime. Thus, SGF is not only a stability intervention; it enables a lower-cost inference schedule. The final Phase II-c plus SGF simulator achieves 193-frame PSNR values of 18.11 dB on ID data and 18.05 dB on OOD data, with corresponding DINO similarities of 0.9834 and 0.9705.

These metrics establish improved visual and motion fidelity on held-out videos, but they do not alone establish reliable counterfactual simulation under arbitrary policy actions. The recovery-data experiment provides the more consequential test because simulator errors can directly affect policy training.

Recovery synthesis and policy self-improvement

XPACE uses the SGF-adapted simulator to construct DAgger-style recovery data. Starting from an expert trajectory, the method selects a pivot frame and perturbs the arm with the largest cumulative end-effector displacement. The perturbation moves the end effector away from the expert trajectory for either 8 or 16 steps and then returns it over an equal horizon. Directions are restricted to angles between 90 and 150 degrees relative to subsequent expert motion, excluding both negligible perturbations and directly contradictory recoveries.

The simulator renders only the inserted deviation-recovery segment. The real prefix and expert continuation are retained, producing a clip that starts from a grounded visual context, passes through a simulated off-trajectory state, and reconnects to the recorded expert tail. The policy-training crop begins at the deviation endpoint so that the model must predict recovery rather than merely imitate the nominal approach.

Generated examples are filtered using DINOv3 similarity, depth-weighted PSNR, and depth-weighted SSIM. A candidate must improve from deviation endpoint to recovery endpoint on all metrics and must satisfy absolute similarity thresholds at recovery. The highest-scoring candidates are retained in a task-balanced mixture, accounting for 8% of policy fine-tuning data; the remaining 92% follows the Phase II-c recipe.

This procedure improves real-robot performance across all three evaluated tasks.

Task Before DAgger SR After DAgger SR Before progress After progress
Banana pick-and-place 80% 90% — —
Water pouring 50% 95% 0.78 0.98
Cola handover 55% 75% — —
Unweighted mean 61.7% 86.7% 0.81 0.93

The mean success rate increases by 25 percentage points, while mean task progress increases from 0.81 to 0.93. Water pouring exhibits the largest improvement: success rises from 50% to 95% and progress from 0.78 to 0.98. This is a meaningful result because pouring requires sustained wrist rotation under object contact, rather than only transporting an object to a receptacle.

The attribution is nevertheless limited. The comparison is before versus after policy fine-tuning, and the fine-tuning stage adds optimization steps as well as synthetic data. Consequently, the experiment supports the utility of the complete recovery-fine-tuning procedure but does not isolate the causal contribution of recovery examples from additional optimization budget.

Real-robot performance and human-to-robot transfer

On the primary three-task benchmark, XPACE is evaluated on banana pick-and-place, water pouring, and bowl stacking, with 20 trials per task. Banana placement and pouring are treated as in-distribution relative to robot teleoperation; bowl stacking is task-level OOD because it is absent from the robot demonstrations but represented in human and bridge data.

Figure 6

Figure 6: Qualitative real-robot transfer includes drawer pushing, bread transfer across receptacles, and sequential bowl nesting.

XPACE achieves an average success rate of 68.3%, compared with 40.0% for DreamZero and 6.7% for GR00T. Average task progress is 0.84 for XPACE, 0.68 for DreamZero, and 0.36 for GR00T. The comparison is end-to-end rather than architectural: the supervised human and robot corpora are matched, but XPACE alone receives Stage I video adaptation. The paper explicitly cautions against interpreting these results as an architecture-only comparison.

The task-level results are more informative than the aggregate. XPACE succeeds in 16 of 20 banana trials, including all five tested object positions, and in 15 of 20 bowl-stacking trials. Pouring remains substantially harder: XPACE succeeds in 10 of 20 trials, although its progress score reaches 0.78. The disparity between progress and success indicates that the policy frequently achieves intermediate milestones without completing the sustained rotation and release sequence.

The human-data ablation evaluates the effect of heterogeneous experience on robustness. Relative to a robot-only model, the full-data model increases banana pick-and-place success across fixed spatial positions from 50% to 80%, with progress increasing from 0.70 to 0.93. In a distractor setting containing a cucumber, flower, and toy duck absent from the robot corpus, success increases from 30% to 50%, and progress from 0.48 to 0.74. These results support a specific conclusion: human and video experience improve execution under visual and spatial variation, not merely transfer on tasks completely absent from robot training.

The qualitative transfer cases are also deliberately differentiated. Drawer pushing demonstrates approach and contact with an object absent from robot data. Bread transfer demonstrates manipulation of an object present in both human and robot data but under an unseen toaster-to-plate configuration. Three-bowl nesting tests sequential composition under one instruction. These examples are qualitative and do not provide trial-level success estimates, but they are consistent with the coverage-stratified offline analysis.

Policy-conditioned simulation as diagnosis

XPACE also closes the loop between policy and simulator for qualitative evaluation. The policy generates actions from visual history, state, and language; those actions are converted into skeleton and camera controls; the simulator generates the next frames; and the resulting frames are fed back to the policy.

Figure 7

Figure 7: Policy-conditioned rollouts expose both correct target selection and plausible but instruction-inconsistent choices.

The examples show that the simulator can reveal instruction-following failures before physical execution. In two cases the policy selects the requested object; in two others it selects an incorrect target, such as a red fruit instead of a green fruit. This mode is useful for diagnosing semantic control failures separately from low-level motion quality.

The evidence here remains qualitative. The paper does not report a calibrated relationship between simulator judgments and real-robot success, nor does it establish that simulated policy rankings reliably predict physical rankings. The simulator should therefore be interpreted as a diagnostic and data-generation instrument rather than a validated replacement for real-robot evaluation.

Limitations and open questions

The empirical scope is concentrated on manipulation with the XPENG IRON humanoid and a limited set of task families. The cross-embodiment action schema benefits from the kinematic similarity between human hands and the humanoid robot; transfer to dissimilar morphologies remains unresolved. The bridge-data construction is also heuristic: the paper uses task and appearance overlap but does not isolate the contribution of each alignment criterion.

The real-robot evaluations use 20 trials per task and, in several comparisons, a single reported training trajectory or checkpoint sweep rather than independent random seeds. The paper reports checkpoint variation for some offline ablations but not uncertainty across complete training runs. External comparisons are confounded by differences in training recipes, especially the exclusive use of Stage I video adaptation by XPACE.

The recovery-data experiment does not disentangle synthetic-data effects from extra policy optimization. Filtering based on DINO similarity, PSNR, and SSIM also imposes a strong assumption: visual closeness to the expert pivot is treated as evidence that the recovery is behaviorally suitable. This may reject valid recoveries or retain visually plausible trajectories with incorrect contact dynamics. More generally, the simulator is trained from recorded controls and is evaluated on prescribed skeleton trajectories near demonstrations; its reliability under substantially novel contacts, occlusions, and long-horizon interventions is not established.

Finally, the paper leaves open whether generated recovery data remain useful as deviation magnitude, recovery horizon, and task complexity increase. It also does not quantify uncertainty or use simulator uncertainty to regulate policy updates. These questions directly bear on the validity of simulation-driven policy improvement beyond the local, filtered recovery setting studied here.

Conclusion

XPACE presents a unified framework in which heterogeneous visual and action experience, world modeling, robot control, and policy refinement are connected through a shared causal video backbone. Its main empirical findings are that human experience remains useful during target-robot adaptation, that cross-embodiment supervision improves both robot-covered and robot-absent behavior families, and that self-gradient-forced simulation can generate recovery supervision that improves real-robot completion.

The strongest result is the closed-loop policy improvement: filtered simulator-generated recoveries raise average success from 61.7% to 86.7% across the evaluated tasks. The result is promising but appropriately bounded by the absence of an optimization-budget control and by the limited scope of the simulator’s validated intervention distribution. Within those constraints, the paper demonstrates a coherent mechanism for turning heterogeneous experience into both executable behavior and additional corrective supervision.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Explain it Like I'm 14

论文概述

这篇论文介绍了一个叫 MoT 的机器人学习系统。它的目标是让机器人能够:

  • 从人类视频和机器人示范中学习;
  • 根据看到的环境和任务指令选择动作;
  • 预测自己的动作会怎样改变周围的世界;
  • 在犯错后学会“补救”,而不是直接失败。

研究人员把这个系统测试在 XPENG 的 IRON 人形机器人上。

简单来说,MoT 像是同时拥有两个能力:

  1. 行动计划员:决定机器人下一步该做什么。
  2. 想象模拟器:预测机器人做出某个动作后,画面会发生什么变化。

研究想解决什么问题?

研究人员主要关心三个问题:

  1. 机器人应该从哪些经验中学习? 机器人示范很有用,因为里面有真实的机器人动作,但收集起来很昂贵。人类视频数量很多,却通常没有直接告诉机器人该怎样移动。
  2. 能不能用同一个模型同时学习“如何行动”和“世界会怎样变化”? 例如,机器人不只要知道“把杯子拿起来”,还要预测拿起后杯子会移动到哪里。
  3. 机器人能不能利用自己的模拟器改进自己? 现实中的示范大多是成功的,但机器人真正执行时可能会偏离正确路线。研究人员希望机器人能在模拟中练习如何从错误中恢复。

研究方法

使用多种不同的数据

研究团队把大约 5,000 小时的身体活动视频整理成几个部分,像一个“数据金字塔”:

数据类型 作用
普通人类第一视角视频 让模型了解物体、场景和动作如何变化
带有人体动作信息的视频 教模型学习手、手腕和身体怎样运动
人类与机器人相似的“桥接”数据 帮助模型把人类动作知识转移给机器人
机器人遥操作示范 教机器人真正可执行的控制动作
失败和恢复数据 让模拟器知道错误动作可能造成什么结果

这里的“第一视角视频”就是像人戴着摄像头拍的视频,画面看起来像是人自己正在看东西。

研究人员还给视频配上不同层次的文字说明。例如:

  • “整理桌面”这样的整体任务;
  • “拿起桌上的杯子”这样的具体动作;
  • 对整个画面的详细描述;
  • 同一个任务的不同说法。

这样,机器人可以学习理解不同类型的任务指令。

让人类和机器人使用相似的动作表示

人类和机器人身体结构不完全一样,但它们都有手、手腕和手臂。研究人员把这些动作转换成一种共同的表示方式,例如:

  • 手的位置;
  • 手的方向;
  • 手指的动作;
  • 身体和摄像头的位置。

这就像把中文和英文都翻译成一种共同的“动作语言”,让模型比较人类和机器人是怎样完成类似任务的。

一个模型,两个工作模式

MoT 使用的是一种基于 Transformer 的模型。Transformer 可以理解为一种擅长处理连续信息的人工智能结构,常用于语言和视频处理。

MoT 有两个主要模式。

1. 行动模式

机器人输入:

  • 当前看到的画面;
  • 自己身体的状态;
  • 任务指令。

模型输出:

  • 机器人接下来应该执行的动作;
  • 这些动作之后可能出现的视频画面。

例如,任务是“把红色杯子放到桌子另一边”,模型会尝试预测机器人手臂怎样移动,以及杯子和场景接下来会怎样变化。

2. 模拟模式

研究人员直接告诉模型要执行什么动作,然后模型预测未来画面。

例如,输入“手向右移动、手指闭合”,模型预测:

  • 手会移动到哪里;
  • 是否会碰到杯子;
  • 杯子之后会怎样。

这有点像在电脑游戏中按下一个操作键,系统提前显示可能发生的结果。

逐步训练模型

研究人员没有一开始就要求模型学会所有事情,而是采用“由粗到细”的训练方法:

  1. 先让模型从视频中学习世界和物体如何变化;
  2. 再加入人类和机器人的动作学习;
  3. 最后逐渐增加机器人数据,让模型更适应真正的机器人身体。

不过,即使到了最后阶段,模型仍保留一部分人类视频和动作数据。这样做是为了避免模型只记住机器人示范,而忘记从大量人类经验中学到的知识。

让机器人练习“犯错后恢复”

现实中的机器人可能会:

  • 手臂偏了一点;
  • 没有准确抓住物体;
  • 把物体推到了错误的位置;
  • 在执行任务时进入没有见过的情况。

为了训练机器人处理这些情况,研究人员使用模拟器制造“偏离—恢复”轨迹:

  1. 先从一个成功的专家示范开始;
  2. 故意让机器人的手臂偏离原来的路线;
  3. 让模拟器预测这次偏离会产生怎样的画面;
  4. 继续模拟机器人如何回到正确路线;
  5. 把这些恢复过程加入训练数据;
  6. 再用这些数据训练机器人控制策略。

这种方法和教学生解决错误很相似:不仅告诉学生正确答案,还让学生练习“如果走错一步,怎样回到正确道路”。

研究人员还使用了一种叫 DAgger 的方法。它的核心思想是:不仅训练机器人模仿成功示范,还训练它处理那些自己可能遇到的错误状态。

主要结果

实验在 XPENG 的 IRON 人形机器人上进行。论文报告了几个重要发现。

混合不同来源的数据效果更好

只使用机器人示范会限制机器人的学习范围,因为机器人数据收集很慢、成本也高。

加入人类视频和人体动作数据后,机器人能够接触更多:

  • 不同的物体;
  • 不同的场景;
  • 不同的操作方式;
  • 更丰富的人类行为。

结果显示,这种混合训练提高了机器人的表现和稳定性。

机器人可以学会示范中没有直接出现的技能

一些技能只在人类视频中出现,并没有对应的机器人遥操作示范。模型仍然可以从人类行为中学习相关知识,并尝试把它转移到机器人身上。

这说明机器人不一定要为每个任务都收集大量专门的机器人示范。

模拟器能够帮助机器人改进

经过专门训练后,模拟器在连续预测多个未来画面时更加可靠,出现的错误更少。

使用模拟器生成的“偏离—恢复”数据后,机器人在真实世界中的任务完成率进一步提高。这说明模拟器不只是一个用来展示未来画面的工具,还可以帮助机器人学习更好的控制方法。

预测未来画面对控制有帮助

模型不仅预测动作,也预测动作带来的视觉结果。这样,机器人可以在执行前“想象”可能发生的事情。

例如,它可以估计:

  • 手是否会到达正确位置;
  • 物体是否可能被碰倒;
  • 当前动作是否会让任务变得更难;
  • 如果偏离,怎样尝试恢复。

研究为什么重要?

这项研究的重要性在于,它把几种能力连接到了一起:

  • 从大量人类视频学习;
  • 从机器人示范学习可执行动作;
  • 预测动作造成的后果;
  • 用模拟经验训练机器人处理错误。

过去,这些能力通常由不同系统分别完成。MoT 尝试让它们共享同一个视觉基础模型,因此人类视频中的视觉知识可以帮助机器人学习,机器人动作数据也可以让模型更了解如何控制真实身体。

可能的影响

如果这种方法继续发展,未来的机器人可能不需要为每项新任务都收集大量专门数据。它们可以先从人类视频中学习大致的行为方式,再通过少量机器人示范学会如何在自己的身体上执行。

这种机器人可能更适合:

  • 家庭服务;
  • 工厂和仓库工作;
  • 医院辅助;
  • 危险环境探索;
  • 需要处理意外情况的任务。

不过,这项研究也有局限。模拟器的预测并不一定总是正确。如果机器人过度相信错误的模拟结果,可能会学到不安全的动作。因此,未来还需要进一步提高模拟器的准确性,并在真实环境中严格测试安全性。

总的来说,这篇论文提出了一种让机器人“观察、想象、行动并纠错”的学习方法。它表明,结合人类经验、机器人示范和模拟练习,可能是制造更灵活、更可靠通用机器人的重要方向。

Practical Applications

Immediate Applications

The paper’s demonstrated capabilities—learning from heterogeneous human and robot experience, predicting action-conditioned visual outcomes, and generating filtered recovery trajectories—support the following applications that are relatively feasible with current robotics infrastructure, provided they are deployed in constrained environments.

  • Robust manipulation policies for humanoid and dual-arm robots — Robotics and manufacturing
    • Train robots from a mixture of teleoperation data, human demonstrations, egocentric video, and task-aligned “bridge” data rather than relying exclusively on expensive robot demonstrations.
    • Apply the policy to tasks such as object pickup, placement, sorting, opening containers, tool use, and tabletop assembly.
    • Use the deployed streaming and overlapping-horizon control workflow to continuously revise action chunks while the robot remains in motion.
    • Potential product/workflow: a robot-learning platform that imports human demonstration videos, aligns them with robot teleoperation data, and produces a closed-loop manipulation policy.
    • Dependencies: sufficiently similar human and robot kinematics; accurate robot state and camera calibration; safe action limits; task and appearance overlap between training data and deployment scenes.
  • Simulation-based recovery training for industrial robots — Manufacturing and warehousing
    • Use the learned simulator to create controlled deviations from successful trajectories and train the robot to recover from misplaced grasps, object shifts, partial collisions, or imperfect arm motions.
    • Integrate generated recovery clips into behavior-cloning pipelines as an alternative to manually collecting every failure case.
    • Potential tool: an automated “failure augmentation” module that samples perturbations, renders their visual consequences, filters inconsistent trajectories, and exports recovery examples for policy fine-tuning.
    • Dependencies: the simulator must accurately preserve action-to-visual correspondence near the operating distribution; generated recoveries require validation before being used on safety-critical hardware.
  • Pre-deployment policy testing and failure diagnosis — Robotics engineering
    • Prescribe candidate action sequences to the simulator and inspect predicted video outcomes before executing them on a physical robot.
    • Use these rollouts to identify likely failures such as collisions, incorrect target selection, loss of grasp, or failure to complete a subtask.
    • Potential workflow: simulate multiple action candidates, rank them using task completion heuristics or a vision-language evaluator, then execute only the safest candidate.
    • Dependencies: predicted videos are not guaranteed to be physically accurate; quantitative simulation-to-reality calibration and conservative execution thresholds are necessary.
  • Human-to-robot skill transfer — Service robotics and assistive robotics
    • Transfer behaviors observed in human videos—such as grasping, handover, drawer opening, or object arrangement—to humanoid robots, including tasks absent from the robot teleoperation set.
    • Use human videos primarily for behavioral and visual coverage, then ground the transferred skills through bridge data and a smaller amount of robot-specific supervision.
    • Potential product: a demonstration interface in which a user records a task with a wearable or phone camera and the robot learns an initial policy from the recording.
    • Dependencies: the target robot must have compatible degrees of freedom, hand articulation, viewpoint coverage, and object affordances; human motion cannot be assumed to be directly executable by every robot embodiment.
  • Reduction of teleoperation-data requirements — Robotics data operations
    • Use broad action-unlabeled video for visual-dynamics pretraining and reserve costly teleoperation for embodiment-specific grounding and task adaptation.
    • This can reduce the amount of robot-collected data required when introducing a new manipulation task or workspace.
    • Potential workflow: pretrain on large video corpora, adapt with bridge demonstrations, and collect only targeted teleoperation episodes for final grounding.
    • Dependencies: the video corpus must contain relevant object interactions and viewpoints; video-only learning provides visual knowledge but does not by itself establish executable robot actions.
  • Closed-loop control with asynchronous inference — Embedded robotics software
    • Adopt the paper’s streaming visual encoding, action-chunk prediction, queued execution, and overlapping-horizon stitching for robots whose perception and policy inference operate at different rates.
    • This is useful when full model inference is slower than the robot’s low-level control loop.
    • Potential tool: a robotics middleware component that maintains an action queue, preserves committed commands, inserts new action suffixes, and smooths transitions between plans.
    • Dependencies: timing guarantees, reliable state estimation, bounded inference latency, and safety checks at the command-publication layer.
  • Robot learning curriculum design — Academia and industrial R&D
    • Replicate the coarse-to-fine curriculum: begin with broad video prediction, add human action supervision, increase bridge and target-robot supervision, and retain a replay stream to reduce catastrophic forgetting.
    • This provides a practical training recipe for multi-source robot foundation models.
    • Dependencies: carefully balanced data sampling; consistent language annotations; compatible action schemas; sufficient compute for joint video-action training.
  • Evaluation of recovery behavior rather than nominal imitation — Robotics benchmarking
    • Evaluate policies on perturbed states and recovery tasks, not only on reproducing successful demonstrations.
    • Metrics can include recovery success rate, task completion after perturbation, collision rate, intervention frequency, and robustness to viewpoint or object-position changes.
    • Dependencies: standardized perturbation protocols and reliable methods for distinguishing valid recovery from merely plausible visual prediction.
  • Data augmentation for simulators and digital-twin systems — Industrial simulation
    • Add successful, failed, low-progress, and recovery trajectories to simulator training so that the model does not learn only idealized success dynamics.
    • This can improve the usefulness of learned simulators for policy evaluation and fault analysis.
    • Dependencies: failure data must be representative; visual prediction quality should be checked against real trajectories across multiple time horizons.
  • Assistive and household robot prototyping — Daily life
    • Use human videos and small robot demonstration sets to prototype robots that perform repetitive household activities such as tidying, placing objects, or handing items to a person.
    • Recovery augmentation could help with common household variability, such as objects being slightly displaced or grasps being imperfect.
    • Dependencies: home environments are substantially less controlled than the evaluated setting; safety, privacy, object diversity, and reliable uncertainty estimation are essential before consumer deployment.

Long-Term Applications

The following applications require additional research, larger-scale validation, stronger safety mechanisms, or better physical and cross-embodiment modeling before they can be considered broadly deployable.

  • General-purpose humanoid household robots — Consumer robotics
    • Combine large-scale human video, robot demonstrations, language instructions, and simulator-generated recoveries into a generalist household robot capable of transferring skills across kitchens, offices, and homes.
    • The joint world-action formulation could allow the robot to predict both what will happen visually and which action sequence should be executed.
    • Potential product: a continuously improving household robot that learns from demonstrations, evaluates candidate actions internally, and expands its skill library through simulated recovery.
    • Dependencies: reliable long-horizon physical simulation, robust object and contact modeling, strong uncertainty estimation, safe exploration, privacy-preserving data collection, and validation across many robot embodiments.
  • Cross-embodiment foundation models — Robotics platforms and academia
    • Extend the shared action schema beyond humanoids to mobile manipulators, grippers, quadrupeds, and other platforms.
    • Embodiment-specific input/output projections could map a shared action representation into different robot control spaces while preserving common visual and behavioral knowledge.
    • Dependencies: a meaningful common representation for embodiments with very different morphology, sensing, balance constraints, and action spaces; simple linear projections may be insufficient.
  • Model-predictive control using generated future video — Robotics and autonomous systems
    • Generate multiple possible future visual trajectories under candidate action sequences, score them for task success or safety, and execute the best candidate in a receding-horizon loop.
    • This would turn the simulator into an action planner rather than only a training-data generator.
    • Potential workflow: sample candidate controls → render future videos → evaluate object state, collision risk, and instruction compliance → execute a short action chunk → replan.
    • Dependencies: fast inference, calibrated uncertainty, reliable candidate ranking, and physical consistency over longer horizons.
  • World-model-based reinforcement learning and autonomous skill discovery — Academia
    • Use the simulator to generate imagined trajectories for policy optimization, value learning, or autonomous exploration.
    • The current method emphasizes DAgger-style recovery around expert demonstrations; future work could extend it to broader model-based reinforcement learning.
    • Dependencies: compounding model error, reward specification, distribution shift, and the risk that policies exploit simulator artifacts rather than real-world dynamics.
  • Autonomous data collection and iterative self-improvement — Industrial robot fleets
    • Deploy a policy across many robots, collect real failures and corrections, update the simulator with those observations, and generate improved recovery supervision for the next policy version.
    • This could create a fleet-learning loop for warehouses, factories, or service environments.
    • Potential system: fleet telemetry → failure clustering → simulator adaptation → synthetic recovery generation → filtered policy update → staged real-world validation.
    • Dependencies: secure data pipelines, versioned models, fleet-wide calibration, reliable failure labeling, rollback mechanisms, and strict safety governance.
  • Robotic surgery, rehabilitation, and clinical assistance — Healthcare
    • In carefully constrained settings, action-conditioned visual prediction could support rehearsal, error recovery, or assistance for repetitive manipulation procedures.
    • Human demonstrations could provide broad procedural coverage, while robot-specific data would ground executable control.
    • Dependencies: substantially higher accuracy and explainability requirements; certified hardware, anatomical and tissue modeling, clinical trials, regulatory approval, and human oversight are mandatory. The paper’s results do not establish readiness for autonomous medical use.
  • Warehouse and logistics robots operating under unexpected disturbances — Logistics
    • Train systems to recover from package displacement, occlusions, grasp failures, conveyor timing errors, and other deviations from nominal demonstrations.
    • Use simulated recovery trajectories to supplement rare real failure data.
    • Dependencies: accurate modeling of rigid-body contacts, packaging variation, conveyor dynamics, multi-agent interactions, and safety around human workers.
  • Energy and infrastructure inspection robots — Energy and utilities
    • Transfer skills from human inspection videos to robots that manipulate valves, switches, panels, or tools while using predicted visual consequences to avoid damaging infrastructure.
    • Recovery training could address misalignment, partial occlusion, or failed contact during inspection and maintenance.
    • Dependencies: domain-specific visual coverage, strict operational safety, weather and lighting robustness, accurate force/contact sensing, and certification for critical infrastructure.
  • Interactive educational and research platforms — Education
    • Create laboratory systems in which students can teach a robot through demonstrations, inspect predicted futures, intentionally introduce deviations, and study how recovery policies are learned.
    • The framework offers a concrete teaching example for multimodal learning, imitation learning, world models, and sim-to-real transfer.
    • Dependencies: simplified interfaces, transparent diagnostics, safe hardware, reproducible datasets, and curricula that do not require industrial-scale compute.
  • Policy and standards for learned robot simulation — Public policy and safety
    • Develop evaluation standards requiring learned world models to report prediction fidelity, uncertainty, recovery validity, and simulation-to-reality performance before synthetic data are used for control.
    • Policies could distinguish between simulators used for visualization, offline planning, policy training, and safety-critical decision-making.
    • Dependencies: shared benchmarks, representative real-world test suites, auditable data provenance, and agreement on acceptable error rates.
  • Everyday personal robotics with privacy-preserving learning — Daily life
    • Train domestic robots from egocentric video while using local or federated learning to avoid sending private household recordings to centralized servers.
    • The data-pyramid strategy could combine public pretraining, user-specific bridge data, and small amounts of household interaction.
    • Dependencies: privacy-preserving perception and storage, consent management, robustness to highly varied homes, secure model updates, and mechanisms to prevent unsafe behavior learned from noisy demonstrations.
  • Synthetic corrective supervision for software agents in embodied interfaces — Software and human-computer interaction
    • Adapt the deviation–recovery principle to agents that operate visual interfaces, simulations, or physical-digital workflows: generate off-nominal states, simulate corrective actions, and train the agent to return to a valid task trajectory.
    • Examples include recovering from an incorrect UI selection, a failed database workflow, or a partially completed automation task.
    • Dependencies: the environment must be sufficiently simulatable, state transitions must be observable, and synthetic corrections must correspond to valid real-system outcomes rather than merely plausible screens.

Overall, the most immediately credible applications are robot-learning data reduction, recovery-policy augmentation, pre-deployment simulation, and robust closed-loop manipulation. Broader claims—general-purpose household autonomy, safety-critical control, and fully autonomous self-improvement—depend on improved physical fidelity, uncertainty calibration, cross-embodiment transfer, and extensive real-world validation beyond the experiments described in the paper.

Glossary

  • Action-conditioned simulation: Simulation in which future visual states are predicted from specified actions. “action-conditioned robot video generation”
  • Action horizon: The number of future action steps predicted by a policy. “Na=16N_a=16 is the action horizon”
  • Action projection: A learned mapping that converts embodiment-specific action representations into shared model dimensions. “embodiment-specific action projections accommodate human and robot action spaces”
  • Action schema: A standardized representation of movements shared across different embodiments. “a common kinematic schema comprising end-effector poses, hand articulation, and torso/camera poses”
  • Action Transformer: A Transformer module that predicts robot actions from visual features. “an Action Transformer reads its multi-level visual features to predict executable actions”
  • Affine flow matching: A generative-training method that learns velocity fields connecting noise distributions to data distributions. “We train video and action prediction with flow-matching losses”
  • Autoregressive inference: Sequential prediction in which each generated output becomes context for subsequent predictions. “autoregressive inference conditions on previous predictions”
  • Behavior cloning: Imitation learning that trains a policy to reproduce demonstrated behavior. “policy behavior cloning primarily covers successful expert demonstrations”
  • Bridge data: Data designed to align human demonstrations with robot tasks and visual environments. “Task- and appearance-aligned bridge data”
  • Causal attention: Attention restricted so that predictions can access only permitted earlier context. “We use chunk-wise causal attention”
  • Closed-loop control: Control in which actions are repeatedly updated using new observations of the system. “We deploy \modelname{} for closed-loop control of the IRON robot”
  • Coarse-to-fine curriculum: A training strategy that progresses from broad or less specific supervision to more target-specific supervision. “a coarse-to-fine curriculum progressively emphasizes robot supervision”
  • Cross-embodiment alignment: The alignment of representations or actions across physically different agents. “human bridge data supporting cross-embodiment alignment”
  • Cross-embodiment representation: A shared representation intended to work across different bodies or robot platforms. “encourages cross-embodiment representations”
  • DAgger: An imitation-learning method that iteratively aggregates expert labels for states visited by the learner. “DAgger aggregates expert labels on learner-visited states”
  • DAgger-style recovery augmentation: The use of synthesized recovery examples in an imitation-learning pipeline inspired by DAgger. “following DAgger-style recovery augmentation rather than on-policy expert querying”
  • Data distribution shift: A mismatch between the data distribution used for training and that encountered during deployment. “to address imitation-learning distribution shift”
  • Data pyramid: A hierarchical organization of datasets with progressively more task- and robot-specific supervision. “The resulting data pyramid connects broad interaction experience with robot-specific supervision”
  • Deviation–recovery trajectory: A trajectory that intentionally moves away from an expert path and then returns to successful behavior. “synthesize deviation--recovery trajectories around expert demonstrations”
  • Diffusion model: A generative model trained to transform noise into structured data through a denoising process. “Diffusion Policy models multimodal action sequences through conditional denoising”
  • Distribution matching: An objective that makes generated samples resemble a target distribution, without necessarily matching individual target examples. “Rather than adopting a distribution-matching objective such as DMD”
  • Embodiment: The physical body, morphology, and action capabilities of an agent. “adapting the shared representation to the deployment embodiment”
  • Embodiment-specific projection: A mapping customized to a particular robot or human body while using shared model components. “Each embodiment has its own linear input projection and final output projection for actions”
  • Egocentric video: Video recorded from the viewpoint of the acting person or robot. “broad egocentric video for learning visual dynamics”
  • End effector: The terminal tool or body part that directly interacts with the environment, such as a hand or gripper. “end-effector poses, hand articulation, and torso/camera poses”
  • Expert manifold: The set of states or trajectories associated with successful expert behavior. “return consistently to the expert manifold”
  • Flow matching: A generative-learning technique that trains a neural network to predict transport velocities between noise and data. “context-aware flow matching against the corresponding recorded targets”
  • Foundation model: A broadly pretrained model that can be adapted to multiple downstream tasks. “Robot foundation models connect perception and control”
  • Forward pass: The computation that maps model inputs to outputs during inference or training. “Information flows from video to action in the forward pass”
  • Gradient detachment: A procedure that prevents gradients from propagating through a particular tensor or computation. “we detach the generated prefix and re-encode it with gradients”
  • Gradient tracking: Recording operations so that derivatives can be computed during optimization. “The complete rollout is executed without gradient tracking”
  • Heterogeneous supervision: Training signals that differ in type, source, or degree of action annotation. “a framework that combines heterogeneous supervision”
  • Imitation learning: Learning a policy by reproducing behavior demonstrated by an expert. “DAgger-style recovery augmentation rather than on-policy expert querying”
  • Inverse dynamics: A model that infers the action required to produce a transition between states. “recovers pseudo-actions with latent-action or inverse-dynamics models”
  • Kinematic schema: A structured representation of body positions, orientations, and articulated motion. “a common kinematic schema comprising end-effector poses, hand articulation, and torso/camera poses”
  • Latent action: An internal, learned representation of an action that may not correspond directly to executable robot commands. “DreamDojo uses continuous latent actions”
  • Latent frame: A compressed representation of a video frame in a learned latent space. “A frozen video VAE encodes each clip into an initial anchor latent”
  • Manifold: A structured space representing a set of states or configurations with meaningful geometric relationships. “return consistently to the expert manifold”
  • Mixture-of-transformers: An architecture composed of multiple Transformer modules with specialized functions. “an asymmetric mixture-of-transformers (MoT) architecture”
  • Multimodal action sequence: An action trajectory representing multiple plausible ways to perform a task. “Diffusion Policy models multimodal action sequences”
  • Multi-level visual features: Features extracted at several layers or resolutions of a visual network. “The 15 action blocks read spatially dense features from video blocks 16--30”
  • On-policy expert querying: Obtaining expert supervision on states visited by the currently deployed policy. “rather than on-policy expert querying”
  • Optical flow: The apparent motion of image pixels between consecutive video frames. “FlowWAM uses optical flow as a shared motion representation”
  • Out-of-distribution transfer: Generalization to tasks, states, or data categories not represented in the training distribution. “analyze transfer across task categories”
  • Policy fine-tuning: Further training of a policy to improve performance on a target setting or augmented dataset. “fine-tune a separate policy on filtered recoveries and expert data”
  • Policy rollout: A sequence of actions and resulting states generated by executing a policy. “real policy rollouts”
  • Predictive representation: A learned representation organized to support prediction of future observations or outcomes. “a shared predictive representation”
  • Pseudo-action: An inferred or synthesized action label rather than a directly recorded executable command. “recovers pseudo-actions with latent-action or inverse-dynamics models”
  • Recovery supervision: Training information showing how to correct deviations and return to successful behavior. “the simulator generates recovery trajectories that provide additional supervision”
  • Replay mixture: A retained combination of earlier data sources included during later training. “a small replay mixture of human demonstrations, bridge data, and action-unlabeled video”
  • Self-gradient forcing: A training procedure that adapts a model to contexts generated by its own predictions while using gradients to update context representations. “We adopt self-gradient forcing (SGF) to adapt the simulator”
  • Self-conditioned rollout: A generated sequence in which later predictions are conditioned on the model’s own earlier outputs. “Self-conditioned rollout generation”
  • Skeleton control: A control signal represented as a sequence of body-joint or keypoint configurations. “prescribed skeleton controls and camera poses”
  • Skeleton-conditioned simulation: Simulation in which future frames are generated according to a specified body skeleton sequence. “Skeleton- and camera-conditioned simulation”
  • Teacher forcing: Training a sequence model using ground-truth previous outputs as context instead of its own predictions. “Stages~I and~II use teacher forcing with recorded visual context”
  • Temporal coherence: Consistency of motion and appearance across successive video frames. “temporally coherent, multi-view generation”
  • Token addition: A conditioning method that adds one sequence of embeddings element-wise to another. “We incorporate skeleton controls through token addition at the Transformer input”
  • Trajectory-conditioned simulation: Simulation conditioned on a prescribed sequence of actions or controls. “trajectory-conditioned simulation must reproduce the consequences of the supplied controls”
  • Visual dynamics: The patterns governing how visual observations change as an environment evolves or is acted upon. “learn visual dynamics”
  • Visual grounding: Connecting language or abstract instructions to relevant visual entities and actions. “visually grounded language generation”
  • Visual history: Previously observed frames or latent representations supplied as context for prediction. “Given visual history and a language instruction gg”
  • World model: A learned model that predicts how an environment changes in response to actions. “We introduce \modelname, a unified embodied world model”
  • World simulator: A model that generates predicted future observations under prescribed controls. “a world simulator, predicting future video from history, prescribed skeleton controls, and camera poses”

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 2 tweets with 160 likes about this paper.