XPACE: Joint World and Action Modeling from Heterogeneous Experience
Abstract: A general-purpose robot needs to draw on diverse experience, choose actions, and anticipate how those actions will change the world. We introduce XPACE, a unified embodied world model that serves as both a world action model, jointly predicting executable robot actions and future video, and a world simulator, predicting the visual consequences of prescribed actions. Our key insight is that video prediction can both connect heterogeneous experience to action learning and generate new experience for policy improvement. With a shared video backbone between the policy and simulator, we use action-unlabeled video to learn visual dynamics and action-labeled human and robot demonstrations to jointly learn video and action prediction. Building on this architecture, a coarse-to-fine training curriculum progressively emphasizes robot control while retaining human experience, allowing the policy to learn behaviors beyond those covered by robot demonstrations. Beyond learning from recorded experience, XPACE uses its simulator to create additional recovery supervision for the policy. Specifically, we adapt the simulator to its own generated context, synthesize deviation-recovery trajectories around expert demonstrations, and fine-tune the policy on filtered recovery examples. Experiments on XPENG's IRON humanoid robot show that heterogeneous training improves robustness and enables transfer of human-observed skills to tasks absent from robot demonstrations, while recovery data generated by the model's own simulator further improves real-world task completion. Together, these results demonstrate how joint world and action modeling connects learning from heterogeneous experience with simulation-driven policy self-improvement.
First 10 authors:
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
论文概述
这篇论文介绍了一个叫 MoT 的机器人学习系统。它的目标是让机器人能够:
- 从人类视频和机器人示范中学习;
- 根据看到的环境和任务指令选择动作;
- 预测自己的动作会怎样改变周围的世界;
- 在犯错后学会“补救”,而不是直接失败。
研究人员把这个系统测试在 XPENG 的 IRON 人形机器人上。
简单来说,MoT 像是同时拥有两个能力:
- 行动计划员:决定机器人下一步该做什么。
- 想象模拟器:预测机器人做出某个动作后,画面会发生什么变化。
研究想解决什么问题?
研究人员主要关心三个问题:
- 机器人应该从哪些经验中学习? 机器人示范很有用,因为里面有真实的机器人动作,但收集起来很昂贵。人类视频数量很多,却通常没有直接告诉机器人该怎样移动。
- 能不能用同一个模型同时学习“如何行动”和“世界会怎样变化”? 例如,机器人不只要知道“把杯子拿起来”,还要预测拿起后杯子会移动到哪里。
- 机器人能不能利用自己的模拟器改进自己? 现实中的示范大多是成功的,但机器人真正执行时可能会偏离正确路线。研究人员希望机器人能在模拟中练习如何从错误中恢复。
研究方法
使用多种不同的数据
研究团队把大约 5,000 小时的身体活动视频整理成几个部分,像一个“数据金字塔”:
| 数据类型 | 作用 |
|---|---|
| 普通人类第一视角视频 | 让模型了解物体、场景和动作如何变化 |
| 带有人体动作信息的视频 | 教模型学习手、手腕和身体怎样运动 |
| 人类与机器人相似的“桥接”数据 | 帮助模型把人类动作知识转移给机器人 |
| 机器人遥操作示范 | 教机器人真正可执行的控制动作 |
| 失败和恢复数据 | 让模拟器知道错误动作可能造成什么结果 |
这里的“第一视角视频”就是像人戴着摄像头拍的视频,画面看起来像是人自己正在看东西。
研究人员还给视频配上不同层次的文字说明。例如:
- “整理桌面”这样的整体任务;
- “拿起桌上的杯子”这样的具体动作;
- 对整个画面的详细描述;
- 同一个任务的不同说法。
这样,机器人可以学习理解不同类型的任务指令。
让人类和机器人使用相似的动作表示
人类和机器人身体结构不完全一样,但它们都有手、手腕和手臂。研究人员把这些动作转换成一种共同的表示方式,例如:
- 手的位置;
- 手的方向;
- 手指的动作;
- 身体和摄像头的位置。
这就像把中文和英文都翻译成一种共同的“动作语言”,让模型比较人类和机器人是怎样完成类似任务的。
一个模型,两个工作模式
MoT 使用的是一种基于 Transformer 的模型。Transformer 可以理解为一种擅长处理连续信息的人工智能结构,常用于语言和视频处理。
MoT 有两个主要模式。
1. 行动模式
机器人输入:
- 当前看到的画面;
- 自己身体的状态;
- 任务指令。
模型输出:
- 机器人接下来应该执行的动作;
- 这些动作之后可能出现的视频画面。
例如,任务是“把红色杯子放到桌子另一边”,模型会尝试预测机器人手臂怎样移动,以及杯子和场景接下来会怎样变化。
2. 模拟模式
研究人员直接告诉模型要执行什么动作,然后模型预测未来画面。
例如,输入“手向右移动、手指闭合”,模型预测:
- 手会移动到哪里;
- 是否会碰到杯子;
- 杯子之后会怎样。
这有点像在电脑游戏中按下一个操作键,系统提前显示可能发生的结果。
逐步训练模型
研究人员没有一开始就要求模型学会所有事情,而是采用“由粗到细”的训练方法:
- 先让模型从视频中学习世界和物体如何变化;
- 再加入人类和机器人的动作学习;
- 最后逐渐增加机器人数据,让模型更适应真正的机器人身体。
不过,即使到了最后阶段,模型仍保留一部分人类视频和动作数据。这样做是为了避免模型只记住机器人示范,而忘记从大量人类经验中学到的知识。
让机器人练习“犯错后恢复”
现实中的机器人可能会:
- 手臂偏了一点;
- 没有准确抓住物体;
- 把物体推到了错误的位置;
- 在执行任务时进入没有见过的情况。
为了训练机器人处理这些情况,研究人员使用模拟器制造“偏离—恢复”轨迹:
- 先从一个成功的专家示范开始;
- 故意让机器人的手臂偏离原来的路线;
- 让模拟器预测这次偏离会产生怎样的画面;
- 继续模拟机器人如何回到正确路线;
- 把这些恢复过程加入训练数据;
- 再用这些数据训练机器人控制策略。
这种方法和教学生解决错误很相似:不仅告诉学生正确答案,还让学生练习“如果走错一步,怎样回到正确道路”。
研究人员还使用了一种叫 DAgger 的方法。它的核心思想是:不仅训练机器人模仿成功示范,还训练它处理那些自己可能遇到的错误状态。
主要结果
实验在 XPENG 的 IRON 人形机器人上进行。论文报告了几个重要发现。
混合不同来源的数据效果更好
只使用机器人示范会限制机器人的学习范围,因为机器人数据收集很慢、成本也高。
加入人类视频和人体动作数据后,机器人能够接触更多:
- 不同的物体;
- 不同的场景;
- 不同的操作方式;
- 更丰富的人类行为。
结果显示,这种混合训练提高了机器人的表现和稳定性。
机器人可以学会示范中没有直接出现的技能
一些技能只在人类视频中出现,并没有对应的机器人遥操作示范。模型仍然可以从人类行为中学习相关知识,并尝试把它转移到机器人身上。
这说明机器人不一定要为每个任务都收集大量专门的机器人示范。
模拟器能够帮助机器人改进
经过专门训练后,模拟器在连续预测多个未来画面时更加可靠,出现的错误更少。
使用模拟器生成的“偏离—恢复”数据后,机器人在真实世界中的任务完成率进一步提高。这说明模拟器不只是一个用来展示未来画面的工具,还可以帮助机器人学习更好的控制方法。
预测未来画面对控制有帮助
模型不仅预测动作,也预测动作带来的视觉结果。这样,机器人可以在执行前“想象”可能发生的事情。
例如,它可以估计:
- 手是否会到达正确位置;
- 物体是否可能被碰倒;
- 当前动作是否会让任务变得更难;
- 如果偏离,怎样尝试恢复。
研究为什么重要?
这项研究的重要性在于,它把几种能力连接到了一起:
- 从大量人类视频学习;
- 从机器人示范学习可执行动作;
- 预测动作造成的后果;
- 用模拟经验训练机器人处理错误。
过去,这些能力通常由不同系统分别完成。MoT 尝试让它们共享同一个视觉基础模型,因此人类视频中的视觉知识可以帮助机器人学习,机器人动作数据也可以让模型更了解如何控制真实身体。
可能的影响
如果这种方法继续发展,未来的机器人可能不需要为每项新任务都收集大量专门数据。它们可以先从人类视频中学习大致的行为方式,再通过少量机器人示范学会如何在自己的身体上执行。
这种机器人可能更适合:
- 家庭服务;
- 工厂和仓库工作;
- 医院辅助;
- 危险环境探索;
- 需要处理意外情况的任务。
不过,这项研究也有局限。模拟器的预测并不一定总是正确。如果机器人过度相信错误的模拟结果,可能会学到不安全的动作。因此,未来还需要进一步提高模拟器的准确性,并在真实环境中严格测试安全性。
总的来说,这篇论文提出了一种让机器人“观察、想象、行动并纠错”的学习方法。它表明,结合人类经验、机器人示范和模拟练习,可能是制造更灵活、更可靠通用机器人的重要方向。
Practical Applications
Immediate Applications
The paper’s demonstrated capabilities—learning from heterogeneous human and robot experience, predicting action-conditioned visual outcomes, and generating filtered recovery trajectories—support the following applications that are relatively feasible with current robotics infrastructure, provided they are deployed in constrained environments.
- Robust manipulation policies for humanoid and dual-arm robots — Robotics and manufacturing
- Train robots from a mixture of teleoperation data, human demonstrations, egocentric video, and task-aligned “bridge” data rather than relying exclusively on expensive robot demonstrations.
- Apply the policy to tasks such as object pickup, placement, sorting, opening containers, tool use, and tabletop assembly.
- Use the deployed streaming and overlapping-horizon control workflow to continuously revise action chunks while the robot remains in motion.
- Potential product/workflow: a robot-learning platform that imports human demonstration videos, aligns them with robot teleoperation data, and produces a closed-loop manipulation policy.
- Dependencies: sufficiently similar human and robot kinematics; accurate robot state and camera calibration; safe action limits; task and appearance overlap between training data and deployment scenes.
- Simulation-based recovery training for industrial robots — Manufacturing and warehousing
- Use the learned simulator to create controlled deviations from successful trajectories and train the robot to recover from misplaced grasps, object shifts, partial collisions, or imperfect arm motions.
- Integrate generated recovery clips into behavior-cloning pipelines as an alternative to manually collecting every failure case.
- Potential tool: an automated “failure augmentation” module that samples perturbations, renders their visual consequences, filters inconsistent trajectories, and exports recovery examples for policy fine-tuning.
- Dependencies: the simulator must accurately preserve action-to-visual correspondence near the operating distribution; generated recoveries require validation before being used on safety-critical hardware.
- Pre-deployment policy testing and failure diagnosis — Robotics engineering
- Prescribe candidate action sequences to the simulator and inspect predicted video outcomes before executing them on a physical robot.
- Use these rollouts to identify likely failures such as collisions, incorrect target selection, loss of grasp, or failure to complete a subtask.
- Potential workflow: simulate multiple action candidates, rank them using task completion heuristics or a vision-language evaluator, then execute only the safest candidate.
- Dependencies: predicted videos are not guaranteed to be physically accurate; quantitative simulation-to-reality calibration and conservative execution thresholds are necessary.
- Human-to-robot skill transfer — Service robotics and assistive robotics
- Transfer behaviors observed in human videos—such as grasping, handover, drawer opening, or object arrangement—to humanoid robots, including tasks absent from the robot teleoperation set.
- Use human videos primarily for behavioral and visual coverage, then ground the transferred skills through bridge data and a smaller amount of robot-specific supervision.
- Potential product: a demonstration interface in which a user records a task with a wearable or phone camera and the robot learns an initial policy from the recording.
- Dependencies: the target robot must have compatible degrees of freedom, hand articulation, viewpoint coverage, and object affordances; human motion cannot be assumed to be directly executable by every robot embodiment.
- Reduction of teleoperation-data requirements — Robotics data operations
- Use broad action-unlabeled video for visual-dynamics pretraining and reserve costly teleoperation for embodiment-specific grounding and task adaptation.
- This can reduce the amount of robot-collected data required when introducing a new manipulation task or workspace.
- Potential workflow: pretrain on large video corpora, adapt with bridge demonstrations, and collect only targeted teleoperation episodes for final grounding.
- Dependencies: the video corpus must contain relevant object interactions and viewpoints; video-only learning provides visual knowledge but does not by itself establish executable robot actions.
- Closed-loop control with asynchronous inference — Embedded robotics software
- Adopt the paper’s streaming visual encoding, action-chunk prediction, queued execution, and overlapping-horizon stitching for robots whose perception and policy inference operate at different rates.
- This is useful when full model inference is slower than the robot’s low-level control loop.
- Potential tool: a robotics middleware component that maintains an action queue, preserves committed commands, inserts new action suffixes, and smooths transitions between plans.
- Dependencies: timing guarantees, reliable state estimation, bounded inference latency, and safety checks at the command-publication layer.
- Robot learning curriculum design — Academia and industrial R&D
- Replicate the coarse-to-fine curriculum: begin with broad video prediction, add human action supervision, increase bridge and target-robot supervision, and retain a replay stream to reduce catastrophic forgetting.
- This provides a practical training recipe for multi-source robot foundation models.
- Dependencies: carefully balanced data sampling; consistent language annotations; compatible action schemas; sufficient compute for joint video-action training.
- Evaluation of recovery behavior rather than nominal imitation — Robotics benchmarking
- Evaluate policies on perturbed states and recovery tasks, not only on reproducing successful demonstrations.
- Metrics can include recovery success rate, task completion after perturbation, collision rate, intervention frequency, and robustness to viewpoint or object-position changes.
- Dependencies: standardized perturbation protocols and reliable methods for distinguishing valid recovery from merely plausible visual prediction.
- Data augmentation for simulators and digital-twin systems — Industrial simulation
- Add successful, failed, low-progress, and recovery trajectories to simulator training so that the model does not learn only idealized success dynamics.
- This can improve the usefulness of learned simulators for policy evaluation and fault analysis.
- Dependencies: failure data must be representative; visual prediction quality should be checked against real trajectories across multiple time horizons.
- Assistive and household robot prototyping — Daily life
- Use human videos and small robot demonstration sets to prototype robots that perform repetitive household activities such as tidying, placing objects, or handing items to a person.
- Recovery augmentation could help with common household variability, such as objects being slightly displaced or grasps being imperfect.
- Dependencies: home environments are substantially less controlled than the evaluated setting; safety, privacy, object diversity, and reliable uncertainty estimation are essential before consumer deployment.
Long-Term Applications
The following applications require additional research, larger-scale validation, stronger safety mechanisms, or better physical and cross-embodiment modeling before they can be considered broadly deployable.
- General-purpose humanoid household robots — Consumer robotics
- Combine large-scale human video, robot demonstrations, language instructions, and simulator-generated recoveries into a generalist household robot capable of transferring skills across kitchens, offices, and homes.
- The joint world-action formulation could allow the robot to predict both what will happen visually and which action sequence should be executed.
- Potential product: a continuously improving household robot that learns from demonstrations, evaluates candidate actions internally, and expands its skill library through simulated recovery.
- Dependencies: reliable long-horizon physical simulation, robust object and contact modeling, strong uncertainty estimation, safe exploration, privacy-preserving data collection, and validation across many robot embodiments.
- Cross-embodiment foundation models — Robotics platforms and academia
- Extend the shared action schema beyond humanoids to mobile manipulators, grippers, quadrupeds, and other platforms.
- Embodiment-specific input/output projections could map a shared action representation into different robot control spaces while preserving common visual and behavioral knowledge.
- Dependencies: a meaningful common representation for embodiments with very different morphology, sensing, balance constraints, and action spaces; simple linear projections may be insufficient.
- Model-predictive control using generated future video — Robotics and autonomous systems
- Generate multiple possible future visual trajectories under candidate action sequences, score them for task success or safety, and execute the best candidate in a receding-horizon loop.
- This would turn the simulator into an action planner rather than only a training-data generator.
- Potential workflow: sample candidate controls → render future videos → evaluate object state, collision risk, and instruction compliance → execute a short action chunk → replan.
- Dependencies: fast inference, calibrated uncertainty, reliable candidate ranking, and physical consistency over longer horizons.
- World-model-based reinforcement learning and autonomous skill discovery — Academia
- Use the simulator to generate imagined trajectories for policy optimization, value learning, or autonomous exploration.
- The current method emphasizes DAgger-style recovery around expert demonstrations; future work could extend it to broader model-based reinforcement learning.
- Dependencies: compounding model error, reward specification, distribution shift, and the risk that policies exploit simulator artifacts rather than real-world dynamics.
- Autonomous data collection and iterative self-improvement — Industrial robot fleets
- Deploy a policy across many robots, collect real failures and corrections, update the simulator with those observations, and generate improved recovery supervision for the next policy version.
- This could create a fleet-learning loop for warehouses, factories, or service environments.
- Potential system: fleet telemetry → failure clustering → simulator adaptation → synthetic recovery generation → filtered policy update → staged real-world validation.
- Dependencies: secure data pipelines, versioned models, fleet-wide calibration, reliable failure labeling, rollback mechanisms, and strict safety governance.
- Robotic surgery, rehabilitation, and clinical assistance — Healthcare
- In carefully constrained settings, action-conditioned visual prediction could support rehearsal, error recovery, or assistance for repetitive manipulation procedures.
- Human demonstrations could provide broad procedural coverage, while robot-specific data would ground executable control.
- Dependencies: substantially higher accuracy and explainability requirements; certified hardware, anatomical and tissue modeling, clinical trials, regulatory approval, and human oversight are mandatory. The paper’s results do not establish readiness for autonomous medical use.
- Warehouse and logistics robots operating under unexpected disturbances — Logistics
- Train systems to recover from package displacement, occlusions, grasp failures, conveyor timing errors, and other deviations from nominal demonstrations.
- Use simulated recovery trajectories to supplement rare real failure data.
- Dependencies: accurate modeling of rigid-body contacts, packaging variation, conveyor dynamics, multi-agent interactions, and safety around human workers.
- Energy and infrastructure inspection robots — Energy and utilities
- Transfer skills from human inspection videos to robots that manipulate valves, switches, panels, or tools while using predicted visual consequences to avoid damaging infrastructure.
- Recovery training could address misalignment, partial occlusion, or failed contact during inspection and maintenance.
- Dependencies: domain-specific visual coverage, strict operational safety, weather and lighting robustness, accurate force/contact sensing, and certification for critical infrastructure.
- Interactive educational and research platforms — Education
- Create laboratory systems in which students can teach a robot through demonstrations, inspect predicted futures, intentionally introduce deviations, and study how recovery policies are learned.
- The framework offers a concrete teaching example for multimodal learning, imitation learning, world models, and sim-to-real transfer.
- Dependencies: simplified interfaces, transparent diagnostics, safe hardware, reproducible datasets, and curricula that do not require industrial-scale compute.
- Policy and standards for learned robot simulation — Public policy and safety
- Develop evaluation standards requiring learned world models to report prediction fidelity, uncertainty, recovery validity, and simulation-to-reality performance before synthetic data are used for control.
- Policies could distinguish between simulators used for visualization, offline planning, policy training, and safety-critical decision-making.
- Dependencies: shared benchmarks, representative real-world test suites, auditable data provenance, and agreement on acceptable error rates.
- Everyday personal robotics with privacy-preserving learning — Daily life
- Train domestic robots from egocentric video while using local or federated learning to avoid sending private household recordings to centralized servers.
- The data-pyramid strategy could combine public pretraining, user-specific bridge data, and small amounts of household interaction.
- Dependencies: privacy-preserving perception and storage, consent management, robustness to highly varied homes, secure model updates, and mechanisms to prevent unsafe behavior learned from noisy demonstrations.
- Synthetic corrective supervision for software agents in embodied interfaces — Software and human-computer interaction
- Adapt the deviation–recovery principle to agents that operate visual interfaces, simulations, or physical-digital workflows: generate off-nominal states, simulate corrective actions, and train the agent to return to a valid task trajectory.
- Examples include recovering from an incorrect UI selection, a failed database workflow, or a partially completed automation task.
- Dependencies: the environment must be sufficiently simulatable, state transitions must be observable, and synthetic corrections must correspond to valid real-system outcomes rather than merely plausible screens.
Overall, the most immediately credible applications are robot-learning data reduction, recovery-policy augmentation, pre-deployment simulation, and robust closed-loop manipulation. Broader claims—general-purpose household autonomy, safety-critical control, and fully autonomous self-improvement—depend on improved physical fidelity, uncertainty calibration, cross-embodiment transfer, and extensive real-world validation beyond the experiments described in the paper.
Glossary
- Action-conditioned simulation: Simulation in which future visual states are predicted from specified actions. “action-conditioned robot video generation”
- Action horizon: The number of future action steps predicted by a policy. “ is the action horizon”
- Action projection: A learned mapping that converts embodiment-specific action representations into shared model dimensions. “embodiment-specific action projections accommodate human and robot action spaces”
- Action schema: A standardized representation of movements shared across different embodiments. “a common kinematic schema comprising end-effector poses, hand articulation, and torso/camera poses”
- Action Transformer: A Transformer module that predicts robot actions from visual features. “an Action Transformer reads its multi-level visual features to predict executable actions”
- Affine flow matching: A generative-training method that learns velocity fields connecting noise distributions to data distributions. “We train video and action prediction with flow-matching losses”
- Autoregressive inference: Sequential prediction in which each generated output becomes context for subsequent predictions. “autoregressive inference conditions on previous predictions”
- Behavior cloning: Imitation learning that trains a policy to reproduce demonstrated behavior. “policy behavior cloning primarily covers successful expert demonstrations”
- Bridge data: Data designed to align human demonstrations with robot tasks and visual environments. “Task- and appearance-aligned bridge data”
- Causal attention: Attention restricted so that predictions can access only permitted earlier context. “We use chunk-wise causal attention”
- Closed-loop control: Control in which actions are repeatedly updated using new observations of the system. “We deploy \modelname{} for closed-loop control of the IRON robot”
- Coarse-to-fine curriculum: A training strategy that progresses from broad or less specific supervision to more target-specific supervision. “a coarse-to-fine curriculum progressively emphasizes robot supervision”
- Cross-embodiment alignment: The alignment of representations or actions across physically different agents. “human bridge data supporting cross-embodiment alignment”
- Cross-embodiment representation: A shared representation intended to work across different bodies or robot platforms. “encourages cross-embodiment representations”
- DAgger: An imitation-learning method that iteratively aggregates expert labels for states visited by the learner. “DAgger aggregates expert labels on learner-visited states”
- DAgger-style recovery augmentation: The use of synthesized recovery examples in an imitation-learning pipeline inspired by DAgger. “following DAgger-style recovery augmentation rather than on-policy expert querying”
- Data distribution shift: A mismatch between the data distribution used for training and that encountered during deployment. “to address imitation-learning distribution shift”
- Data pyramid: A hierarchical organization of datasets with progressively more task- and robot-specific supervision. “The resulting data pyramid connects broad interaction experience with robot-specific supervision”
- Deviation–recovery trajectory: A trajectory that intentionally moves away from an expert path and then returns to successful behavior. “synthesize deviation--recovery trajectories around expert demonstrations”
- Diffusion model: A generative model trained to transform noise into structured data through a denoising process. “Diffusion Policy models multimodal action sequences through conditional denoising”
- Distribution matching: An objective that makes generated samples resemble a target distribution, without necessarily matching individual target examples. “Rather than adopting a distribution-matching objective such as DMD”
- Embodiment: The physical body, morphology, and action capabilities of an agent. “adapting the shared representation to the deployment embodiment”
- Embodiment-specific projection: A mapping customized to a particular robot or human body while using shared model components. “Each embodiment has its own linear input projection and final output projection for actions”
- Egocentric video: Video recorded from the viewpoint of the acting person or robot. “broad egocentric video for learning visual dynamics”
- End effector: The terminal tool or body part that directly interacts with the environment, such as a hand or gripper. “end-effector poses, hand articulation, and torso/camera poses”
- Expert manifold: The set of states or trajectories associated with successful expert behavior. “return consistently to the expert manifold”
- Flow matching: A generative-learning technique that trains a neural network to predict transport velocities between noise and data. “context-aware flow matching against the corresponding recorded targets”
- Foundation model: A broadly pretrained model that can be adapted to multiple downstream tasks. “Robot foundation models connect perception and control”
- Forward pass: The computation that maps model inputs to outputs during inference or training. “Information flows from video to action in the forward pass”
- Gradient detachment: A procedure that prevents gradients from propagating through a particular tensor or computation. “we detach the generated prefix and re-encode it with gradients”
- Gradient tracking: Recording operations so that derivatives can be computed during optimization. “The complete rollout is executed without gradient tracking”
- Heterogeneous supervision: Training signals that differ in type, source, or degree of action annotation. “a framework that combines heterogeneous supervision”
- Imitation learning: Learning a policy by reproducing behavior demonstrated by an expert. “DAgger-style recovery augmentation rather than on-policy expert querying”
- Inverse dynamics: A model that infers the action required to produce a transition between states. “recovers pseudo-actions with latent-action or inverse-dynamics models”
- Kinematic schema: A structured representation of body positions, orientations, and articulated motion. “a common kinematic schema comprising end-effector poses, hand articulation, and torso/camera poses”
- Latent action: An internal, learned representation of an action that may not correspond directly to executable robot commands. “DreamDojo uses continuous latent actions”
- Latent frame: A compressed representation of a video frame in a learned latent space. “A frozen video VAE encodes each clip into an initial anchor latent”
- Manifold: A structured space representing a set of states or configurations with meaningful geometric relationships. “return consistently to the expert manifold”
- Mixture-of-transformers: An architecture composed of multiple Transformer modules with specialized functions. “an asymmetric mixture-of-transformers (MoT) architecture”
- Multimodal action sequence: An action trajectory representing multiple plausible ways to perform a task. “Diffusion Policy models multimodal action sequences”
- Multi-level visual features: Features extracted at several layers or resolutions of a visual network. “The 15 action blocks read spatially dense features from video blocks 16--30”
- On-policy expert querying: Obtaining expert supervision on states visited by the currently deployed policy. “rather than on-policy expert querying”
- Optical flow: The apparent motion of image pixels between consecutive video frames. “FlowWAM uses optical flow as a shared motion representation”
- Out-of-distribution transfer: Generalization to tasks, states, or data categories not represented in the training distribution. “analyze transfer across task categories”
- Policy fine-tuning: Further training of a policy to improve performance on a target setting or augmented dataset. “fine-tune a separate policy on filtered recoveries and expert data”
- Policy rollout: A sequence of actions and resulting states generated by executing a policy. “real policy rollouts”
- Predictive representation: A learned representation organized to support prediction of future observations or outcomes. “a shared predictive representation”
- Pseudo-action: An inferred or synthesized action label rather than a directly recorded executable command. “recovers pseudo-actions with latent-action or inverse-dynamics models”
- Recovery supervision: Training information showing how to correct deviations and return to successful behavior. “the simulator generates recovery trajectories that provide additional supervision”
- Replay mixture: A retained combination of earlier data sources included during later training. “a small replay mixture of human demonstrations, bridge data, and action-unlabeled video”
- Self-gradient forcing: A training procedure that adapts a model to contexts generated by its own predictions while using gradients to update context representations. “We adopt self-gradient forcing (SGF) to adapt the simulator”
- Self-conditioned rollout: A generated sequence in which later predictions are conditioned on the model’s own earlier outputs. “Self-conditioned rollout generation”
- Skeleton control: A control signal represented as a sequence of body-joint or keypoint configurations. “prescribed skeleton controls and camera poses”
- Skeleton-conditioned simulation: Simulation in which future frames are generated according to a specified body skeleton sequence. “Skeleton- and camera-conditioned simulation”
- Teacher forcing: Training a sequence model using ground-truth previous outputs as context instead of its own predictions. “Stages~I and~II use teacher forcing with recorded visual context”
- Temporal coherence: Consistency of motion and appearance across successive video frames. “temporally coherent, multi-view generation”
- Token addition: A conditioning method that adds one sequence of embeddings element-wise to another. “We incorporate skeleton controls through token addition at the Transformer input”
- Trajectory-conditioned simulation: Simulation conditioned on a prescribed sequence of actions or controls. “trajectory-conditioned simulation must reproduce the consequences of the supplied controls”
- Visual dynamics: The patterns governing how visual observations change as an environment evolves or is acted upon. “learn visual dynamics”
- Visual grounding: Connecting language or abstract instructions to relevant visual entities and actions. “visually grounded language generation”
- Visual history: Previously observed frames or latent representations supplied as context for prediction. “Given visual history and a language instruction ”
- World model: A learned model that predicts how an environment changes in response to actions. “We introduce \modelname, a unified embodied world model”
- World simulator: A model that generates predicted future observations under prescribed controls. “a world simulator, predicting future video from history, prescribed skeleton controls, and camera poses”







