Appropriate Role of Generative Video Models in Robotic Manipulation

Determine the appropriate functional role that large generative video models should serve within robotic manipulation systems to effectively leverage their visual predictions given the embodiment gap between humans and robots. The goal is to ascertain how such models should be integrated into manipulation pipelines so their imagined object motions can be translated into executable robot actions.

Background

Generative video models can synthesize plausible interactions conditioned on an initial image and language instruction, capturing intuitive physics and object priors that are valuable for open-world robot manipulation. However, these models predominantly depict human embodiments, creating an embodiment gap between the predicted motions and robot action spaces.

The paper proposes Dream2Flow to bridge video generation and robot control via 3D object flow, arguing that reconstructing and tracking object motion provides a general interface. Nonetheless, the broader question of how video models should be positioned within manipulation systems remains explicitly posed by the authors as unclear.

References

Despite their promise, it remains unclear what role such models should serve in a robot manipulation system.

— Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow  (2512.24766 - Dharmarajan et al., 31 Dec 2025) in Section 1 (Introduction)

Connecting temporal knowledge from video to closed-loop control on the target robot is an important open problem.

— The Embodiment Gap in Robot Foundation Models  (2608.18433 - Domae et al., 19 Aug 2026) in Section 6, Learning Correspondence Across Embodiments

While current models can produce visually plausible output, they are prone to errors in object identity, target configuration, action execution, or temporal continuity; how these errors manifest and the extent to which they may degrade instructional utility remain unclear.

— Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks  (2609.24955 - Wu et al., 21 Sep 2026) in Section 1, Introduction

The disagreement is the result: it localizes the open problem to the rewrite channel itself, which at present can edit only words. Whether a text-only edit can move a generator's physics, or whether the correction must instead enter through the geometric control channel this report builds, is an open question this setup can now measure.

— VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement  (2609.03153 - Xu et al., 2 Sep 2026) in Conclusion, final discussion of the outward channel

Finally, although we evaluate the visual fidelity of synthesized robot observations, their effectiveness for downstream manipulation policy training remains to be explored.

— EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos  (2610.18822 - Wang et al., 13 Oct 2026) in Conclusion and Limitations, paragraph “Limitations”