Papers
Topics
Authors
Recent
Search
2000 character limit reached

Hydra-0: Action Flow for Generalist World Modeling and Control

Published 18 Aug 2026 in cs.RO | (2608.18077v1)

Abstract: We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.

Summary

  • The paper introduces action flow, a visibility-aware image-plane representation that connects executable robot commands with predicted video and supports heterogeneous embodiments such as robot arms, grippers, and human hands.
  • Hydra-0 improves motion prediction over native action conditioning, with its four-step Wan2.2 A14B model achieving 21.84 PSNR, 0.830 SSIM, 5.27-pixel object EPE, and 3.29-pixel gripper EPE, while generating 61.98 FPS under the reported protocol.
  • The paper demonstrates strong transfer and early data efficiency, including a Pearson correlation of 0.96 for open-loop policy success prediction and a proof-of-concept inverse controller that converts desired object flow into executable robot motion.

Problem formulation and central contribution

“Hydra-0: Action Flow for Generalist World Modeling and Control” (2608.18077) addresses a structural problem in action-conditioned video prediction: robot commands are typically represented in embodiment-specific coordinates, whereas the desired prediction is an image-space consequence. Joint-space actions expose robot morphology directly; end-effector commands remain ambiguous across kinematic configurations and embodiments. Consequently, a video model must learn an additional embodiment-dependent mapping from commands to visible motion.

The paper proposes action flow, a kinematically grounded representation of robot motion as visibility-aware image-plane trajectories of points on the acting embodiment. This representation provides a common interface for robot arms, grippers, human hands, and manipulated objects. Hydra-0 uses the same flow representation in two directions. In forward mode, executable robot commands are converted into projected embodiment trajectories and used to predict future video. In inverse mode, desired object trajectories are supplied as task intent, and the model predicts compatible robot motion whose latent features are decoded into executable actions.

The distinction between action flow and generic trajectory conditioning is important. Hydra-0 does not merely provide arbitrary two-dimensional motion prompts to a video generator. During deployment, its embodiment flow is obtained by executing candidate commands through a robot controller and physics simulation, propagating visible robot-surface points through link transforms, and projecting them through calibrated camera models. The resulting condition is therefore tied to a realizable command sequence while remaining visually aligned with the generated observation.

Action-flow representation and data construction

For each tracked point, Hydra-0 stores image coordinates across the prediction horizon together with visibility indicators. The resulting flow condition can describe visible robot links, grippers, human hands, manipulated objects, or unassigned scene tracks. Its embodiment independence follows from expressing trajectories in camera coordinates rather than in joint, Cartesian end-effector, or robot-base coordinates.

The paper supports two construction pathways. When robot geometry and camera calibration are available, visible points are sampled on the robot surface and propagated using recorded or simulated link transforms. At deployment, a candidate action sequence is executed in Isaac Lab; the resulting configurations determine the surface-point trajectories, which are projected using camera intrinsics and extrinsics. Visibility is retained only when projected points are in front of the camera, within image bounds, and consistent with a rendered depth buffer. This produces a causal action condition from the candidate executable action.

For metadata-poor videos, Hydra-0 first obtains dense tracks using a flow tracker and then assigns tracks to the acting embodiment and manipulated object using grounded segmentation masks. This enables training on human and robot interaction videos without robot descriptions or camera calibration. The paper is explicit, however, that this relaxation applies primarily to training-time flow recovery: deployment still requires geometry and calibration to derive action flow from a candidate robot command.

Training samples use four conditioning modes: Embodiment, Object, All, and None. Embodiment is the principal forward-action condition; Object supplies manipulated-object flow or, at inference, desired object flow; All provides a fallback when semantic grounding is incomplete; and None implements conditioning dropout. The canonical mode probabilities are (0.05,0.40,0.40,0.15)(0.05, 0.40, 0.40, 0.15) for None, Embodiment, Object, and All, respectively.

https://nvidia-isaac.github.io/video_to_data/hydra-0/static/images/embodiment_flow_sampling.png

Figure 1: Action-flow construction from dense tracks and sampling through Embodiment, Object, All, and None conditioning modes.

The multi-embodiment corpus contains 1,565,634 filtered five-second windows, corresponding to approximately 2,201.7 hours of video across 178,187 episodes. It combines single-arm robot data, bimanual teleoperation, human-hand manipulation, handheld-gripper demonstrations, and cloth-folding data. The filtering procedure removes static windows, frozen-gripper windows, and selected DROID episodes with contentless language annotations. The corpus is intentionally concentrated on deformable-object interaction, including cloth, cable, rope, bags, and paper.

Motion-conditioned video prediction

Hydra-0 builds on pretrained Cosmos 2.5 and Wan2.2 video-generation backbones. The initial RGB observation is encoded into a spatial latent state. For every trajectory, the model bilinearly samples a source feature from the initial latent and propagates that feature along the trajectory through the future latent grid using localized Gaussian weights. A presence gate records the accumulated trajectory mass, allowing the backbone to distinguish regions containing propagated motion features from untouched visual context.

At each latent destination, Hydra-0 retains the two trajectories with the largest raw Gaussian weights. The implementation uses a Gaussian locality parameter of β=220\beta=220 and does not normalize the propagated features by a softmax or weight sum. This design preserves both spatial locality and an explicit indication of whether a location is covered by a valid track. The resulting motion condition is injected into the video backbone alongside the noisy future latent.

https://nvidia-isaac.github.io/video_to_data/hydra-0/static/images/motion_conditioning.png

Figure 2: First-frame latent features are propagated through time along action-flow trajectories and combined with a visibility-derived presence gate.

The architecture is deliberately portable across backbones. In Cosmos 2.5, the propagated features and presence gate are projected through a zero-initialized side pathway. In Wan2.2 I2V-A14B, the condition is integrated into the existing visual-conditioning tensor; in Wan2.2 TI2V-5B, a separate motion side input is concatenated with the noisy latent. The backbone weights are largely frozen, while the model trains the DiT patch embedding and rank-64 LoRA modules on attention and feed-forward projections.

For long-horizon generation, the paper converts the model to a causal autoregressive formulation. The future latent sequence is partitioned into chunks, generated chunks replace clean history, and a KV cache reuses previous context. Crucially, the flow condition is computed once in full-window coordinates and sliced at absolute chunk offsets. This prevents trajectory re-anchoring at chunk boundaries. DMD2-based few-step distillation subsequently reduces each chunk to four denoising steps.

Predictive performance across embodiments and backbones

The central controlled comparison replaces Cosmos 2.5’s native relative 6D end-effector action representation with action flow while holding the dataset mixture and backbone family fixed. The reported average metrics favor action flow across image fidelity, motion accuracy, and distributional quality.

Model PSNR SSIM Object EPE Gripper EPE FID FVD VLM
Cosmos 2.5 baseline 15.62 0.668 13.23 34.28 39.1 405.8 3.88
Hydra-0, Cosmos 2.5 18.41 0.725 6.27 13.80 32.4 277.4 3.83
Hydra-0, Wan2.2 5B 19.64 0.770 6.61 3.88 24.1 248.8 3.90
Hydra-0, Wan2.2 A14B 20.76 0.805 6.00 3.83 20.7 193.7 3.98
Hydra-0, Wan2.2 A14B, four-step 21.84 0.830 5.27 3.29 18.7 155.9 4.23

The strongest four-step Wan2.2 A14B configuration improves the displayed average object-flow endpoint error from 13.23 to 5.27 pixels and gripper-flow endpoint error from 34.28 to 3.29 pixels relative to the native-action Cosmos baseline. The abstract reports these differences as 90.40% lower robot-motion error and 60.16% lower object-motion error than the action-conditioned baseline, although the exact percentage depends on the paper’s aggregate error definition rather than the simple ratio of the displayed average EPE values.

The controlled Cosmos comparison is particularly informative. Action flow improves the displayed point estimates for PSNR, SSIM, gripper EPE, FID, and FVD on all five evaluation datasets, and improves object EPE on all datasets where object tracking is available. The result supports the paper’s claim that image-space action flow communicates visible embodied motion more effectively than native relative 6D actions. The evidence is less uniform for semantic or perceptual judgment: VLM scores are mixed because the judge measures physical plausibility, object permanence, temporal consistency, and motion realism, not exact adherence to commanded trajectories.

The four-step distilled model also improves the displayed metrics over its multi-step counterpart. On the aggregate evaluation, PSNR rises from 20.76 to 21.84, SSIM from 0.805 to 0.830, FVD falls from 193.7 to 155.9, and the VLM score rises from 3.98 to 4.23. These results indicate that distillation is not merely a throughput optimization in the reported setting; it also yields the best measured generation quality. Nevertheless, the paper does not establish that this behavior generalizes beyond the evaluated checkpoints and sampling protocol.

https://nvidia-isaac.github.io/video_to_data/hydra-0/static/images/qualitative_checkpoint_eval_xvla_soft_fold.png

Figure 3: Qualitative comparisons on cloth folding and deformable-object interaction, including trajectory-conditioned baselines, Hydra-0 predictions, and ground truth.

Hydra-0 also produces qualitatively coherent predictions for DROID wrist-camera observations by combining interaction motion with camera egomotion. This experiment is only a proof of concept. It does not demonstrate systematic robustness to substantial camera motion, mobile manipulation, or broader wrist-mounted-camera distributions.

Data efficiency and inference speed

The IWS experiments test whether multi-embodiment action-flow mid-training transfers to held-out tasks. The IWS data are excluded from mid-training, and task-specific adaptation uses matched nested subsets at fractions from 1% to 100%. The mid-trained model, denoted Ours (MT), outperforms the directly pretrained model, Ours (PT), at the zero-data point on LPIPS, object-flow EPE, and FVD for all six tasks.

The transfer advantage persists after adaptation. At 100% of the task-specific data, Ours (MT) has the lowest LPIPS and FVD on all six tasks and the lowest flow EPE on four. Most gains occur by the 20% data fraction: from 20% to 100%, the reported per-task changes are at most 3.4% for LPIPS, 6.7% for flow EPE, and 6.8% for FVD. The implication is specific and substantial: multi-embodiment mid-training can reduce the amount of task-specific data required to obtain a competent action-flow-conditioned world model.

The paper appropriately qualifies the FVD interpretation. Each FVD estimate uses only 40 clips per point and exhibits high variance, including small non-monotonic changes at larger data fractions. Thus, the apparent plateau after 20% is evidence of early saturation in this benchmark, not a statistically established convergence law.

For causal rollout, autoregressive conversion reduces generation time from 20.92 seconds to 12.48 seconds per 81-frame clip on one H100, a 1.68×1.68\times speedup. Four-step distillation reduces latency further to 1.31 seconds, corresponding to 61.98 FPS and a reported 16.0×16.0\times generation-only speedup.

Sampling stage Seconds per clip FPS Relative speed
Bidirectional teacher 20.92 3.87 1.0×1.0\times
Autoregressive teacher 12.48 6.49 1.68×1.68\times
Four-step student 1.31 61.98 16.0×\times

These measurements exclude classifier-free guidance, VAE decoding, and pixel-level postprocessing. They also use a short three-chunk horizon for which the long-rollout attention cap is inactive. The result therefore establishes fast generation under the stated benchmark protocol, not end-to-end control-loop latency.

Open-loop policy evaluation

Hydra-0 evaluates policies in forward mode by conditioning generation on the complete recorded gripper trajectory. The policy is not queried on generated observations, so the experiment measures open-loop outcome preservation rather than interactive closed-loop behavior. The protocol covers 300 RoboLab episodes: five policies, six tasks, and ten rollouts per policy-task pair.

The generated success rates correlate strongly with reference environment success rates. The paper reports Pearson correlation r=0.96r=0.96, Spearman correlation ρ=0.93\rho=0.93, and a mean absolute error of 5.7 percentage points. Averaged across tasks, Hydra-0 also preserves the ranking of all five policies.

https://nvidia-isaac.github.io/video_to_data/hydra-0/static/images/policy_evaluation_correlation.png

Figure 4: Hydra-0-generated success rates closely track reference success rates across RoboLab policy-task aggregates.

The result supports Hydra-0 as an open-loop policy-evaluation model, particularly for ranking policies and reproducing aggregate success statistics. It does not show that generated observations are sufficiently faithful for closed-loop policy execution, because the action sequence is fixed and no policy feedback is introduced during generation. The real-world cloth-folding replay further provides qualitative evidence that successful and failed recorded trajectories can produce distinguishable predicted outcomes, but it is not a quantitative validation of real-world policy ranking.

Inverse control from desired object flow

The inverse mode is the paper’s strongest conceptual extension. Instead of conditioning on embodiment flow, the world action model receives desired object flow extracted from a held-out human demonstration. The model predicts latent dynamics containing compatible robot motion, and an embodiment-specific action head maps pooled DiT features to robot actions and states.

Training uses paired real-world rollouts, including successes and failures. These demonstrations need not be expert trajectories because each rollout associates observed object motion with the robot actions that generated it. The action and state heads are trained with masked Huber losses, while a velocity loss regularizes temporal action differences. Rank-32 LoRA adapters, the motion projection, and lightweight readout heads are optimized during this post-training stage.

https://nvidia-isaac.github.io/video_to_data/hydra-0/static/images/policy_learning_pipeline.png

Figure 5: Desired object flow transferred from a human demonstration is converted into compatible robot motion and executed on a flexible-pipe-bending task.

The real-robot demonstration shows a flexible-pipe-bending task in which only desired object flow is supplied. Hydra-0 generates compatible robot motion without embodiment-flow input, and the action head produces executable commands. This result supports the claim that action flow can separate task specification, expressed in object coordinates, from embodiment-specific execution.

The evidence remains proof-of-concept rather than a complete inverse-control evaluation. The paper reports one demonstrated task and acknowledges centimeter-scale grasp imprecision. The result therefore establishes feasibility of object-flow-conditioned action decoding, but not broad task generalization, robust contact handling, or performance under perturbations.

Limitations and open questions

Hydra-0 depends on calibrated geometry and camera models when converting candidate executable commands into deployment-time action flow. Calibration errors perturb the visual condition; sufficiently large projection errors can destroy correspondence between commanded motion and generated motion. Unknown object occlusions are not explicitly removed from the depth-based visibility test, and the method may therefore receive imperfect embodiment tracks in cluttered scenes.

The inverse model has unresolved depth and contact ambiguities. The paper reports centimeter-scale grasp errors and cases in which it is unclear whether an object has been successfully grasped. Since action flow is fundamentally a two-dimensional representation, depth, force, and tactile state are only indirectly represented. This is a direct limitation for manipulation behaviors whose success depends on contact geometry rather than visible image motion.

The evaluation is also concentrated on deformable-object interaction and open-loop prediction. The wrist-camera experiment is qualitative, the inverse-control experiment is a single flexible-pipe-bending demonstration, and closed-loop policy evaluation is not performed. The principal open question is whether the same interface remains reliable when generated observations feed back into a policy, particularly under compounding model error, occlusion, contact transitions, and camera motion.

Conclusion

Hydra-0 presents action flow as a shared visual interface between heterogeneous robot commands, observed interaction dynamics, and task-level object motion. Its forward model improves motion fidelity over native action conditioning across multiple embodiments and video backbones, while multi-embodiment mid-training yields strong zero-shot transfer and reduced task-specific data requirements. Causal conversion and distillation provide 61.98 FPS generation under the reported generation-only protocol. In RoboLab, open-loop predictions reproduce policy success rates with Pearson correlation r=0.96r=0.96.

The inverse experiment extends the same interface from prediction to control: desired object flow from a human demonstration is converted into compatible robot motion and executable actions without task-specific expert robot demonstrations. The current evidence supports action flow as a technically coherent interface for forward world modeling, open-loop evaluation, and preliminary inverse control, while leaving depth-aware contact reasoning, closed-loop validation, and systematic real-world generalization unresolved.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

This paper introduces Hydra-0, an AI system designed to help robots understand and predict what will happen when they move.

A robot usually describes its actions using special commands, such as:

  • moving a joint by a certain amount,
  • moving its hand to a particular position, or
  • opening and closing its gripper.

The problem is that different robots have different body shapes and moving parts. A command that works for one robot may mean something completely different for another.

Hydra-0 uses a more universal description: how the robot’s movement appears in the camera image. For example, instead of saying “rotate joint 2,” it describes, “the gripper moves 30 pixels to the left.”

The researchers call this description action flow.

2. What questions did the researchers ask?

The paper mainly investigates whether action flow can help one robot-learning system work with many different types of robots and tasks.

The researchers wanted to know:

  • Can robots with different bodies be described using the same visual movement language?
  • Can Hydra-0 predict what will happen after a robot moves?
  • Can it learn from videos of robots, grippers, and even human hands?
  • Can it predict whether a robot action is likely to succeed before the robot performs it?
  • Can it use a human demonstration to figure out how a robot should move?
  • Does this approach require less robot-training data than older methods?

In simple terms, the researchers were asking:

Can a robot learn to understand actions by watching how things move on screen, rather than memorizing the special controls of one particular robot?

3. How did the researchers build and test Hydra-0?

Representing movement as image flow

Hydra-0 watches a video and tracks points on moving objects. These points might be:

  • parts of a robot arm,
  • a gripper,
  • a human hand, or
  • an object being moved.

The system records where each point appears in each frame. If a point moves from one place to another, Hydra-0 creates a path for it.

This is similar to drawing arrows on a video to show where different parts are going. These paths are called action flow.

Because the paths are written in camera-image coordinates, the same system can describe a human hand, a small gripper, or a large robotic arm.

Learning from many kinds of videos

The researchers trained Hydra-0 using about 1.6 million video clips, totaling about 2,200 hours of interaction data. The data included:

  • single-arm robots,
  • two-arm robots,
  • human hands,
  • handheld grippers,
  • robots folding clothes,
  • robots handling ropes, bags, paper, and other flexible objects.

Some videos included detailed information about the robot’s shape and camera position. For other videos, the researchers used computer vision tools to track movement directly from the images.

Predicting future video frames

Hydra-0 receives:

  1. an image showing the starting situation, and
  2. the action flow showing how the robot will move.

It then predicts future video frames.

This is like asking:

“If the gripper follows this path, what will the cloth, rope, or other object look like a few seconds later?”

The model is based on large video-generation systems. These systems have learned general patterns about how videos usually change over time. The researchers adapted them so that the predicted changes follow the robot’s action flow.

Testing the forward direction

In its forward mode, Hydra-0 receives a possible robot movement and predicts its result.

For example:

  • the gripper moves toward a piece of cloth,
  • the cloth bends or folds,
  • the robot may succeed or fail.

The researchers compared Hydra-0’s predictions with real recorded videos. They measured how closely the predicted robot and object movements matched reality.

They also used it to evaluate robot policies. A policy is a set of rules or a learned strategy telling a robot what to do. Instead of testing every policy on a physical robot, Hydra-0 can replay the planned movements in its predicted video world.

Testing the inverse direction

Hydra-0 also has an experimental inverse mode.

Here, the system is given the desired movement of an object, such as:

  • a cloth moving into a folded shape, or
  • an object sliding to a target location.

It then tries to predict how the robot should move to cause that result.

The researchers even transferred object movement from a human demonstration to a robot. The human might fold a piece of cloth, and Hydra-0 would use the visible object motion to help a robot perform a similar action.

4. What did the researchers find?

Hydra-0 predicted movement more accurately

The best version of Hydra-0 produced much smaller errors than a baseline system that used traditional robot commands.

According to the paper, it achieved:

  • 90.4% less error when predicting robot movement,
  • 60.2% less error when predicting object movement.

This means the predicted videos followed the intended robot and object paths much more closely.

Hydra-0 also generally produced clearer and more realistic future video frames than the comparison systems.

It worked across different robots and video models

The researchers tested Hydra-0 with several video-generation systems, including versions of Cosmos 2.5 and Wan2.2.

The results suggest that action flow is not limited to one particular video model. It can act as a general connection between robot actions and video prediction.

This is important because researchers could potentially improve or replace the video-generation part without completely redesigning the action interface.

It helped evaluate robot policies

On the RoboLab benchmark, Hydra-0 compared predicted policy success with real reference results.

The correlation was r=0.96r=0.96, which is very high. This means that policies predicted to perform well usually did perform well, and policies predicted to perform poorly usually performed poorly.

This could allow researchers to test many robot strategies in a virtual prediction system before trying them on expensive and sometimes fragile physical robots.

It could learn from different types of demonstrations

Hydra-0 was trained using videos involving robots with different bodies as well as human hands.

This suggests that the model can share information between different forms of interaction. For example, it may learn general ideas about pushing, pulling, folding, and grasping even when the person or robot performing the action is different.

It showed promise for controlling robots from human examples

In inverse mode, Hydra-0 used desired object movement from a human demonstration to predict compatible robot movement.

The robot did not need task-specific expert demonstrations for this experiment. Instead, it learned from ordinary paired examples showing robot actions and their results, including both successful and unsuccessful attempts.

This was only a proof of concept, but it suggests a possible way for humans to teach robots by demonstrating what should happen rather than specifying every robot movement.

5. Why is this research important?

Robots today often need large amounts of special training for each robot model, task, and environment. A system trained only on one robot may not work well with another robot because their bodies and controls are different.

Hydra-0 offers a shared visual language for movement. Since cameras can observe many different robots and human actions, action flow could make it easier to:

  • train one model using many kinds of data,
  • transfer knowledge from humans to robots,
  • adapt a robot to a new task using less data,
  • predict the results of actions before performing them, and
  • test robot strategies more safely in a learned virtual world.

However, the system is not yet a complete solution. Its predictions can still be wrong, especially when the camera view, robot position, or object movement is complicated. Flexible objects such as cloth and rope are also difficult to model accurately. The inverse-control experiments were early demonstrations rather than a full solution for every task.

Overall, the paper shows that describing actions as visible movement in a video may help robots learn in a more general and flexible way. Hydra-0 could be an important step toward robots that learn from many different robots, people, and environments instead of needing to be trained from scratch each time.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

  • Limited real-world validation: Most quantitative prediction results are reported on offline video datasets or simulation, leaving unclear how reliably Hydra-0 predicts outcomes during sustained, closed-loop physical interaction.
  • Small-scale real-robot control evidence: The inverse-control demonstration appears to be a proof of concept, but the paper does not establish success rates, failure modes, repeatability, or robustness across many tasks, objects, environments, and robot embodiments.
  • Unclear scale of inverse-control adaptation: The amount, diversity, and collection procedure of the paired real-world rollouts used to train the world action model are not sufficiently characterized, making its data efficiency difficult to assess.
  • No direct comparison with strong inverse-control baselines: The real-robot control experiments do not clearly compare against behavior cloning, visuomotor policies, trajectory optimization, diffusion policies, or other methods that infer robot actions from demonstrated object motion.
  • Dependence on privileged deployment information: Geometry-aware action flow requires robot geometry, controller-and-physics rollouts, camera calibration, depth rendering, and visibility testing. The paper does not quantify how performance degrades when these inputs are inaccurate or unavailable.
  • Simulation-to-real mismatch is unresolved: Deployment flow is generated through Isaac Lab, whereas the target video reflects real-world execution. The sensitivity of predictions to discrepancies in dynamics, controller behavior, contact, friction, compliance, and latency is not systematically evaluated.
  • Calibration robustness is not quantified: The paper states that moderate calibration errors perturb the condition and large errors degrade correspondence, but it does not provide controlled experiments over camera intrinsic, extrinsic, depth, or robot-model perturbations.
  • Occlusion and visibility limitations remain unexplored: Action flow is defined only for visible or successfully tracked points. The model’s behavior when the robot or manipulated object is heavily occluded, exits the image, self-occludes, or reappears is not systematically studied.
  • Tracking and segmentation errors are not isolated: Training relies on AllTracker trajectories and automatically generated embodiment/object masks, but the contribution of tracking drift, incorrect visibility labels, and semantic-mask errors to final performance is not measured.
  • Object annotations are incomplete across datasets: Object-flow metrics cannot be reported for DROID, and object masks are unavailable for DROID and EgoDex during preprocessing. This limits evaluation of task-relevant motion across the full multi-embodiment corpus.
  • Human-hand and robot-motion equivalence is assumed rather than demonstrated: Although human hands, grippers, and robot links share the same pixel-space representation, the paper does not determine whether differences in appearance, articulation, contact mechanics, or motion style cause systematic transfer failures.
  • Insufficient embodiment diversity for broad generalization claims: The corpus is dominated by a few robot types and manipulation settings, with only limited humanoid and handheld-gripper data. Transfer to mobile manipulators, aerial robots, articulated tools, substantially different cameras, or unseen kinematic structures remains unknown.
  • Limited task and object diversity: The experiments emphasize deformable-object manipulation and selected folding, rope, cloth, and grasping tasks. Generalization to rigid-body assembly, tool use, articulated objects, navigation, locomotion, liquids, transparent objects, and dynamic environments is not established.
  • Long-horizon stability is insufficiently evaluated: The model is trained primarily on approximately five-second windows, while autoregressive conversion is intended for long-horizon rollout. Error accumulation, scene drift, action-flow re-anchoring, and degradation over substantially longer horizons are not reported.
  • Closed-loop control performance is unknown: The evaluation uses fixed achieved trajectories for open-loop replay. It remains unresolved whether model predictions are accurate enough to support replanning or feedback control when actions are selected based on generated future observations.
  • Open-loop policy evaluation may not measure policy quality independently: Replaying a complete recorded gripper trajectory can conceal the policy’s online decision-making and recovery behavior. The reported correlation with success rates therefore may not generalize to interactive evaluation under perturbations.
  • The reported policy-evaluation correlation is statistically underpowered: A Pearson correlation of r=0.96r=0.96 is computed across only five aggregated policies. Confidence intervals, significance testing, per-episode analyses, and robustness to alternative metrics are not provided.
  • No calibration of predictive uncertainty: The generative model may produce visually plausible but physically incorrect futures, yet the paper does not assess uncertainty, multimodality, confidence calibration, or mechanisms for detecting unsafe predictions.
  • Physical validity is incompletely assessed: PSNR, SSIM, FID, FVD, VLM scores, and flow EPE do not fully test conservation, contact consistency, object rigidity or deformability, collision avoidance, force plausibility, or 3D geometric correctness.
  • Pixel-space flow may be ambiguous: Different 3D motions can induce similar image-plane trajectories, particularly under depth variation, occlusion, camera motion, or object deformation. The paper does not analyze when 2D action flow is insufficient to identify the underlying physical consequence.
  • Camera-motion handling is only a qualitative proof of concept: Wrist-camera egomotion is demonstrated qualitatively, but quantitative results are not provided for moving cameras, substantial viewpoint changes, rolling-shutter effects, or simultaneous camera and object motion.
  • Action-flow density and sampling choices are not fully justified: The effects of the number of tracks, top-KK aggregation, Gaussian parameter β\beta, visibility rules, track-selection modes, and flow resolution are not comprehensively ablated.
  • Conditioning-mode interactions are unclear: The training uses None, Embodiment, Object, and All modes, but the paper does not isolate their individual contributions or determine whether object-flow conditioning genuinely causes inverse reasoning rather than exploiting correlated visual cues.
  • The emergent inverse mode may rely on dataset-specific correlations: It is unclear whether desired object flow alone is sufficient for action inference, or whether the model uses scene appearance, object identity, language, camera viewpoint, or memorized task structure to infer the robot motion.
  • Ambiguous inverse solutions are not addressed: A desired object trajectory may be achievable by multiple robot motions, or may be infeasible. The world action model’s behavior under multiple valid solutions, infeasible goals, conflicting constraints, or unobserved obstacles is not characterized.
  • Action-head generalization across target embodiments is untested: The action readout is embodiment-specific, and the paper does not show whether a single model can support multiple action heads, adapt to a new robot with minimal data, or transfer to an embodiment absent from training.
  • No systematic study of action-space normalization and temporal alignment: The effects of per-dimension normalization, action/state alignment, control frequency, latency, and the velocity smoothness loss on executable behavior are not reported.
  • The role of the video backbone is confounded with model scale and architecture: Comparisons among Cosmos 2.5 and Wan2.2 variants differ in parameter count, backbone design, conditioning pathway, and denoising schedule, making it difficult to attribute gains solely to action flow.
  • Baseline comparisons are not fully controlled: ATI and Wan-Move are evaluated using released checkpoints without equivalent multi-embodiment mid-training, while Hydra-0 receives extensive corpus-specific training. The relative benefit of the representation versus training scale, preprocessing, and optimization is therefore unresolved.
  • Ablations against alternative visual action representations are missing: The paper does not compare action flow with optical flow, dense trajectories, keypoint trajectories, depth-aware flow, 3D point tracks, segmentation-conditioned motion, or learned action tokens under matched training conditions.
  • Four-step distillation is not evaluated for control fidelity: Although four-step models improve reported video metrics, the effects of distillation on action adherence, inverse-control accuracy, uncertainty, temporal consistency, and downstream robot success are not established.
  • Computational and deployment costs are incompletely reported: Training resource requirements are given for one configuration, but inference latency, memory use, energy consumption, throughput, and feasibility for real-time control are not quantified.
  • Dataset filtering may bias the learned dynamics: Removing static, frozen-gripper, and contentless-caption windows may discard failure cases, pauses, recovery behaviors, and negative examples that are important for robust world modeling.
  • Failure trajectories are not analyzed in sufficient detail: The paper states that successful and failed rollouts can provide supervision, but it does not report whether failure-inclusive training improves prediction, detects failure, or prevents the model from reproducing unsafe behavior.
  • Language conditioning is underexplored: Captions are generated or inherited from source datasets, but the dependence of predictions and transfer on caption quality, linguistic variation, ambiguity, and language grounding is not evaluated.
  • Generalization to unseen viewpoints and environments remains uncertain: The reported validation sets are drawn from related data sources and task distributions. Cross-camera, cross-lab, cross-background, lighting, workspace, and object-instance generalization require more rigorous held-out tests.
  • Reproducibility is incomplete: The paper does not provide sufficient information about exact train/validation splits, random seeds, annotation quality, hyperparameter searches, failed runs, model-selection criteria, and complete inference settings to independently reproduce all claims.
  • Safety and constraint handling are absent: Hydra-0 does not appear to explicitly enforce joint limits, collision constraints, force limits, human-safety constraints, or task-specific forbidden regions when converting predicted motion into executable commands.
  • The relationship between visual plausibility and executable feasibility is unresolved: A generated trajectory can match image-space motion while requiring impossible, unstable, or unsafe robot actions. The paper does not quantify the rate of physically executable predictions or introduce feasibility verification.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Robot policy evaluation before physical execution — Robotics, manufacturing, and logistics. Hydra-0 can replay a recorded or proposed gripper trajectory as image-plane action flow and generate the likely visual outcome without executing the policy on a real robot. Organizations could build a simulation-based policy screening workflow that ranks candidate policies by predicted object displacement, task completion, collision risk, or deformation quality. The reported Pearson correlation of r=0.96r=0.96 between replayed and reference RoboLab success rates supports this as a useful evaluation signal, although not as a replacement for physical validation. Dependencies: reliable robot-controller simulation, camera calibration, representative training data, and task-specific success detectors. Accuracy may decline under severe occlusion, unseen materials, lighting changes, or contact dynamics not represented in training.
  • Pre-deployment testing of manipulation policies — Industrial automation. Robot vendors can use action-flow-conditioned rollouts to test policies for cloth handling, cable routing, bag manipulation, paper folding, and other deformable-object tasks. A practical workflow would generate multiple predicted futures for a candidate action sequence, reject trajectories with implausible object motion, and send only the safest candidates to the robot. Dependencies: calibrated cameras and robot geometry are required for the geometry-aware deployment route; uncertainty estimation and conservative safety thresholds would be needed for production use.
  • Cross-embodiment transfer of visual-dynamics models — Robotics software platforms. Action flow provides a common pixel-space interface for single-arm robots, bimanual systems, handheld grippers, humanoids, and human hands. Robotics platforms can therefore train one predictive model on heterogeneous video datasets and adapt it to a new robot with less robot-specific data than a native joint- or end-effector-action model would require. The reported reductions in robot-motion and object-motion error indicate a concrete advantage for model initialization and adaptation. Dependencies: the new robot must have sufficient visual overlap with the training distribution, and its visible motion must be projected accurately into the camera plane. Pixel-space equivalence does not guarantee equivalent forces, torques, grasp stability, or actuator limits.
  • Data-efficient adaptation for new manipulation tasks — Robotics research and industry. Hydra-0’s multi-embodiment mid-training can serve as a pretrained checkpoint for new tasks such as pushing, sweeping, grasping, folding, or rope manipulation. Engineering teams could fine-tune only LoRA modules and lightweight task heads using a small fraction of task-specific demonstrations, reducing data collection and labeling costs. Dependencies: the target task must share relevant visual and physical regularities with the pretraining corpus. The paper evaluates data efficiency in simulation and selected domains; real-world gains require additional validation.
  • Learning from videos without robot metadata — Academic robotics and dataset construction. The video-only construction route uses point tracking and grounded embodiment/object masks to create action-flow annotations from videos that lack robot descriptions, joint states, or camera calibration. This enables researchers to convert existing robot, teleoperation, and human-demonstration datasets into a common training format. Potential tool: an automated preprocessing pipeline combining dense tracking, segmentation, visibility estimation, captioning, and latent extraction. Dependencies: tracker and segmentation quality; errors in identity tracking or object masks can produce misleading action conditions. Licensing, privacy, and consent constraints remain important for human videos.
  • Human-demonstration-assisted robot control — Robotics and assistive automation. The inverse world-action model can receive desired object flow extracted from a human demonstration and infer compatible robot motion. This supports workflows in which a person folds cloth, moves a cable, or manipulates a soft object while a robot reproduces the intended object motion, without requiring task-specific expert robot demonstrations. Dependencies: the human and robot must share a suitable camera-space viewpoint or require robust viewpoint transfer; the action head must be adapted to the target robot; safety checks are essential because multiple robot actions may produce similar object flows.
  • Visual trajectory interfaces for robot programming — Industrial and educational robotics. Instead of specifying joint trajectories or robot-specific end-effector commands, users could draw or demonstrate desired pixel trajectories for a gripper or manipulated object. Hydra-0 could translate these trajectories into candidate robot actions or use them to evaluate execution plans. This could simplify programming across heterogeneous robot brands. Dependencies: object-flow-to-action inversion is currently demonstrated as a proof of concept, not a guaranteed general-purpose planner. The interface needs explicit handling of depth ambiguity, kinematic reachability, force requirements, and collision constraints.
  • Rapid prototyping of video-conditioned robotics systems — Robotics academia and software development. Because action flow is portable across Cosmos 2.5 and Wan2.2 backbones and can be injected through parameter-efficient adapters, researchers can experiment with alternative video-generation backbones without redesigning the action representation. This can accelerate benchmarking, ablation studies, and integration with vision-language-action systems. Dependencies: training remains computationally expensive—the strongest configuration used 32 H100 GPUs for five days—and access to compatible video models and datasets may be limited.
  • Robot teleoperation monitoring and post hoc diagnosis — Manufacturing and field robotics. Operators can compare executed robot flow with predicted flow and compare predicted object motion with observed object motion. Large discrepancies could flag camera calibration drift, mechanical wear, grasp failure, tracking errors, or unexpected environmental changes. Dependencies: reliable online tracking and temporal alignment; anomaly thresholds must be calibrated to each robot, camera, task, and material.
  • Research infrastructure for embodied AI — Academia and public-sector laboratories. Hydra-0 provides a shared representation for studying forward prediction, inverse control, open-loop evaluation, multimodal demonstrations, and multi-embodiment transfer. Laboratories can use it as a baseline or common benchmark interface rather than comparing models that use incompatible robot action spaces. Dependencies: reproducible code, accessible checkpoints, standardized evaluation protocols, and careful separation between visual prediction quality and actual physical task success.

Long-Term Applications

  • General-purpose robot foundation models — Robotics, warehousing, and service automation. A scaled Hydra-like model could function as a general predictive simulator for many robot embodiments and environments. A policy could propose actions, the world model could predict several possible futures, and a planner could select actions that maximize task progress while avoiding unsafe outcomes. This would support flexible robots operating across packaging, sorting, household assistance, and soft-object handling. Dependencies: substantially broader data, robust uncertainty and multimodal prediction, long-horizon physical consistency, force and tactile modeling, and real-world safety guarantees. Pixel motion alone cannot represent all relevant state variables.
  • Closed-loop model-predictive control from visual flow — Robotics and autonomous systems. Future systems could repeatedly predict short-horizon consequences, compare them with a desired object flow, execute a small action segment, and replan from the next camera observation. This would extend Hydra-0 from open-loop policy evaluation to closed-loop manipulation under disturbances. Dependencies: sufficiently low inference latency, calibrated action-to-flow conversion, stable long-horizon rollouts, uncertainty-aware replanning, and integration with collision, force, and joint-limit constraints.
  • Cross-robot skill libraries and marketplace interfaces — Industrial automation. Skills could be stored as visual object-flow goals—such as “move the cable into this shape” or “flatten the cloth along this trajectory”—rather than as robot-specific programs. A deployment service could compile the same skill into actions for different robot arms or humanoids. Dependencies: validated embodiment-specific action decoders, standardized camera coordinates, task-level semantics, force-aware planning, and contractual guarantees about performance across robot hardware.
  • Natural-language-to-physical-task systems — Service robotics and human–robot interaction. A LLM could convert instructions such as “fold the towel neatly” into desired object trajectories, while Hydra-0 predicts feasible robot motion and expected outcomes. The system could use predicted videos to ask for clarification, identify ambiguous goals, or present a preview before acting. Dependencies: reliable language grounding, object identity and state estimation, planning beyond visible motion, and mechanisms for human approval and safe failure recovery.
  • Simulation-to-real transfer for deformable-object manipulation — Manufacturing, apparel, and logistics. Action-flow conditioning could help bridge simulated and real interaction data by representing the visible consequences of motion consistently across domains. This may enable training systems for garment folding, textile inspection, flexible packaging, cable assembly, and agricultural produce handling. Dependencies: realistic material and contact simulation, domain adaptation for appearance and cameras, depth or tactile sensing, and validation across diverse deformable materials.
  • Assistive and rehabilitation robots — Healthcare and elder care. Human demonstrations of desired object motion could be converted into robot assistance for dressing, towel handling, meal preparation, or object repositioning. In rehabilitation, the same interface might compare a patient’s intended or achieved motion with a desired trajectory and adapt assistance levels. Dependencies: clinical validation, medical-device regulation, personalized safety constraints, force control, privacy-preserving collection of human demonstrations, and reliable operation under unusual body poses or occlusions.
  • Physical-world digital twins for operations and policy — Industry and government. Aggregated action-flow predictions could support digital twins that estimate how robots, workers, or automated equipment will affect objects and workflows. Policymakers and safety engineers could use such systems to test warehouse layouts, automation strategies, or human–robot interaction protocols before deployment. Dependencies: calibrated uncertainty, validated causal predictions rather than visually plausible generations, representative operational data, and governance for surveillance and worker data.
  • Standardized visual control protocols for robotics — Academia, standards bodies, and industry. Action flow could evolve into an interoperable protocol for describing robot intent, observed motion, and desired object changes. Such a standard could allow datasets, simulators, planners, video models, and robot controllers from different vendors to exchange motion information in a common visual format. Dependencies: agreed coordinate conventions, visibility and occlusion standards, depth representation, temporal resolution, metadata schemas, and demonstrated superiority or complementarity to 3D and joint-space interfaces.
  • Autonomous discovery of manipulation strategies — Scientific and industrial automation. A world model could generate and evaluate candidate object-flow goals, allowing robots to discover new ways to fold, arrange, deform, or assemble objects. This could be useful in materials research, automated laboratory workflows, and manufacturing process optimization. Dependencies: reliable exploration constraints, physical outcome verification, reward or objective specification, handling of irreversible failures, and integration with tactile and force sensing.
  • Everyday consumer tools for demonstration-based automation — Daily life and consumer robotics. A household robot could learn tasks from a short phone or wearable-camera video: for example, demonstrating how to fold laundry, organize objects, or manipulate a flexible package. The robot would infer object motion, adapt it to its embodiment, preview the result, and execute with human confirmation. Dependencies: robust viewpoint and scale transfer, privacy protections, reliable segmentation in cluttered homes, safety around people and pets, affordable onboard compute, and substantially stronger generalization than demonstrated in the paper.
View Paper Prompt  View All Prompts 

Glossary

  • Action flow: A visual representation of robot or object motion as trajectories in image coordinates. “We bridge this gap with action flow, a shared image-plane motion representation that supports both forward dynamics prediction and inverse motion prediction.”
  • Action-conditioned baseline: A comparison model whose predictions are directly conditioned on robot actions. “Our best configuration achieves 90.4\% lower robot-motion error and 60.2\% lower object-motion error than our action-conditioned baseline”
  • Action head: A prediction module that converts learned latent features into executable robot commands. “A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations.”
  • Action space: The set of actions available to an agent or robot. “the same interface can represent robot arms, grippers, and even human hands without exposing their native action spaces to the video model.”
  • Actionable flow: Image-space motion information that corresponds to executable robot behavior. “This controlled comparison isolates the conditioning representation and shows that image-space actionable flow communicates embodied motion more effectively than the native relative 6D action.”
  • Autoregressive conversion: The transformation of a model into one that generates sequential chunks while reusing previously generated context. “We therefore adopt LongLive-2.0's autoregressive conversion”
  • Backbone: The main pretrained neural architecture underlying a model. “We instantiate our model on different video-generation backbones”
  • Bilinear sampling: An interpolation method that estimates a value from the four nearest grid points using distance-based weights. “For each selected trajectory, we then bilinearly sample a source feature”
  • Causal autoregressive model: A sequential generative model that predicts later content using only preceding context. “producing embodiment action flow that conditions the causal autoregressive model to predict the command sequence's real-world consequence.”
  • Camera calibration: The estimation of a camera’s internal parameters and its pose relative to a coordinate system. “We propagate the sampled surface points using these link transforms and project the resulting motion into the camera plane.”
  • Camera egomotion: The apparent image motion caused by movement of the camera itself. “camera egomotion can be projected into image-space flow and combined with interaction motion”
  • Camera extrinsics: Parameters describing a camera’s position and orientation in the world. “TCW\mathbf{T}_{CW} and K\mathbf{K} are the camera extrinsics and intrinsics, respectively.”
  • Camera intrinsics: Internal camera parameters such as focal length and principal point. “TCW\mathbf{T}_{CW} and K\mathbf{K} are the camera extrinsics and intrinsics, respectively.”
  • Conditioning dropout: A training technique that randomly removes a conditioning signal so a model can also operate without it. “None removes the trajectory condition and acts as conditioning dropout”
  • Diffusion time: The time or noise-level variable used during the diffusion-generation process. “Let ZtZ_t denote the noised target-video latent at diffusion time tt”
  • DiT: A diffusion transformer architecture that processes diffusion-model representations with transformer layers. “This condition is concatenated with the noisy video latent ZtZ_t before DiT patch embedding.”
  • Distribution-matching distillation (DMD2): A method for training a smaller or faster generative model to reproduce the distribution of a larger model. “with improved distribution-matching distillation (DMD2)”
  • Embodiment: The particular physical body or robot platform through which an action is performed. “This unified interface enables a single generalist world model to learn from multi-embodiment data”
  • End-effector: The terminal component of a robot, such as a gripper or tool, that interacts with the environment. “the same end-effector command can produce different joint trajectories”
  • Endpoint error (EPE): The distance between a predicted point displacement and its ground-truth displacement. “object and gripper endpoint error (EPE) as motion error”
  • Flow matching: A generative-model training objective that learns a velocity field connecting noise to data. “We optimize the standard flow-matching objective of the video backbone”
  • Foundation world model: A broadly trained predictive model intended to represent general environmental dynamics. “A foundation world model trained on diverse interaction data could serve as a common predictive simulator”
  • Forward dynamics: The prediction of future states resulting from specified actions. “This shared visual interface enables generalist world modeling and control by learning action consequences”
  • Gaussian locality: A weighting mechanism that gives greater influence to features nearer a trajectory location according to a Gaussian function. “where NK\mathcal{N}_K indexes the KK trajectories with the largest raw Gaussian weights and β\beta is the Gaussian locality”
  • Grounded masks: Segmentation masks aligned with semantic entities identified using visual or textual information. “then segment the tracks using grounded masks for the visible embodiment and manipulated objects”
  • Homogeneous coordinates: An augmented coordinate representation that enables translations and projective transformations through matrix multiplication. “$\bar{\mathbf{X}_n$ is the homogeneous surface point”
  • Image-plane flow: Motion represented in the two-dimensional coordinate system of an image. “We formulate robot action conditioning as kinematically grounded image-plane action flow.”
  • Inverse mode: A mode that infers robot motion from a desired outcome or object motion. “We instantiate this inverse mode as a world action model.”
  • Kinematic projection: The computation of an image position from a robot’s configuration, geometry, and camera parameters. “\label{eq:kinematic_projection}”
  • Kinematics: The study and computation of relationships between robot joint configurations and positions or motions. “while retaining the constraints imposed by its kinematics”
  • Latent dynamics: Learned temporal relationships among internal representations rather than directly among pixels. “Desired object flow specifies task-relevant motion and conditions latent dynamics that encode compatible robot motion.”
  • Latent grid: The spatial and temporal discretized coordinate system of a model’s internal representation. “We express the resulting trajectory locations and destination cells in a common normalized latent-grid coordinate system”
  • Latent state: An internal encoded representation of an observation or scene state. “An encoder eϕe_\phi maps the initial observation to a spatial latent state”
  • LoRA (low-rank adaptation): A parameter-efficient fine-tuning method that learns low-rank updates to pretrained weights. “We freeze the pretrained video-backbone weights except for the DiT patch embedding and train rank-64 low-rank adaptation (LoRA) modules”
  • Motion feature propagation: The process of transporting visual features along tracked trajectories over time. “At latent time kk and normalized latent-grid location $\widetilde{\mathbf{p}$, the motion feature is”
  • Motion presence gate: A learned or computed mask indicating where trajectory-propagated visual information is present. “which indicates where the visual condition contains trajectory-propagated appearance”
  • Multi-embodiment training: Training on interaction data involving different robot forms or human bodies. “We show that action flow can serve as a shared interface for generalist world modeling across embodiments”
  • Open-loop policy evaluation: Assessing a policy by executing or replaying its actions without feedback-based correction during evaluation. “Our open-loop policy evaluation uses achieved-trajectory replay”
  • Perspective projection: Mapping three-dimensional points into two-dimensional image coordinates according to a camera model. “$\mathbf{x}_{n,t} =\pi\!\left(\mathbf{K}\,\begin{bmatrix}\mathbf{I}_3 & \mathbf{0}\end{bmatrix}\mathbf{T}_{CW}\,\mathbf{T}_{\ell(n)}(\mathbf{q}_t)\,\bar{\mathbf{X}_n\right)$”
  • Pixel-aligned: Precisely associated with corresponding locations in an image’s pixel coordinate system. “These pixel-aligned trajectories specify where the robot should move in the observed video”
  • Privileged metadata: Information available during data processing or training but not assumed to be available in the deployment setting. “This produces the same trajectory representation F\mathcal{F} for training without requiring privileged metadata.”
  • Readout: A learned mapping from internal model features to task-specific outputs. “a target-embodiment readout decodes its latent motion features into executable actions.”
  • Rollout: A simulated or generated sequence of future states obtained by repeatedly applying a model or policy. “At deployment, Isaac Lab produces this sequence by executing the candidate command through the robot controller and physics simulation.”
  • Semantic grounding: Associating visual tracks or regions with meaningful entities such as robots or objects. “All samples across grounded and unassigned tracks so that training retains approximate motion correspondence when semantic grounding is incomplete.”
  • Spatial latent state: An internal representation that preserves the spatial arrangement of visual features. “For motion conditioning, we retain the spatial structure of the initial encoded state s0\mathbf{s}_0.”
  • Teleoperation: Control of a robot by a human operator, typically through a remote interface. “ABC-130k~\cite{allshire_scalable_2026} and MolmoAct2~\cite{fang_molmoact2_2026} provide bimanual teleoperation.”
  • Trajectory-conditioned video model: A video-generation model whose predictions are guided by specified motion paths. “ATI and Wan-Move achieve nontrivial zero-shot performance, indicating that generic trajectory-conditioned video models can transfer to unseen robot tasks.”
  • Variational autoencoder (VAE) latent: A compressed learned representation produced by a variational autoencoder. “with dense point tracks, a language caption, precomputed VAE latents”
  • Video latent: A compact internal representation of video frames used by a generative model. “The video backbone receives $C_{\mathrm{motion}$ as an additional visual condition while denoising the future latent sequence.”
  • Visibility label: A binary or categorical indicator specifying whether a tracked point is observable at a particular time. “then recover dense image-plane trajectories and visibility labels with a flow tracker”
  • World action model: A model that infers executable robot actions from desired environmental or object motion. “We construct a world action model for the inverse mode by conditioning the model on desired object flow”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 7 tweets with 366 likes about this paper.

Sign Up for Free