From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data

Abstract: Video is a scalable observation of physical dynamics: it captures how objects move, how contact unfolds, and how scenes evolve under interaction -- all without requiring robot action labels. Yet translating this temporal structure into reliable robotic control remains an open challenge, because video lacks action supervision and differs from robot experience in embodiment, viewpoint, and physical constraints. This survey reviews methods that exploit non-action-annotated temporal video to learn control interfaces for robotic manipulation. We introduce an \emph{interface-centric taxonomy} organized by where the video-to-control interface is constructed and what control properties it enables, identifying three families: direct video--action policies, which keep the interface implicit; latent-action methods, which route temporal structure through a compact learned intermediate; and explicit visual interfaces, which predict interpretable targets for downstream control. For each family, we analyze control-integration properties -- how the loop is closed, what can be verified before execution, and where failures enter. A cross-family synthesis reveals that the most pressing open challenges center on the \emph{robotics integration layer} -- the mechanisms that connect video-derived predictions to dependable robot behavior -- and we outline research directions toward closing this gap.

• The paper introduces a principled taxonomy that categorizes video-based manipulation strategies into direct video-action, latent-action, and explicit visual interfaces.
• It analyzes each method’s performance and challenges, emphasizing the trade-offs between opaque model capacities and transparent, modular control interfaces.
• The survey highlights empirical results showing robust video-pretrained model performance while identifying limitations in domain transfer and closed-loop physical integration.

Interface-Centric Taxonomy of Learning Manipulation Interfaces from Temporal Visual Data

Introduction

The survey "From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data" (2604.04974) provides a comprehensive analysis of methods that leverage non-action-annotated video for robotic manipulation. It introduces a principled, interface-centric taxonomy that organizes the field along the spectrum of how temporal visual structure is transformed and integrated into executable control. By departing from traditional categorizations focused solely on model or generative class, the work foregrounds the control interface as the locus of design variation, modularity, and practical challenges.

Interface Families and Design Space

The core contribution is an organizing framework based on the explicitness and position of the video-to-control interface. Three interface families are identified:

1. Direct Video–Action Policies: Temporal prediction shapes internal representations, actions are decoded directly, and the interface remains implicit.
2. Latent-Action Methods: Temporal structure is routed through a compact, learned action-like intermediate. This enables abstract planning or model-predictive control in latent space before grounding in the robot's action space.
3. Explicit Visual Interfaces: The model predicts interpretable visual targets (e.g., subgoal images, planned trajectories, 6D/SE(3) poses) that downstream controllers track.

These families are placed along two primary axes: (i) interface explicitness and (ii) distance from robot actions. The interface can be implicit (direct policies), compact yet abstract (latent-action), or highly explicit and interpretable (explicit interfaces).

Figure 1: Video-based manipulation interfaces are organized into direct video-action policies, latent-action intermediates, and explicit visual interfaces.

Figure 2: Schematic of the three interface families’ architectures, showing the increasing explicitness of the interface and the separation of prediction from control.

Direct Video–Action Policies

Direct policies predict robot actions from visual observations, shaped by temporal video objectives; the video-to-control linkage is fully implicit within a monolithic model. Modern instantiations commonly utilize highly-capacitated transformers—autoregressive (GR-1/GR-2), diffusion-based (PAD, UWM, UVA), or frozen diffusion encoders (VidMan, VPP)—that are pretrained on massive corpora of action-free video and fine-tuned on limited robot demonstration data.

Figure 4: Taxonomy of direct video-action modeling approaches, spanning joint video–action generators, video-prediction-pretrained policies, and action-free world model pretraining.

While this design enables utilization of large-scale video priors, it fundamentally entangles perception and control, yielding opaque policies with limited debuggability, weak interpretability, and hard-to-separate embodiment transfer. Execution strategies range from token-level stepwise to open-loop chunked/receding-horizon, trading off adaptivity, smoothness, and computational cost.

Empirical results on benchmarks such as CALVIN, MetaWorld, and multi-robot datasets demonstrate significant performance improvements and generalization with video-pretrained models, particularly on long-horizon, multi-task, and language-conditioned scenarios. However, the latent–to–action mapping is brittle under domain/embodiment shift, and there is no explicit pre-execution verification.

Latent-Action Interfaces

Latent-action approaches decompose the problem into two decoupled phases: (1) action-free video is used to learn a compact, transition-descriptive latent space (via information bottlenecks, vector quantization, or variational inference); (2) lightweight groundings (MLP decoders, codebooks, adapters) relate these latents to actual robot controls using small labeled datasets or online interaction.

Figure 6: Generic latent-action pipeline: Stage 1, discover latent actions from action-free video via inverse/forward dynamics; Stage 2, ground latents to robot actions via limited labeled data.

Figure 3: Latent actions learned without action labels consistently induce coherent behavior in visual scenes (from Genie).

Discrete latent interfaces (e.g., Genie, FICC, LAPO, UniVLA) are increasingly preferred for their grounding simplicity and compatibility with modern VLA frameworks. Continuous approaches (e.g., CLASP) support expressive planning but are susceptible to entangling exogenous and controllable factors. Integration with VLA models for instruction-following (LAPA, UniVLA) is achieved via pretraining or persistent latent-token heads. The primary risks are latent identifiability, compositionality, and brittleness in the grounding step—multiple real actions may map to similar visual transitions, and physical or embodiment constraints are only weakly enforced.

Explicit Visual Interfaces

This family exposes the temporal interface as an interpretable, manipulable signal: subgoal images, video plans, pixel trajectories, 3D/6D poses, or affordance waypoints. Video-based predictors (diffusion or transformer models) are trained on large-scale action-free human/robot video. Grounding of the visual interface to actions is performed via goal-conditioned controllers, inverse-dynamics models, geometric transfer, or alignment/tracking controllers.

Figure 5: Explicit visual interface methods decouple perception and control via a human-interpretable interface, e.g., subgoal images, video plans, and 2D/3D trajectories.

This explicitness permits human and robotic assessment (plan check, pose feasibility), transparent cross-embodiment transfer (by specifying goals or trajectories in image or world coordinates), and pipeline modularity—interface predictors and policies/controllers can be improved or swapped independently. However, accuracy of the geometric/semantic transfer from predicted interface to executable action is often the limiting factor; catastrophic errors in pose estimation, object segmentation, or physical plausibility are common, and multi-stage perception pipelines are fragile under distribution shift, occlusion, and real-world complexity.

Among other trends, methods favor closed-loop and short-horizon execution with regular interface replanning, reflecting the current limitations in long-horizon predictive accuracy and physical realism of video-generated plans.

Comparative Analysis and Theoretical/Practical Implications

Numerical results: Across standardized evaluation suites (CALVIN, LIBERO, MetaWorld), explicit visual interface approaches that employ closed-loop, error-driven replanning outperform open-loop dense-plan baselines; e.g., subgoal-image-based CLOVER significantly exceeds dense-plan UniPi on the CALVIN D split. Latent-action integration (UniVLA) achieves strong results on LIBERO with minimal robot data, but reveals sensitivity to latent disentanglement.

Contradictory claims: The study clarifies that direct video–action methods, while highly performant on in-distribution data, lack the structured transparency and modular transfer of explicit interfaces. In contrast, explicit interfaces are not universally more robust; performance degrades significantly when perception transfer (pose/track extraction) fails, highlighting that interpretability does not guarantee practicality without robust estimation.

Theoretical implications: The taxonomy surfaces a critical “robotics integration layer,” which remains the bottleneck for dependability—grounding video-derived predictions into physically feasible, verifiable motor actions under real-world constraints. Simple scaling of video pretraining does not obviate the need for this integration. Real-world challenges—contact, force, compliance, occlusion—are not intrinsically solved by larger or more abstract video-model-based representations.

Practical implications: The survey underscores that explicit interfaces enable modular, inspectable, and potentially cross-embodiment transfer (given robust estimation), but require further research in interface design (to balance abstraction and realizability), grounding strategy (to ensure physical and embodiment feasibility), execution monitoring, and safety verification.

Future Directions

Key open directions identified:

• Execution-aware/physically grounded learning: Integration of feasibility constraints, physical priors, and uncertainty estimation into video-based control interfaces.
• Robust cross-embodiment grounding: Efficient, scalable grounding mechanisms and adapters for diverse robots and domains with minimal data.
• Multimodal integration: Leveraging tactile, force, and proprioceptive sensing alongside vision to master contact-rich and deformable manipulation.
• Systematic evaluation and safe deployment: Standardizing benchmarks, budgeting robot data, and developing verification modules for interface-based pipelines.

Conclusion

This survey systematizes the rapidly evolving field of video-based robot manipulation by foregrounding the interface as the structural locus of functional and theoretical trade-offs. It shows that while large-scale video priors have unlocked new capabilities, robust integration with physical robots remains unsolved, and the primary challenges now reside in interface specification, grounding, and closed-loop deployment rather than in generative model capacity alone. Future progress will depend on closing the loop between advanced, scalable video learning and physically reliable, safe, and interpretable robotic behavior.