- The paper introduces a novel agentic paradigm that recasts robot manipulation as a visual tool-use task using a browser-based 3D interface.
- Methodology leverages a closed-loop observe-act pipeline with frontier foundation models, achieving success rates up to 96.7% on diverse tabletop tasks.
- The approach decouples control from robot-specific training, allowing scalable improvements without bespoke perception or task-specific abstractions.
VIA: Visual Interface Agent for Robot Control
Overview and Motivation
The VIA (Visual Interface Agent) framework introduces a new paradigm for robot manipulation, leveraging the capabilities of frontier foundation models (FMs) in an agentic context. Rather than training vision-language-action models (VLA) on limited robot-specific datasets or relying on handcrafted abstraction libraries as in Code-as-Policies (CaP), VIA recasts robot control as a visual tool-use task directly operated through a browser-based 3D interface. The agent performs closed-loop control by observing screenshots, reasoning about scene geometry, issuing intuitive commands through a custom suite of Model Context Protocol (MCP) tools, and continually adjusting based on outcomes, all without privileged state information or robot-specific fine-tuning.
This decoupling of robot control from robot-centric training exploits recent advances in general-purpose computer-use agents, which have demonstrated robust visual reasoning and UI interaction skills. VIA’s approach allows robot manipulation to inherit the scalable improvements of FMs, transforming robot control into a generic agentic task that is accessible to any highly capable FM agent.
VIA Architecture and Design
VIA operates in three principal phases: observation, action, and execution, iterated in an agentic “observe-act” loop. A browser-based interface presents a 3D point cloud scene constructed from multiple RGB-D cameras, complemented by third-person and wrist-view camera feeds. The agent observes the UI by capturing screenshots—never accessing raw scene states or simulator internals—and manipulates the robot via MCP tools that mirror human-level operations within the interface:
- Observation Tools: Screenshot acquisition, gripper pose visualization, rotation gizmo overlays, and pixel-based hover/crosshair functionalities enable the agent to inspect the environment, probe salient points, and reason about geometric relationships entirely from rendered views.
- Action Tools: Gripper teleportation, drag, translation, rotation, and toggle functions enable flexible and precise waypoint setting for end-effector positioning and state adjustment.
- Camera Tools: Orbit, pan, and zoom operations permit the agent to reframe the scene from arbitrary viewpoints, resolving occlusions or facilitating spatial reasoning.
- Execution Tools: Waypoint execution moves the real robot to the set target pose with a PI controller, closing the loop for continuous, feedback-driven manipulation.
The interface design is minimalist and ergonomic, exposing only generic kinematic controls and raw sensory views, thereby sidestepping the bottlenecks of predefined perception or skill abstractions and fully delegating spatial reasoning and planning to the FM agent.
Evaluation and Results
VIA was evaluated across six diverse tabletop manipulation tasks, including LIBERO-Goal (turning on a stove, opening a drawer, placing a bowl on a plate), custom long-horizon assembly (rainbow arrangement), and BuilderBench’s T-block stacking challenge. Agents (Claude Code and Codex) were deployed in multiple model configurations (e.g., CC-Opus, CC-Fable, Codex-5.5, Codex-5.6-Sol) and tested with both minimal (goal-only) and detailed (text-based demonstration) prompts.
Key empirical findings:
- VIA achieved zero-shot end-to-end solutions for all tasks with success rates up to 88% (CC-Fable) under minimal prompt conditions (2607.11119).
- On LIBERO-Goal tasks, VIA with CC-Fable yielded 96.7% average success; the rainbow assembly task was solved perfectly (100%) with the strongest model and minimal prompt.
- Task performance scaled markedly with model strength; replacing CC-Opus with CC-Fable universally improved success rates. Similarly, higher-tier Codex models showed gains in precision tasks.
- Weak models (CC-Opus, Codex-5.5) benefitted substantially from text-based demonstrations, with CC-Opus improving from 77% to 100% success on LIBERO-Goal when provided with example waypoint lists.
- Tool call efficiency and API cost analyses revealed that Codex configurations were more economically efficient (average $4.1–$7.2 per successful episode), while CC agents, though more costly ($9.5–$15.1 per episode), delivered higher success and more deliberate planning behaviors.
VIA’s closed-loop agentic policy allows robust error recovery and dynamic re-planning, handling manipulation, articulated object interaction, and long-horizon assembly without bespoke robot training or perception primitives.
Theoretical and Practical Implications
VIA establishes that state-of-the-art computer-use agents—when interfaced through generic human-legible controls—possess transferable skills for robot manipulation tasks, even in unfamiliar settings. This finding has several implications:
- Foundation Model Scaling: Future advances in FM perception, reasoning, and tool-use transfer directly to robot control without additional fine-tuning, obviating the need for robot-specific adaptation or expertise.
- Prompt-based Demonstration: VIA enables rapid generation of “robot demonstrations” from plain-text or screenshot lists, opening possibilities for prompt optimization and self-improving policies via reflective learning and text feedback.
- Decoupled Abstractions: Unlike CaP and VLA models, VIA relies only on generic controls and visual observations, freeing it from task-specific abstraction design and data requirements.
- Flexible Deployment: VIA’s architecture is readily extensible to real robots, assembly tasks where reliability, not speed, is paramount, reset mechanisms in reinforcement learning, and data collection for fast policy training.
The limitations of VIA at present include reliance on high-cost, high-latency frontier models and restriction to quasi-static manipulation tasks; dynamic activities remain challenging due to inference speed constraints.
Future Directions
Several extensions are clear:
- Deployment of VIA in real-world robotic environments, leveraging advances in model latency and cost reduction.
- Integration of more sophisticated controllers for collision avoidance and enhanced precision.
- Automatic tool improvement and agent-driven interface adaptation through feedback.
- Large-scale prompt optimization, demonstration generation, and reflective learning strategies for continual policy improvement.
- Combining VIA-generated demonstrations with policy distillation or imitation learning for training lightweight, real-time controllers.
Conclusion
VIA demonstrates that recasting robot control as agentic visual tool-use is sufficient for robust, zero-shot manipulation in complex tabletop environments with state-of-the-art FM agents. This paradigm enables direct scalability with FM improvements and provides a practical pathway to harnessing the advances of generalist agents in robotics (2607.11119). As FMs continue to progress, VIA offers a scalable, efficient, and flexible strategy for generalist robot control, promising broad utility across simulation and real-world deployment.
