Democratizing Robot-Use Agents: A Unified Harness with Open Models
This lightning talk explores how a unified interaction harness can unlock robot-use capability in open-weight foundation models without relying on proprietary systems or privileged simulator access. We examine the architectural choices that enable open models to exceed previously published closed-model baselines, the capabilities that determine robot-use performance, and the trade-offs between cost, throughput, and effectiveness in both simulation and real-world manipulation.Script
Most demonstrations of zero-shot robotic manipulation depend on proprietary frontier models, leaving researchers unable to inspect, fine-tune, or even reproduce the underlying policy. The authors of this paper ask whether a sufficiently structured interaction harness can expose substantial robot-use capability in open-weight models without privileged access to simulator state or distillation from closed systems.
The harness introduces resource-aware pipelining that overlaps model inference with physical execution while preserving ordering constraints. In one task, this reduced episode duration from 34.7 minutes to 12.5 minutes, with over 90 percent of physical execution time overlapping model requests, compared with zero overlap under serial execution.
In simulation, Qwen 3.8 Flash Next with the proposed harness achieves a RoboDojo score of 31.39, exceeding the published GPT 6 Astra with RoboProbe result of 28.97. When Astra is evaluated with the same harness, it reaches 51.90, demonstrating that both the model and the interaction design jointly determine performance.
Trajectory analysis reveals that effective robot use depends on the joint availability of four capabilities: visual understanding, spatial grounding in 3D, geometric coding accuracy, and embodied reasoning about physical interaction. A model may understand an object visually but fail to ground it in 3D, or compute a pose correctly but select an infeasible grasp.
Removing depth while retaining camera calibration changes Astra's score by only 1.4 percent and Qwen's by 12.5 percent, but removing camera parameters as well causes both scores to drop sharply. Depth is not necessary for strong robot use when calibration is available, but geometric information itself is essential.
This harness makes robot-use research reproducible and compatible with open models, enabling researchers to collect structured interaction data at a fraction of the cost of proprietary systems. To explore how modular interaction design can unlock embodied capabilities in foundation models, visit EmergentMind.com and create your own video from the latest research.
