Understanding Robot-use Agents (RUAs)

• Robot-use Agents (RUAs) are embodied, goal-directed systems that use multimodal observations and feedback to accomplish complex physical tasks, characterized by the PAPAV (Perceive–Anticipate–Plan–Act–Verify) loop.
• RUAs utilize various techniques, including vision-language models, conventional planners, and code-generation agents, to ensure task-level decisions are connected to physical execution and feedback,covering systems with varying degrees of autonomy, such as those selecting predefined tools and generating executable programs.
• Evaluation dimensions for RUAs include perception, anticipation planning, action, verification, autonomy, and safety mechanisms.

Robot-use Agents (RUAs) are embodied, goal-directed systems that use multimodal observations, robot interfaces, executable tools, and feedback to accomplish physical tasks. Unlike systems that merely generate robot instructions, an RUA must connect task interpretation, perception, anticipation, planning, action, execution monitoring, and outcome verification in a persistent interaction loop. The term encompasses architectures ranging from cloud-hosted robotic services and ROS-based tool interfaces to coding agents that synthesize robot programs, multi-agent systems, and vision-LLMs augmented with calibrated perception tools. Contemporary formulations characterize the core loop as Perceive–Anticipate–Plan–Act–Verify (PAPAV) (Chen et al., 21 Oct 2026).

1. Concept and scope

An RUA operates at the boundary between general-purpose multimodal reasoning and embodied robotics. It receives a task instruction, obtains information about the robot and environment, selects or constructs an action, executes it through a robot-specific interface, observes the result, and revises its subsequent behavior. The defining property is not the use of a particular model architecture. An RUA may employ an LLM, VLM, conventional planner, finite-state machine, hybrid controller, code-generation agent, or end-to-end VLA policy, provided that task-level decisions remain connected to physical execution and feedback (Chen et al., 21 Oct 2026).

The distinction between an RUA and an ordinary robot controller is primarily functional. Conventional robotic systems typically provide sensing, state estimation, motion planning, control, and execution monitoring for predefined objectives. An RUA additionally supports open-ended task interpretation, multimodal grounding, task decomposition, dynamic tool or skill selection, persistent memory, feedback-driven replanning, active information gathering, assistance or abstention, and task-level verification.

A standalone VLA or WAM can serve as an executor within an RUA but is not necessarily an RUA itself. A model that maps observations directly to actions without persistent task-level state, adaptive recovery, or outcome-dependent revision provides evidence for the action capability but not necessarily for the complete robot-agent loop. RobotAgentBench reports a substantial difference between manipulation and broader autonomy: the best standalone policy obtained a task-level average of 18.0 and an open-ended scenario score of 1.0, whereas the strongest tested agent configuration obtained 48.5 and 15.4, respectively (Mao et al., 26 Oct 2026).

The term also covers systems with different degrees of autonomy. Some RUAs select among predefined robot tools; others write and revise executable programs; others generate or deploy new ROS 2 nodes. The principal architectural question is where to place the boundary between agent-level decisions and backend execution.

2. Functional architecture

The PAPAV framework decomposes RUA operation into five recurring capabilities:

1. Perceive: construct an action-relevant belief from RGB, depth, video, language, proprioception, force, tactile data, maps, robot state, and memory.
2. Anticipate: predict the consequences, risks, and likely progress associated with candidate actions.
3. Plan: select and organize feasible actions, subgoals, constraints, recovery branches, or assistance requests.
4. Act: ground a task-level decision into a robot-compatible skill, waypoint, trajectory, action chunk, joint command, velocity command, or torque command.
5. Verify: determine whether the intended physical outcome occurred and whether to continue, retry, replan, stop, or request help (Chen et al., 21 Oct 2026).

These functions need not correspond to separate software modules. They may be distributed across a LLM, perception tools, motion planners, controllers, safety monitors, and external evaluators. A complete RUA requires that their outputs influence subsequent task-level decisions. An external simulator checker or low-level controller monitor does not by itself demonstrate agent-level verification.

The execution interface is typically hierarchical. A high-level agent may issue a semantic command such as “move the cup to the tray,” while a backend resolves object references, computes poses, checks reachability and collision constraints, generates a trajectory, and controls the robot. RobotUse makes this decision boundary explicit: the agent retains target selection, candidate choice, orientation, approach, and correction, while the backend performs calibrated perception, geometric conversion, inverse kinematics, path checking, control, and execution (Lee et al., 4 Oct 2026).

Other systems place more responsibility in the agent. WetRobo gives a coding agent access to images, depth, joint state, gripper state, effort, torque, a repository, and external software tools. The agent writes perception, calibration, motion, verification, and recovery programs, then revises them after physical execution (Oikawa et al., 16 Sep 2026). OpenRUA goes further by exposing only a terminal, ROS 2 documentation, native ROS 2 interfaces, and ordinary starter programs. Perception becomes file I/O and manipulation becomes coding, allowing the agent to discover interfaces and construct its own robot programs (Chu et al., 1 Oct 2026).

The central division is therefore not between “agent” and “robot” in a binary sense, but among:

• Deliberative agency: interpreting goals, selecting targets, decomposing tasks, choosing tools, and deciding when to revise.
• Embodied computation: perception, geometry, calibration, dynamics, and spatial estimation.
• Execution: trajectory generation, low-level control, actuator interfacing, and real-time stabilization.
• Verification and safety: checking physical outcomes, enforcing limits, detecting faults, and preventing unsafe commands.

3. Robot interfaces, tools, and embodiment

RUAs require interfaces that expose robot capabilities without forcing the agent to manage every low-level detail. These interfaces may be cloud services, ROS topics and actions, HTTP APIs, terminal commands, spatial specifications, or generated code.

SRCA represents an early cloud-robotics architecture. It packages ROS software into isolated Kubernetes services, automatically generates Dockerfiles, WSGI applications, OpenAPI specifications, client libraries, Kubernetes objects, and reverse-proxy routes. Robots or applications invoke remote recognition, planning, speech, or mathematical services through HTTP. The architecture supports C++, Python, and JavaScript clients, but focuses on non-time-critical computation; real-time motion control remains local (Remmas et al., 2018).

The Robot Context Protocol (RCP) provides a runtime-agnostic interface based on HTTP, WebSocket, and Server-Sent Events. Its semantic operations are read, write, execute, and subscribe, with path-based resources such as /sensor/pose, /action/move_to, and /param/speed_limit. RCP adds runtime discovery, schema-defined payloads, asynchronous feedback, namespaces, type validation, and adapter-based integration with ROS 2, simulators, cloud services, dashboards, and agent systems (Lee et al., 13 Jun 2025).

ROS 2 is a common backend because it supplies topics, services, actions, parameters, controllers, and middleware integration. RAI organizes embodied agents around Agents, Connectors, and Tools. Connectors expose publish-subscribe, service-based, and action-based communication; tools serialize requests, invoke ROS 2 operations, parse perception results, and return observations to an agent. The framework supports conversational agents, state-based agents, physical and virtual agents, RAG-based embodiment information, and multi-agent systems (Rachwał et al., 12 May 2025).

Tool design strongly affects RUA behavior. Tools may be:

• Low-level: joint commands, Cartesian velocities, gripper commands, or sensor reads.
• Geometric: depth measurement, point-cloud fitting, pose transformation, calibration, or spatial relations.
• Skill-level: pick-and-place, pouring, stacking, navigation, manipulation, or recovery.
• Verification-oriented: grasp checks, object-motion tests, force thresholds, task-state estimation, or endpoint comparison.
• Meta-level: code synthesis, simulation, memory retrieval, playbook updates, or service discovery.

Robo-Harness K1 exposes calibrated depth, persistent visual anchors, spatial measurements, and grasp hypotheses as VLM-callable perception tools while preserving generic motion primitives. It does not provide hidden object poses or a task-specific planner. The VLM chooses regions, measurements, grasp candidates, motions, and recovery actions (Li et al., 24 Sep 2026).

Embodiment information may be represented through robot documentation, URDFs, images, PDFs, vector databases, system prompts, calibration files, or live state. RAI’s experiments show that visual embodiment information can improve hazard judgments in a tractor simulation, while language-only descriptions led to excessive caution around small obstacles (Rachwał et al., 12 May 2025). Conversely, embodiment information is not sufficient for reliable control: the same work reports inconsistent mission-state synchronization, weak completion detection, and spatial-reasoning failures during manipulation.

4. Planning, execution, and learning mechanisms

RUA planning ranges from language-mediated decomposition to explicit geometric planning, code synthesis, hierarchical multi-agent orchestration, and online exploration.

LA-RCS uses a dual-agent structure. A Host Agent interprets the user request and generates a Global Plan; an App Agent selects successive control functions, executes them through a mobile robot, receives camera and sensor feedback, and decides whether to continue or finish. This creates a closed loop between natural language, robot commands, observations, memory, and replanning. On a 20-request benchmark, the system reported 90% average success in the abstract, while the detailed tables reported 95% for GPT-4o and 60% for GPT-4-Turbo; the paper contains an internal inconsistency in these aggregate figures (Park et al., 23 May 2025).

Multi-agent collaboration can improve internal reliability without guaranteeing better physical behavior. In a physical Spot experiment, a coder–reviewer–manager architecture reduced initial code-execution errors relative to a single-agent baseline and performed better on vague or abstract prompts. Adding a planner produced more code errors and no overall task-performance advantage. The study therefore supports role specialization and generation–review separation rather than the assumption that increasing agent count is inherently beneficial (Rosser et al., 2024).

UROSA distributes perception, scene reasoning, motion planning, diagnostics, capability assessment, digital-twin curation, and code synthesis across Agentic ROS 2 Nodes. Its Code Synthesis Agent can generate ROS 2 nodes, unit tests, and runtime software components, while safety parsers validate syntax and operational limits before publication. The system reported 70–80% success for several generated-node classes and used retrieval-augmented generation, prompt-level teacher–student adaptation, and trajectory negotiation among underwater vehicles (Buchholz et al., 31 Jul 2025).

Coding-agent RUAs treat executable programs as policy representations. WetRobo uses a repository, demonstrations, an AGENTS.md skill file, laboratory equipment, sensors, and robot-control code. The agent may use SAM 3, OpenCV, NumPy, MuJoCo, optical flow, RANSAC, RGB-D registration, regression, and external packages to construct local programs. It successfully lifted a Petri dish lid, removed a bottle cap in two laboratories, and opened an incubator door in one laboratory (Oikawa et al., 16 Sep 2026).

URAI separates programming from runtime decision-making. A programming agent writes and validates reusable robot tools, while a frozen execution agent selects and parameterizes one complete tool at a time, observes the result, and decides what to call next. Across five RoboDojo tasks, URAI increased aggregate success from 18.0% under direct fingertip control to 53.0%; a pre-written program using the same tools achieved 24%, compared with 56% when the execution agent retained control between tool calls (Ge et al., 30 Sep 2026).

RAX scales online exploration rather than demonstrations or value-function training. A coding agent writes programs, executes exploratory trials, observes outcomes, revises code, and compiles successful decision sequences into persistent skills. The architecture was evaluated across single-arm, bimanual, dexterous, mobile, and humanoid systems. Reported results include 78.3% on RoboDojo at large budgets, 46.5% average Q-score on BEHAVIOR-1K, 89.8% on HumanoidArena, 96.7% on DexJoCo at 32 episodes, and 98.5% on standard LIBERO (Huang et al., 15 Oct 2026).

OpenRUA challenges the necessity of specialized harnesses by giving a coding agent terminal access to native ROS 2 interfaces, documentation, and ordinary starter programs. Claude Code with Claude Opus 5 achieved 99.0% on CaP-Bench, 87.0% on LIBERO-PRO, and 39.4% overall on RoboCasa365. Across 1,500 episodes, the agent wrote metric-perception programs in 96.80% of episodes, motion-control clients in 95.87%, and feedback-control programs in 50.13% (Chu et al., 1 Oct 2026).

5. Safety, verification, and failure recovery

Physical execution makes safety and verification central RUA problems. A command can be syntactically valid, geometrically plausible, and successfully executed at the controller level while failing the user’s task or creating a hazard.

Safety mechanisms reported across RUA systems include workspace limits, joint limits, self-collision checks, collision-aware planning, force and torque thresholds, gripper monitoring, keep-out zones, speed and motion caps, dry runs, pre-execution refusal, emergency stops, human approval, safety parsers, action cancellation, and deterministic low-level controllers.

These mechanisms are distributed across layers. SRCA keeps safety-critical control local because cloud latency is nondeterministic (Remmas et al., 2018). RAI places Nav2, MoveIt 2, localization, collision checking, motion planning, and low-level control below the agent (Rachwał et al., 12 May 2025). UROSA uses safety parsers and deterministic ROS 2 controllers but does not provide formal verification or sandboxing for generated nodes (Buchholz et al., 31 Jul 2025). WetRobo combines workspace clamps, keep-out zones, motion caps, pauses before contact, horizontal-jaw constraints, torque monitoring, proof pulls, simulation checks, and endpoint verification, while acknowledging the absence of a general collision-reaction mechanism (Oikawa et al., 16 Sep 2026).

Verification must be distinguished from execution feedback. Execution feedback may report that a trajectory completed, a gripper closed, or an action was accepted. Task verification asks whether the intended physical state was achieved. K1 explicitly rejects the assumption that a closed gripper implies successful attachment and requires visual and depth feedback after grasping (Li et al., 24 Sep 2026). RobotUse similarly treats target selection, pose editing, execution, and outcome interpretation as connected but distinct stages (Lee et al., 4 Oct 2026).

RobotWorld identifies recurrent verification failures: agents reach commanded poses but lose object state, continue ineffective actions, recover too late, or mistake unfinished tasks for completion. Its benchmark therefore separates agent intent, requested actions, execution evidence, and evaluator evidence. Across 84 tasks, success rates were 19.0% for Astra, 15.5% for Opus 5.5, 2.4% for Kimi K3, and 1.2% for both DeepSeek V4.1 Flash and Gemini 3.8 Flash (Yang et al., 7 Oct 2026).

RobotAgentBench similarly evaluates task discovery, contextual reasoning, manipulation, failure recovery, and robustness. Its best agent configuration achieved a task-level average of 48.5 and an open-ended scenario score of 15.4, compared with 18.0 and 1.0 for the best standalone policy. The benchmark emphasizes that manipulation competence does not imply open-world autonomy (Mao et al., 26 Oct 2026).

Common RUA failure modes include:

• Perceptual misgrounding: incorrect object identity, position, orientation, or frame.
• State loss: failure to track whether an object is grasped, supported, moved, released, or unstable.
• Action ineffectiveness: repeated commands that do not address the active subgoal.
• Late recovery: strategy changes after the remaining budget or physical margin is insufficient.
• False completion: termination based on a favorable image, finished command, or verbal assumption rather than the required postcondition.
• Tool misuse: invalid schemas, unsupported capabilities, stale observations, or incorrect parameterization.
• Context overload: excessive multi-agent history, stale visual memory, or loss of relevant state.
• Backend mismatch: commands that are semantically reasonable but infeasible for the embodiment.
• Contact failure: inadequate force, grasp, insertion, release, or dynamic stabilization.

6. Evaluation, generalization, and open problems

RUA evaluation increasingly combines end-to-end task success with process diagnostics. RobotWorld contains 84 tasks from 20 source projects across manipulation, mobile manipulation, locomotion, driving, and aerial control. It distinguishes robot requests, tool calls, confirmed control steps, wall-clock time, tokens, and simulated time, and uses executable success checkers rather than textual judgments (Yang et al., 7 Oct 2026).

The PAPAV survey finds that current benchmarks emphasize Act. Across 64 benchmarks, Act is explicitly assessed by 10 of 10 MMEA benchmarks, 16 of 23 MMA benchmarks, and 29 of 31 robotics benchmarks. Anticipate is explicitly evaluated by only 5 of 64 benchmarks, and Verify by only 3 of 64. Planning is often required but rarely isolated as an explicit capability (Chen et al., 21 Oct 2026).

This imbalance creates several methodological problems. A successful episode does not reveal whether anticipation was used, whether planning was coherent, whether recovery was available, or whether verification was performed by the agent rather than an external checker. Conversely, failure does not identify whether the cause was perception, planning, grounding, control, timing, or verification.

Useful evaluation dimensions include:

• Perception: state-estimation accuracy, identity persistence, spatial grounding, uncertainty calibration, memory freshness, contradiction correction, and active-perception cost.
• Anticipation: action-conditioned prediction, collision and contact prediction, calibrated uncertainty, counterfactual ranking, decision regret, and out-of-distribution detection.
• Planning: task decomposition, ordering, precondition satisfaction, feasibility, recoverability, replanning, assistance, abstention, and retained alternatives.
• Action: semantic grounding, frame correctness, control validity, interruption latency, deadline misses, execution progress, embodiment transfer, and safety violations.
• Verification: false-success and false-failure rates, partial-progress recognition, evidence sufficiency, verdict latency, causal failure localization, and recovery after negative verdict.
• Whole-system autonomy: long-horizon performance, occlusion, irreversible actions, dynamic environments, hardware variation, human assistance, sim-to-real transfer, and recovery after intermediate failure.

Generalization can be achieved through several mechanisms. Perception-driven programs can recalibrate and re-localize across layouts; persistent tools can retain validated skills; playbooks can transfer textual operating guidance; RAG can retrieve embodiment and mission knowledge; multi-agent systems can separate HRI, perception, planning, and control; and native interfaces can permit adaptation without retraining a policy.

However, transfer is conditional. It depends on calibrated sensors, reliable action semantics, appropriate robot documentation, low-level safety mechanisms, adequate computational budgets, and sufficiently observable environments. RAX, WetRobo, OpenRUA, and HexaShell all show that executable code can adapt to new scenes or embodiments, but their evaluations also expose software brittleness, high token costs, limited real-world trials, and the need for independent safety enforcement (Huang et al., 15 Oct 2026, Oikawa et al., 16 Sep 2026, Chu et al., 1 Oct 2026, Gao et al., 16 Oct 2026).

The principal open problems are:

1. Persistent physical belief: representing identity, relations, uncertainty, freshness, and contradictions across long horizons.
2. Action-conditioned anticipation: predicting contact, stability, risk, reachability, and progress rather than merely generating plausible futures.
3. Adaptive commitment: choosing when to execute long action chunks and when to interrupt for new perception or replanning.
4. Uncertainty-conditioned control: acting, gathering information, abstaining, or requesting help according to risk and information value.
5. Verification-guided improvement: attributing failures correctly before modifying prompts, tools, memory, plans, or policies.
6. Safe software evolution: validating generated tools and ROS 2 nodes through sandboxing, regression testing, rollback, permissions, and independent monitors.
7. Real-time integration: coordinating slow deliberation with asynchronous physical execution, action cancellation, progress telemetry, and deadline constraints.
8. Cross-embodiment semantics: preserving task meaning across different frames, grippers, sensors, controllers, and actuator interfaces.
9. Open-world robustness: handling novel objects, dynamic humans, unmodeled disturbances, degraded connectivity, and incomplete or contradictory observations.

RUAs are therefore best understood as a family of embodied agent systems rather than a single algorithmic paradigm. Their common principle is the persistent coupling of multimodal reasoning, physical action, and feedback-driven revision. Current research demonstrates strong local capabilities in perception, geometric computation, tool use, code synthesis, online exploration, and execution. The principal unresolved issue is composition: maintaining task-relevant state, anticipating physical consequences, selecting feasible actions, recovering before options disappear, and verifying genuine completion under uncertainty.