Agent as Policy for Robotic Manipulation

Abstract: We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

• The paper introduces 'Agent as Policy for Robotic Manipulation,' utilizing a general-purpose multimodal coding agent for robotic control, improving task flexibility and runtime adaptability with zero-shot capabilities.
• The method achieves a high success rate across various tasks, such as an 8/10 success rate in assembly and 29/30 in block construction, while maintaining at least 80% success in seven out of eight reported task configurations.
• Runtime programming allows the agent to learn and adapt from physical feedback, reducing repeated execution costs significantly, with a 30% time reduction observed in the model-dependent experiments with experience reuse.

Problem formulation and central claim

“Agent as Policy for Robotic Manipulation” proposes a control architecture in which a general-purpose multimodal coding agent serves as the robot policy itself, rather than selecting among separately trained skills or producing a fixed program before execution (2609.12541). The central claim is deliberately stronger than conventional language-conditioned planning: the agent controls not only task decomposition but also runtime perception, geometric estimation, motion generation, observation scheduling, error recovery, and program construction.

The paper positions AGP against two common agentic robotics paradigms. In program-synthesis systems, an agent generates code that is subsequently executed, but runtime contingencies must be encoded in advance through explicit logic. In policy-orchestration systems, an agent selects among learned skills while the invoked policies retain control over low-level motion generation. AGP instead keeps the agent in the action loop and permits it to write or revise local programs whenever new physical evidence changes the appropriate response. This distinction is closely related to earlier code-generation approaches such as “Code as Policies” (Liang et al., 2022), but AGP emphasizes continual runtime interaction rather than principally precomputed program synthesis.

The policy is formalized at the tool-decision level. Given the task specification, robot interface, interaction history, and persistent workspace, the fixed-parameter agent samples a tool call. A call may request an observation, inspect a saved image or video, execute locally generated Python, or submit a motion and gripper target. The model weights remain unchanged throughout a trial; adaptation is implemented through new observations, generated programs, modified geometric estimates, and accumulated files.

Figure 1: AGP separates task preparation from runtime execution while allowing the execution agent to construct programs, request observations, issue robot actions, and revise its behavior from physical feedback.

This formulation makes the workspace part of the effective policy state. It contains scripts, measurements, observations, execution reports, and notes, allowing the agent to reuse validated procedures within a session or across repeated executions. The paper’s strongest architectural claim is therefore that a general-purpose agent can operate as a policy without a task-specific action model, provided that the robot interface exposes sufficiently expressive sensing, geometric grounding, and motion primitives.

Robot interface and runtime programming

The robot bridge provides overhead RGB observations, wrist-mounted RGB-D observations, proprioceptive state, calibrated camera information, end-effector poses, gripper state, and target-achievement errors. The agent may command joint targets, Cartesian end-effector poses, gripper opening, and additional observations. Cartesian actions are converted to joint targets using inverse kinematics and then executed under workspace, velocity, acceleration, and step-size limits.

Figure 2: The physical platform uses one or two I2RT YAM arms, a parallel gripper, an overhead RGB camera, and a wrist-mounted RGB-D camera in a shared calibrated workspace.

The interface is intentionally intermediate between high-level symbolic planning and direct torque control. AGP does not synthesize arbitrary feedback controllers at the motor level. Instead, it computes geometric targets and timed motion programs that are passed to an existing robot controller. This division is important for interpreting the results: the agent determines what motion to request and when, while trajectory tracking, inverse kinematics, and low-level constraint enforcement remain delegated to the robot runtime.

The runtime programs are not limited to simple coordinate arithmetic. Across 65 executions, the authors identify 37 retained Python working files, of which 29 implement perception or geometry, seven compose robot calls, and one provides visual annotation. The documented computations include depth measurement, color segmentation, ray-plane intersection, triangulation, surface fitting, grasp-offset estimation, camera-image projection, and concurrent bimanual execution. In one assembly trace, the agent estimates a planar surface from wrist depth, infers the orientation and position of a held pin, generates a corrected insertion target, examines the returned target error, and modifies the subsequent motion.

This mechanism supplies a concrete interpretation of “reasoning with physical feedback.” The agent does not merely reflect verbally after failure; it changes the computational procedure used to estimate state or action targets. However, the paper does not isolate the contribution of runtime programming from that of the underlying model, calibration, controller, or task-specific interface design. The program traces demonstrate capability and mechanism, but not causal necessity.

Task suite and experimental design

The evaluation covers five manipulation categories selected to stress different capabilities:

Task	Instruction	Primary challenge	Successful trials
Four-pair assembly	Human video	Long-horizon sequencing and precision insertion	8/10
Block construction	Goal image	Support relations and placement order	29/30
Dice flipping	Language	Grasp-dependent reorientation	10/10
Targeted throwing	Language	Timed dynamic release	2/2
Sequential towel folding	Human video	Bimanual deformable manipulation	5/5
Simultaneous towel folding	Human video	Coordinated bimanual transport	3/5

The standard configuration uses GPT-6 Astra at high thinking effort with an I2RT YAM arm, overhead RGB, wrist RGB-D, and fixed task budgets. Trials begin without prior task-execution experience or simulation rehearsal. The task preparation agent creates a reusable task definition, while a separate execution agent handles each runtime instance. This separation prevents the main runtime agent from being directly initialized with a hand-designed execution trace, although the task definition itself contains goals, constraints, reference materials, and success criteria.

The task suite is intentionally heterogeneous. Assembly requires preserving previously completed pairs while inferring mating relationships from a demonstration video. Block construction requires recovering object identities, orientations, and support constraints from static goal images. Dice flipping requires selecting grasps and rotations that yield a specified upward face after release. Throwing introduces a timed, joint-driven swing and gripper release. Towel folding tests coordinated manipulation of a deformable object under substantial state uncertainty.

The figures illustrate that the evaluation is not restricted to nominal pick-and-place. In assembly, the robot must execute eight-part, four-pair construction while maintaining earlier progress.

Figure 3: Four-pair assembly requires video interpretation, precise insertion, retry behavior, and preservation of completed subassemblies.

For block construction, the agent must infer an order that maintains structural stability rather than simply matching the final image through unconstrained placements.

Figure 4: Pyramid, two-tower, and six-block constructions require the agent to infer support relationships from static goal images.

The protocol evaluates physical outcomes after gripper release and arm withdrawal. This criterion exposes failures that would be missed by measuring only controller arrival error. For example, the six-block tower can be correctly assembled and subsequently collapse during withdrawal; the trial is therefore unsuccessful.

Main manipulation results

AGP achieves at least 80% success in seven of eight reported task configurations, with 100% observed success in five. The strongest results are 29/30 for block construction and 10/10 for dice flipping. The assembly result of 8/10 is notable because the task includes submillimeter-scale nominal radial clearances for some mating parts and requires preserving multiple completed pairings. The result supports the claim that a general-purpose agent can perform nontrivial long-horizon manipulation without task-specific policy training.

The block-construction results show sensitivity to task complexity. Pyramid construction and two-tower construction both succeed in 10/10 trials, whereas the six-block tower succeeds in 9/10. Mean successful-trial completion times are 21.6, 20.6, and 28.2 minutes, respectively. The six-block tower also consumes more tokens and inference cost than the simpler structures. This pattern is consistent with the additional visual verification and stability management required as the structure grows, although the experiments do not provide a controlled decomposition of which source of complexity dominates.

Dice flipping achieves 10/10 despite requiring object-specific orientation reasoning and repeated grasp selection.

Figure 5: Dice flipping requires choosing grasp and rotation strategies that produce the requested upward face after release.

The result is important because the agent is not merely transporting objects to specified locations; it must reason about how a grasped die’s orientation changes under manipulation. The paper reports examples in which the agent combines a tilt and low release with subsequent verification and regrasping. Nevertheless, the evaluation uses six dice under predefined conditions, so it does not establish robustness to substantially different die geometries, friction properties, or clutter distributions.

Targeted throwing succeeds in both trials. The agent performs motion probes, modifies the swing and gripper-opening schedule, and achieves separation into the target bowl.

Figure 6: Targeted throwing combines a circular joint trajectory with timed release and post-release verification.

The reported mean elapsed time is 22.8 minutes, but the throw itself completes after 13.9 minutes; approximately 40% of the reported duration is attributed to trajectory analysis and verification. This distinction illustrates a central limitation of the approach: successful physical action may be substantially faster than the full agent-mediated reasoning and reporting loop.

The towel experiments are more demanding in a different way. Sequential folding succeeds in 5/5 trials, but takes 50.8 minutes on average and costs USD 24.14 per successful trial. Simultaneous folding is faster, at 21.4 minutes and USD 7.46 on successful trials, but succeeds in only 3/5 trials. The apparent efficiency advantage of simultaneous execution therefore comes with lower reliability and cannot be interpreted as a uniformly superior strategy. Qualitative sequences show recovery from slipped corners, staged placement of flaps, and residual skew or buckling.

Figure 7: Sequential folding achieves the required arrangement after recovery from slips, but retains visible buckling and edge offsets.

Figure 8: Simultaneous folding completes the required inward folds in successful trials but exhibits residual skew and alignment error.

These outcomes support the paper’s claim that deformable-object manipulation remains a significant bottleneck. The agent can recover from local errors and satisfy the relatively permissive success criterion, but the long execution time and 3/5 simultaneous-folding success rate indicate that general-purpose reasoning does not remove the need for robust deformable-object state estimation and contact control.

Efficiency, model dependence, and experience reuse

AGP’s success results are accompanied by substantial inference and execution costs. Successful assembly trials average 37.2 minutes, 13.02 million tokens, and USD 16.62. Dice flipping averages 37.9 minutes, 17.08 million tokens, and USD 21.07. Sequential towel folding is the most expensive main task in the reported set, averaging 50.8 minutes, 18.12 million tokens, and USD 24.14. The time-allocation analysis attributes large fractions of execution to the visual loop and remaining model-response latency, particularly for assembly and towel folding.

The model comparison demonstrates that performance depends strongly on the underlying agent-model combination. On a two-pair assembly task, GPT-6 Astra succeeds in 5/5 trials at low, medium, and high thinking effort, with mean completion times of 9.9, 9.2, and 9.2 minutes. Increasing effort therefore provides little observed benefit for this task. GPT-5.6 Sol also succeeds in 5/5 but takes 14.2 minutes on average. GPT-5.6 Terra succeeds in only 1/5 trials, while GPT-5.6 Luna succeeds in 0/5. Within Claude Code, Claude Opus 5 succeeds in 5/5, whereas Claude Fable 5.1 succeeds in 3/5.

These results contradict any interpretation of AGP as model-agnostic. The interface and runtime architecture are transferable across agents, but task reliability remains highly sensitive to model capability, agent scaffolding, and tool-use behavior. The results also show that inference cost and physical completion time need not move together: Sol has lower reported inference cost than Astra despite longer execution time, while effort increases do not materially improve Astra’s success on this task.

The experience studies provide the paper’s clearest evidence for improving efficiency without changing model parameters. In repeated two-pair assembly, saved measurements, procedures, corrections, and scripts reduce task execution time by 29.3% from the first to the fifth execution. The reasoning-and-programming component falls by 47.0%, while tool execution increases by 9.3%. This pattern indicates that reuse primarily reduces repeated deliberation and program construction rather than physical motion time.

The result should be interpreted as evidence of experience-conditioned efficiency, not as online learning in the model-parameter sense. Each trial begins with a fresh agent context, and improvements are mediated by files that encode prior work. Moreover, the repeated trials are dependent and share a fixed task configuration. The paper itself notes that comparisons against repeated trials with empty experience would be needed to isolate the causal effect of reuse from ordering and familiarity effects.

The strong-to-weak transfer experiment is more consequential. GPT-6 Astra first constructs frozen experience files; GPT-5.6 Terra then performs the same assembly task with or without access to those files. Terra’s success increases from 1/5 without experience to 4/5 with Astra’s experience. Among successful trials, experience reduces mean completion time by 34.3% and token usage by 11.4%, while also lowering inference cost. This result suggests that a stronger agent can amortize exploration for a weaker, less expensive agent.

The evidence remains limited by the small sample size and unequal successful-trial counts: the resource comparison uses one successful no-experience trial and four successful experience trials. Consequently, the magnitude of the reduction should not be generalized beyond the evaluated task without larger matched experiments. The study nevertheless establishes a specific design principle: persistent procedural artifacts can transfer operational competence across model instances and even across model capability levels.

Cross-embodiment evidence

The appendix reports a qualitative cross-embodiment experiment using a seven-joint P7 arm and a dexterous RealHand L6 hand rather than the YAM platform.

Figure 9: The cross-embodiment platform combines a P7 arm, RealHand L6 dexterous hand, overhead RGB-D sensing, and wrist-mounted RGB-D sensing.

The agent reorients the wrist, approaches a bottle, closes the fingers in stages, and commands a 2 cm lift.

Figure 10: AGP coordinates wrist reorientation, staged finger closure, and bottle lifting on a different arm and end-effector embodiment.

This experiment supports the narrower claim that the AGP interaction pattern can be instantiated through a different robot interface. It does not provide a quantitative cross-embodiment success rate, systematic comparison, or evidence that the same task definition and programs transfer without adaptation. The bottle subsequently slips during holding, further limiting the result to qualitative evidence of interface-level portability rather than reliable dexterous manipulation.

Limitations and open questions

The principal limitation is scale. Most main-task configurations contain only five or ten trials, and targeted throwing contains two. The reported percentages therefore have wide uncertainty, particularly for the 3/5 simultaneous-folding result and the 2/2 throwing result. The experiments establish feasibility on selected tabletop tasks, not population-level reliability across environments.

The evaluation also relies on substantial infrastructure. Camera calibration, predefined workspace envelopes, fixed motion limits, an existing inverse-kinematics and trajectory controller, and task-specific interface support are essential components. Targeted throwing uses a specialized timed-motion runtime, so it is not generated solely through ordinary Cartesian and gripper commands. The claim that AGP itself is a complete robot policy should therefore be understood as a claim about policy-level decision generation over a constrained robot interface, not replacement of the full control stack.

The agent’s latency and cost are practical constraints. Many successful trials require tens of minutes and millions of tokens. Visual observation, model response, code generation, and verification dominate several tasks. The paper demonstrates that persistent experience reduces this burden, but it does not establish how experience should be represented, validated, versioned, or prevented from propagating erroneous procedures.

Safety is addressed through interface-level motion and workspace limits, but the experiments do not evaluate human-robot interaction, adversarial instructions, hardware faults, calibration drift, or recovery from unsafe agent-generated programs. The agent can issue motion commands and construct executable code; independent protective mechanisms remain necessary for deployment beyond the controlled tabletop setting.

Several scientific questions remain open within the paper’s scope. It is unclear how AGP compares with trained VLA policies or hybrid systems under equal hardware, time, and compute budgets. It is also unclear whether runtime-written programs remain robust under substantial scene variation, whether accumulated experience transfers across robot embodiments rather than merely across model instances, and how much reliability is contributed by the coding-agent scaffolding relative to the multimodal model itself. Finally, the permissive success criteria for towel folding accept residual skew and buckling, leaving open how AGP would perform under tighter geometric or cosmetic requirements.

Conclusion

AGP demonstrates that a general-purpose multimodal coding agent can control real robot manipulation through a calibrated interface, runtime-generated programs, and iterative physical feedback. Across assembly, construction, dice flipping, throwing, and towel folding, it obtains strong zero-shot results, including 29/30 block-construction trials and 10/10 dice-flipping trials. The same experiments expose substantial execution latency, inference cost, model dependence, and difficulty with coordinated deformable manipulation.

The paper’s most substantive contribution is the treatment of the agent as a runtime policy rather than merely a planner, skill selector, or one-shot code generator. Persistent programs and measurements further provide a mechanism for reducing repeated execution cost and transferring operational experience between agents. The evidence supports AGP as a viable architecture for research-scale physical manipulation, while leaving reliability, efficiency, safety, and cross-embodiment transfer as unresolved empirical questions (2609.12541).