RHO: Your Coding Agent is Secretly a Roboticist

Abstract: Code-as-Policies (CaP) has shown that LLMs can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and $π{0.5}$ averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than $π{0.5}$. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE benchmark, RHO optimizes the deployed agent's multi-file harness of prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.

• The paper demonstrates that repository-level mutation, combining both prompt and toolbody edits, yields significant policy improvements on ROS 2-based manipulation tasks.
• It introduces a multi-file evolution method that reduces tool calls and computation while delivering more deterministic outcomes compared to prompt-only search.
• The study validates that closed-loop joint evolution of agent prompts and tool codes enhances performance across diverse robotic scenarios, ensuring robust real-world applicability.

RHO: Systematic Evolution of Robotics Codebases via Repository-Level Mutation

Overview

The paper "RHO: Your Coding Agent is Secretly a Roboticist" (2606.16458) systematically investigates RHO, an evolutionary programming framework for closed-loop optimization of robotic agent repositories interfacing with LLMs. The main contribution lies in demonstrating that repository-level mutation—encompassing both agent prompt and toolbody edits, under an enforced fixed tool API—enables significant policy improvement on realistic, ROS 2-based manipulation benchmarks, surpassing prompt-only approaches (GEPA-style single-file search) in both accuracy and computational efficiency. Importantly, RHO demonstrates robust cross-domain transfer, scaling from primitive simulation to scenarios employing production-grade robotic middleware and open-vocabulary perception, all with a minimal hardware footprint.

The RAI Manipulation Benchmark as a Stress Test

The manipulation_o3de suite in the RAI framework is targeted as a challenging stress test. Unlike toy LLM-agent environments, RAI is structured around the ROS 2 middleware and utilizes genuine perception (Grounding DINO + SAM2), planning (MoveIt), and control stacks, bridging to the Open 3D Engine for physical simulation. Tasks require grounding free-form natural language to concrete robotic actions through a trinary tool interface (get_object_positions, get_ros2_camera_image, move_to_point) exposed to the agent. The five task classes, ranging from straightforward coordinate placement to object grouping and tower construction, span several difficulty levels, with scene complexity and evaluation tolerances carefully tuned. The manipulated scenarios faithfully reflect the semantics and noise of real deployed systems.

Experimental Design and Mutation Protocols

RHO is instantiated in two variants:

• Multi-file evolution (MF, headline): Mutations are permitted in both the agent's prompt and the implementation details of the robotics tool functions, provided an Abstract Syntax Tree check enforces a fixed tool-name/sig contract. This mirrors industrial settings, where the agent API is immutable but implementation optimizations are permitted.
• Single-file evolution (SF, ablation): Only agent prompt mutations are allowed (GEPA-style), with tool bodies locked to upstream RAI implementations.

Evolution is performed over ten generations, using a stratified batch for each task class in both the easy and hard difficulty splits (i.e., scenes with low/high clutter, simple/compound task statements, and loose/tight pass criteria). Selection is by scalar mean reward on a training suite, with held-out evaluation reserved for test tasks. Three independent runs per configuration average out LLM sampling variance.

Results: Numeric Gains and Analysis

Easy Regime

The multi-file configuration yields a mean held-out test lift from $68.3\% \pm 9.9$ to $88.9\% \pm 2.8$ ($+20.6$ pp; $p<0.001$), whereas the single-file ablation lifts from $65.1\% \pm 7.3$ to $79.4\% \pm 2.8$ ($+14.3$ pp; $p\approx0.012$). Thus, prompt-only search captures only about two-thirds of MF's policy improvement. Notably, evolved policies are more deterministic (variance drops to 2.8 pp post-evolution, from 7–10 pp in seeds).

Strikingly, both MF and SF candidates reduce inference costs: the MF agent completes all easy tasks with $\mathbf{17.6\%}$ fewer tool calls and modestly lower wall time compared to the unmodified seed. The additional tool code does not incur overhead—rather, optimized control logic reduces redundant perception, corrects error modes, and encodes efficient subroutines.

Hard Regime

Performance improvements are preserved under increased task complexity. Multi-file test lift is $+20.8$ pp ($88.9\% \pm 2.8$0 to $88.9\% \pm 2.8$1), while single-file increases $88.9\% \pm 2.8$2 pp ($88.9\% \pm 2.8$3 to $88.9\% \pm 2.8$4). The MF–SF gap persists at $88.9\% \pm 2.8$5 pp, with the MF agent showing substantially lower sample variance.

A per-class breakdown reveals that MF and SF approaches are complementary, not merely differing in overall magnitude:

• SF (Prompt edits) succeeds in planning-bound tasks: For instance, it recovers BuildCubeTower ($88.9\% \pm 2.8$6 pp) and MoveObjectsToLeft ($88.9\% \pm 2.8$7 pp), both dominated by coordinate-frame and sequence errors addressable in the prompt.
• MF (Toolbody edits) succeeds in perception-bound tasks: It unlocks GroupObjects ($88.9\% \pm 2.8$8 pp; MF: $88.9\% \pm 2.8$9; SF: $+20.6$0) and delivers further gains in PlaceCubes; these require direct manipulation of downstream perception outputs (color masking, query fallbacks, detection caching) inaccessible to the prompt.

Compute-wise, MF once more reduces tool invocations ($+20.6$1) and wall-time ($+20.6$2). Conversely, the SF prompt-instructed agent becomes more thorough, at the cost of $+20.6$3 more tool calls and $+20.6$4 wall time, underscoring a trade-off where prompt search can buy accuracy only by spending more inference budget.

Mechanistic Interpretability: What RHO Discovers

An analysis of evolved repositories reveals several systematic corrections and innovations:

• Prompt rewrites correct coordinate frame bugs, enforce singular object naming, disallow double grabs, and specify explicit per-task recipe steps (e.g., stack heights in cube towers, adjacency heuristics).
• Toolbody mutations introduce:
  • Color-mask verification for open-vocabulary detections, eliminating false positives.
  • Robust query fallback strategies to counteract LLM or detector sensitivity to phrasing/plurality.
  • Detection caches with spatial snaps, closing the loop between perceived and acted-upon object identity.

Notably, the seed prompt includes a sign-inversion bug for the Y axis (stating "positive is right" when the benchmark defines "positive is left"). RHO systematically discovers and corrects this misalignment without direct supervision.

Theoretical and Practical Implications

RHO demonstrates that LLM agent policy improvement should proceed by repository-level search rather than prompt-only editing. When failures are rooted in the physical interface—especially perception ambiguities—toolbody mutation is critical and cannot be offset by more intricate prompting. For robotics teams, this aligns with industrial best practices where agents call utility libraries exposed via immutable tool APIs; RHO then becomes a closed-loop optimizer for the system's implementation details.

Methodologically, RHO is task-agnostic: zero changes were needed to transfer the closed-loop evolution protocol from CaP-Bench to RAI/O3DE, other than enforcement of an API contract. This robustness, together with the demonstrated variance suppression and strong compute efficiency, suggests RHO is directly applicable to closed-loop, real-world learning-on-hardware scenarios.

Furthermore, the strong attenuation in agent output variance is notable: optimized repositories yield more deterministic plans and behavior under stochastic LLM sampling, which is a necessary property for safety-critical and reproducible hardware deployments.

Future Directions

Two clear research avenues emerge from these results:

• Scalability to broader tool surfaces: As robotics systems integrate more complex and high-dimensional sensorimotor pipelines, investigating RHO's limits and scaling properties becomes essential.
• Hybridization with credit assignment and RL: While evolutionary search is effective for global repository improvement, compositional credit assignment strategies or local gradient-based RL may further accelerate convergence or unlock new behaviors.

More generally, RHO forces a re-examination of the LLM-agent split: it is the holistic agent-repo interface, including both tool and prompt, that constitutes the real policy, and optimal control may necessitate joint, closed-loop co-design.

Conclusion

RHO is established as a robust, general framework for iteratively refining collaborative LLM agent stacks via closed-loop, repository-level evolution. Its capacity to identify failure origins and target improvements to prompt or tool code as appropriate leads to substantial and statistically significant gains on industrially relevant manipulation tasks, all with reduced compute and variance. These findings confirm that for realistic agents integrating language and physical interfaces, code-level evolution outperforms prompt-only search, advancing the automation of robotics system optimization and establishing new best practices for agent policy improvement.