Recursive Agent Optimization

Abstract: We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents to scale to longer contexts and generalize to more difficult problems via divide-and-conquer. RAO provides a method to train models to best take advantage of such recursive inference, teaching agents when and how to delegate and communicate. We find that recursive agents trained in this way enjoy better training efficiency, can scale to tasks that go beyond the model's context window, generalize to tasks much harder than the ones the agent was trained on, and can enjoy reduced wall-clock time compared to single-agent systems.

• The paper introduces a novel RL framework that trains a unified policy for recursive delegation in language model agents.
• It employs local node-centric rewards and a leave-one-out baseline for efficient credit assignment in deep recursive trees.
• Experimental results demonstrate significant gains in success rate and speed across complex tasks like TextCraft-Synth, Oolong-Real, and DeepDive.

Recursive Agent Optimization: Reinforcement Learning for Hierarchical LLM Agents

Overview and Motivation

"Recursive Agent Optimization" (RAO) (2605.06639) introduces a reinforcement learning (RL) paradigm for training LLM agents capable of recursive delegation—i.e., agents that dynamically spawn sub-agents, each tasked (potentially recursively) with subproblems of a given challenge. Unlike existing approaches where recursion and multi-agent orchestration are inference-time scaffolds layered on top of a pretrained base model, RAO trains a single, shared policy across all nodes in dynamically constructed recursive execution trees. Through this end-to-end optimization, RAO enables the agent to not only solve assigned tasks but also learn when and how to delegate, formulate subproblems, communicate across recursive levels, and synthesize results.

Recursive Agent Formulation and Policy Optimization

Recursive agent inference reframes rollouts as a dynamically generated tree:

• Each node represents an agent instance tackling an assigned subtask.
• At any node, the policy governs decisions to spawn further sub-agents, thus expanding the execution tree.
• The action space includes a dedicated asynchronous delegation primitive, enabling both sequential and parallel sub-agent launches using standard Python concurrency.

  Figure 1: Recursive execution tree induced by an agent recursively delegating tasks, where each agent instance operates under a unified policy trained across all levels.

RAO defines a local node-centric reward that integrates the agent's own task success and the successful completions of any subproblems it delegates:

$$R(X, \tau_X) = \tilde{s}(X, \tau_X) + \lambda \cdot \frac{1}{|C(X)|} \sum_{c \in C(X)} \tilde{s}(c, \tau_c)$$

$\tilde{s}(X, \tau_X)$ denotes node-local success; $C(X)$ is the set of immediate children; $\lambda$ controls the weight of the delegation bonus. This local credit assignment approach mitigates the credit diffusion problem in deep trees and supports both dense (node-level) and sparse (root-level only) reward regimes.

Policy optimization proceeds jointly across all nodes in all rollout trees, using a weighted objective that normalizes by the frequency of trajectories at each recursion depth. Advantage estimation for each node uses a leave-one-out baseline over root rewards, an unbiased variance-reduction technique suitable for heterogeneous and asynchronous tree structures. Importantly, all agent instances, regardless of tree position, are parameterized by the same policy and jointly optimized.

Experimental Evaluation

The efficacy of RAO is assessed on three domains demanding long-horizon reasoning and decomposition:

• TextCraft-Synth: A synthetically generated crafting environment with parametrizable recursive subtask structure.
• Oolong-Real: Long-context information aggregation tasks requiring agents to operate beyond their context window via chunking and delegation.
• DeepDive: Multi-hop web research QA tasks necessitating sequential recursive decomposition and synthesis.

TextCraft-Synth highlights the core advantages of recursion for compositional, deep tasks. With an 8K context limit (below task length), recursive agents trained with RAO achieve a striking 95% overall success rate and 88% on hard tasks, compared to 24% and 0% for single-agent baselines. Even with a 40K/256K context regime where the single-agent’s context is never limiting, recursive agents maintain superior performance and are up to 2.5× faster on the hardest instances due to parallel sub-agent execution.

Figure 2: Training curves on Textcraft-Synth showing accelerated learning and higher final performance from recursive agents under RAO training.

Oolong-Real evaluations confirm that RAO-trained recursive agents generalize to much longer contexts (up to 220K tokens), vastly outperforming single-agent baselines restricted by the training context limit. Recursive agents approach or match the performance of much larger models by leveraging divide-and-conquer chunking.

Figure 3: Training curves for Oolong-Real, indicating recursive agents make superior use of limited context during training.

On DeepDive, the recursive agent demonstrates a 16 point improvement in test success rate (0.40 vs 0.24) over single-agent learning and leverages increased delegation depth to uniquely solve more complex, sequential tasks. Notably, in settings where subproblems are not parallelizable (DeepDive), wall-clock time increases due to the sequential nature of the decompositions.

Ablation and Analysis

RAO's design choices are validated through ablations on reward structure and trajectory weighting:

• Dense (node-level) rewards and depth-level inverse-frequency weighting are each critical for stable, efficient training and improved pass rates.
• In shallow, trivial tasks (e.g., ART-E email search), the benefits of recursion recede, indicating that RAO's overhead is best amortized in settings with substantial compositional or memory demands.

  Figure 4: ART-E training progress, showing convergent performance of single-agent and recursive agent in short-horizon domains.

Implications and Theoretical Contributions

RAO advances the field of agentic LLMs in several key aspects:

• Dynamic Hierarchical Decomposition: It demonstrates generalization to arbitrary recursion depths and problem hardness, supporting dynamically induced curricula and implicit long-horizon skill acquisition through recursive self-play and delegation.
• Efficient Credit Assignment: Local reward structures circumvent the issues of sparse global signals in deep reasoning tasks.
• Resource Adaptation: RAO-trained policies adapt the degree and depth of delegation to the structural depth and parallelizability of the task, optimizing both success rate and (where possible) wall-clock compute efficiency.
• Joint Parameter Sharing: Unlike conventional hierarchical RL or swarm-agent designs with fixed role assignments, RAO enables a single policy to robustly mediate complex, context-dependent role allocation across a recursive call graph.

Limitations and Future Directions

While RAO handles homogeneous recursive trees within individual domains, real-world agentic settings demand:

• Heterogeneous Sub-agent Typing: Orchestrating policies of varying capacity, specialization, or RL regimes at different nodes (e.g., experts supervising generalists or vice versa).
• Generalization Across Domains: Training recursive agents to transfer decomposition strategies beyond a fixed environment remains an open research problem.
• Efficient Surrogate Training: The high cost of recursive rollouts in live RL poses scaling and latency challenges. Advances in surrogate training—potentially leveraging distilled synthetic curriculum or actor simulation—are required for further scaling.
• Robust Delegation Verification: Tasks with ambiguous or poorly specified rewards for partial progress or sub-task correctness remain challenging, especially in settings where LLM-based supervision may induce biases.

Conclusion

RAO represents a substantial advance toward scalable, adaptive, and compositionally robust LLM agents capable of recursive delegation and efficient division of labor. By integrating recursive inference as a primitive into the training regime, it unlocks capabilities in long-horizon problem solving, resource allocation, and generalization that elude single-agent or inference-only scaffolded models. RAO's empirical results show substantial gains in training efficiency, context window extension, and successful generalization to harder or unseen tasks. The framework points toward a future in which agent policies are holistically aligned with complex inference-time scaffolds, enabling further innovations in multi-agent, distributed, and open-ended agentic architectures.