Playful Agentic Robot Learning

Abstract: Current agentic robot systems can write executable Code-as-Policy programs, observe feedback, and revise behavior across multiple attempts, but they remain largely task-driven: reusable skills are acquired only after explicit instructions. We study Playful Agentic Robot Learning, where an embodied coding agent uses self-directed play as a continual skill-learning stage before downstream tasks arrive. We introduce RATs, Robotics Agent Teams designed for play-time skill acquisition. During play, RATs proposes novel yet learnable exploratory tasks, plans and executes robot-code policies, verifies intermediate progress, diagnoses failures, retries with dense, step-level feedback, and distills successful executions into a persistent code skill library. At test time, the agent reuses relevant skills from this frozen library to help solve new tasks. Experiments in LIBERO-PRO and MolmoSpaces show that play-learned skills improve held-out downstream tasks over no-play and random-play baselines, with 20.6 and 17.0 percentage-point gains over CaP-Agent0 on LIBERO-PRO and MolmoSpaces, respectively. Moreover, the learned skills can be plugged into other inference-time Code-as-Policy agents by simply retrieving them into the context, improving RoboSuite and real-world transfer by 8.9 and 8.8 points, respectively, without finetuning the underlying model.

• The paper introduces a novel framework that proactively discovers and curates a library of code-based robot skills through self-directed play.
• The multi-agent RATs architecture employs a Write-Execute-Verify-Diagnose loop to localize failures and iteratively refine skill modules.
• Empirical results demonstrate significant gains in in-domain and sim-to-real tasks, validating the framework’s superiority over task-driven baselines.

Playful Agentic Robot Learning: Autonomous Code-as-Policy Skill Acquisition

Motivation and Problem Formulation

Large multimodal and LLMs have enabled agentic robot systems capable of formulating and executing plans as Code-as-Policy (CaP) programs, observing feedback, and adjusting their behavior. However, these systems are predominantly task-driven: skill acquisition and code library growth happen reactively, contingent on explicit instructions or reward signals. This stands in contrast to biological intelligence, where pre-task play facilitates the autonomous, continual acquisition of a repertoire of flexible skills.

The "Playful Agentic Robot Learning" framework (2606.19419) offers a systematic method for proactive skill discovery in agentic coding agents. The research introduces RATs (Robotics Agent Teams), a multi-agent architecture operationalizing play as a continual learning phase: the robot proposes and practices its own goals, verifies and diagnoses outcomes, and curates a reusable code skill library. Critically, this library can be plugged into subsequent downstream tasks, demonstrating both in-domain generalization and cross-environment transfer.

RATs Architecture and Methodology

RATs decomposes the agentic robot learning pipeline into three collaborative subteams, emphasizing modularity and robust feedback:

• Task Proposer: The agent autonomously generates a pool of candidate tasks using LLM-driven intrinsic motivation, leveraging a novelty-learnability scoring mechanism inspired by the Goldilocks principle. The proposer receives a compact skill summary and context-aware historic performance data to select the most informative tasks for play.
• Execution Team: Consisting of planning, policy-writing, verification, and diagnosis agents, this team orchestrates a Write-Execute-Verify-Diagnose loop. Plans are assessed for step-wise physical soundness, with multi-level verification to provide dense, intermediate feedback and bottleneck isolation.
• Memory Management: Successful executions are distilled into callable, parameterized Python skills with explicit preconditions and effects. A reliability-based tiering system tracks skill robustness, promoting or deprecating skills accordingly. Failures are catalogued, distilled into correction lessons, and maintained for high-utility retrieval during future planning or code synthesis.

This multi-layer architecture enables not only rapid code failure localization and iterative correction, but also active skill curation: periodic memory and skill merging, duplicate culling, and failure-driven skill proposal maintain a lean but effective library.

Play-Driven Skill Acquisition: Learning Pipeline

During play-time, RATs operates in an environment devoid of external objectives. The system's internal loop is as follows:

1. Candidate task proposals are generated based on current environment state, library usage profiles, and dense records of prior failures and successes.
2. The selected task is instantiated as a concrete simulation (e.g., BDDL/MuJoCo for LIBERO, or scene groundings for MolmoSpaces).
3. The Execution team synthesizes and verifies policy code; failures trigger dense diagnostic analysis, memory updates, and, if recurrent, isolated SubAgent training on problematic sub-behaviors.
4. On success, modular skill routines are extracted directly from code execution traces, parameterized, statically validated, and curated into the persistent library.

By the end of pre-deployment play, the agent has autonomously accumulated a set of robust, compositional primitives that cover not only well-explored combinations but also frontier tasks that expand capability. The learnability criterion ensures practice is focused on tasks balancing informativeness and feasibility.

Evaluation and Empirical Results

The empirical investigation addresses several core hypotheses:

• Do self-directed, curiosity-driven play skills improve downstream manipulation performance relative to both passive ("No Play") and undirected random play?
• Are skills acquired via play transferable across environments and physical robot embodiments?

In-Domain Performance

On the LIBERO-PRO suite (object, goal, and spatial generalization), 50 RATs play iterations led to a +20.6 percentage point gain in success (23.2% $\rightarrow$ 43.8%) over a standard CaP-Agent0 baseline. In MolmoSpaces, the gain was +17 points (21.0% $\rightarrow$ 38.0%), with the library disproportionately boosting success for manipulation with articulated objects. These improvements are not explainable by increased test-time inference compute alone: matched-compute ablations show only modest gains for baselines with extended retry budgets versus the substantial boost from RATs-trained skill libraries.

Cross-Domain and Sim-to-Real Transfer

The frozen skill library, trained entirely in simulation (e.g., LIBERO-PRO), provided plug-and-play performance gains when injected into different environments and robot suites. For RoboSuite, transfer elevated performance by +8.9 points (40.3% $\rightarrow$ 49.1%), with specific tasks such as two-arm collaborative manipulation benefiting most. Sim-to-real experiments demonstrated that the same plug-in approach yielded +8.8 points in physical robot manipulation success for tasks not previously practiced in the real world.

Ablation Studies

Ablations reveal strong complementarity between skill learning during play and the structured execution framework at test time; both are necessary for optimal results. Random play (undirected by curiosity or novelty-learnability) has marginal benefit over no play (23.2% $\rightarrow$ 24.7%), while curiosity-driven play confers substantially larger gains, confirming the necessity of mechanism-driven curriculum discovery.

Token and Compute Analysis

RATs incurs substantial token usage during play, especially for policy writing and failure diagnosis, but these costs amortize over downstream rollouts. The up-front compute expenditure at play-time produces more performant, robust transfer than simply allocating extra inference-time retries during test-time rollouts.

Theoretical and Practical Implications

This research operationalizes classical concepts of intrinsic motivation, curriculum discovery, and skill abstraction within the agentic Code-as-Policy paradigm. Rather than being bottlenecked by task-driven skill formation, the robot autonomously explores, practices, and internalizes a diverse set of behaviors prior to any explicit instruction.

Practical implications:

• The plug-and-play nature of the skill library makes it directly composable with standard CaP-based and LMM-driven robot controllers, without retraining, fine-tuning, or environmental replay.
• Robust intermediate supervision, structural skill tiering, and failure-driven revision foster rapid adaptation and improved reliability on complex, long-horizon manipulation tasks.
• Cross-environment and sim-to-real transferability suggest generalizability and robustness of autonomously extracted skill abstractions.

Theoretical implications:

• Autonomous curriculum generation via novelty-learnability scoring maps well to developmental psychology concepts and overcomes limitations of fixed-goal or fixed-feature-space exploration.
• Multi-role agent team structure embodies an architectural pattern for integrating learning, verification, and continual skill library growth in agentic robotics.

Limitations and Future Directions

RATs remains constrained by the domain and diversity of the simulation environments. The skill library's effectiveness is limited by the richness of physical interaction primitives and scene variation available during play. Reliance on vision-LLMs and token-heavy runtime agents elevates operational cost and can confound fine-grained physical manipulation lacking in the primitive API specification. Further, skill misuse or inappropriate retrieval at test time can degrade performance, emphasizing the need for robust context-aware selection mechanisms.

Future extensions should pursue:

• Scaling to richer, more dexterous low-level control interfaces, potentially integrating continuous control primitives or model-predictive controllers for fine actuation.
• Broader, more diverse pre-play simulation with richer physical affordance, multi-modal feedback, and open-world object/task variety.
• Deeper theoretical analysis of how intrinsic reward shaping and adaptive novelty/learnability metrics affect long-horizon skill composition and generalization.
• More thorough physical robot deployment for evaluating robustness under real-world perception, actuation, and environment variation.

Conclusion

"Playful Agentic Robot Learning" introduces a rigorously validated, multi-agent system for skill acquisition via self-directed play, demonstrating strong absolute and transfer gains in robotic manipulation benchmarks (2606.19419). The results provide a powerful empirical foundation for intrinsic-motivation-driven continual learning in agentic robotics, and highlight the value of proactive, curriculum-guided code library construction for persistent, cross-task, and cross-environment large-model robot control.