Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning

Abstract: Agentic reinforcement learning (RL) has become an important post-training paradigm for turning LLMs from static chatbots into interactive agents, giving rise to representative applications such as OpenClaw. Existing work mainly focuses on policy optimization algorithms and training frameworks, but pays less attention to the full data lifecycle of agent-environment interactions, from data production to training consumption. To bridge this gap, we present Claw-R1, an interactive step-level data middleware system for agentic RL. Claw-R1 connects heterogeneous agent runtimes with RL training backends through two core components: a Gateway Server and a Data Pool. The Gateway Server captures multi-turn interaction steps through a unified LLM API entry point, while the Data Pool organizes them into step-level records consisting of prompt IDs, response IDs, rewards and other metadata. In our demo, users can interactively inspect live trajectories, examine the state, action, and reward of each step, curate data by quality and readiness, and configure training-ready batches for different downstream RL algorithms. Overall, Claw-R1 treats agent interaction traces as managed data assets rather than temporary runtime logs. Through this demonstration, we hope to encourage the community to recognize the importance of data management in agentic RL. Our code is available at https://github.com/AgentR1/Claw-R1 and the demonstration video can be found at link https://youtu.be/Pw47dAOw6B0.

• The paper introduces Claw-R1, a middleware that captures interaction data at step-level granularity and decouples data production from RL training.
• It employs a Gateway Server and Data Pool to normalize and optimize interaction traces, reducing redundancy through prefix-tree merging.
• The system enables real-time data curation and transparent backend consumption, paving the way for reproducible and scalable agentic RL experiments.

Claw-R1: A Step-Level Data Middleware for Agentic RL

Introduction

The increasing complexity and heterogeneity of agentic reinforcement learning (RL) environments present significant data management challenges, particularly as LLMs transition from static chatbots to interactive, multi-turn agents. While recent research has advanced RL algorithms, frameworks, and data synthesis pipelines for agentic RL, the production, curation, and lifecycle management of agent-environment interaction traces have been notably overlooked. "Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning" (2606.09138) directly addresses this gap by proposing an architecture that abstracts, organizes, and interfaces interaction data between diverse agent runtimes and RL training backends.

Data Lifecycle Management in Agentic RL

Scaling agentic RL involves the orchestration of data pipelines across environments that leverage tool use, human feedback, code execution, and long-context processing. Approaches like PPO and GRPO have driven preference alignment and reasoning in LLM RL, and frameworks such as veRL and slime have tackled infrastructure bottlenecks. However, RL pipelines remain tightly coupled to agent-specific runtime implementations, impeding reusable, scalable, and efficient consumption of interaction traces in downstream optimization. In contrast, Claw-R1 shifts the perspective from transient logging to persistent, managed data assets, emphasizing the centrality of standardized, step-level data representations.

System Architecture and Step-Level Abstraction

Claw-R1 interposes a Gateway Server and Data Pool between agent runtimes and RL training backends. The Gateway Server provides an OpenAI-compatible LLM API for black-box agents and explicit submission interfaces for white-box agents, capturing all interaction events—including prompts, responses, actions, rewards, and metadata—at a step granularity. This decouples data production from consumption, allowing arbitrary agent systems to interact with the Claw-R1 middleware without intrusive modifications.

Within the Data Pool, interaction steps are normalized into a unified schema that preserves the Markov decision process (MDP) semantics over state, action, reward, and trajectory. Notably, Claw-R1 institutes prefix-tree merging at the token sequence level, preventing redundant long-context recomputation across trajectories that originate from the same prompts but diverge at later decision points. This optimization directly reduces memory and compute for RL training backends while maintaining data fidelity necessary for credit assignment and policy updates.

Interactive Workflow and Data Curation

A core innovation of Claw-R1 lies in providing an interactive dashboard to monitor and manage the data lifecycle:

• Live Trajectory Ingestion: Users observe data flow from diverse sources, including tool-driven rollouts, human feedback, and automated environments. The system facilitates direct, observable ingestion rather than opaque log accumulation.
• Step-Level Inspection: The Data Pool exposes all collected steps, enabling trajectory-level inspection, state-action-reward visualization, and lineage tracking via prompt and response IDs.
• Quality-Driven Curation: The dashboard enables filtration and selection based on reward availability, policy freshness, trajectory completeness, and metadata attributes, supporting the construction of high-quality, algorithm-ready training batches.
• Prefix-Tree Optimization Visualization: Users can inspect shared and divergent context regions, attention mask construction, and the impact on token savings, reinforcing the importance of structural data organization for efficient computation.
• Transparent Backend Consumption: RL trainers fetch curated, reward-filtered, and context-optimized batches through standardized APIs, decoupled from agent-specific runtime idiosyncrasies.

By making curation and trace management explicit, Claw-R1 turns data quality and pipeline readiness into observable and actionable system components—a marked departure from traditional, backend-coupled RL training paradigms.

Implications and Future Directions

Claw-R1’s data-centric approach presents immediate practical advantages: system interoperability, streamlined scaling of agentic RL experiments, and improved reproducibility due to persistent, queryable trace retention. The framework lays the groundwork for generalizable RL data platforms capable of addressing future challenges, such as:

• Cross-agent Policy Transfer: The decoupling of data and runtime supports meta-learning and distillation across heterogeneous agent pools.
• Advanced Curation Strategies: Explicit metadata and reward tracking enable dynamic, active curation, online human-in-the-loop labeling, and continual rollouts with selective sample reweighting.
• Benchmarking and Reproducibility: The persistent and queryable representation of agentic trajectories enhances the fidelity of experimental replication and multi-system benchmarking.

The theoretical abstraction of agent-environment data as first-class managed assets invites further development of middleware-driven RL pipelines, automated data quality assessment, and integration with advanced algorithmic credit assignment techniques.

Conclusion

Claw-R1 introduces a step-level middleware paradigm facilitating the comprehensive management of agentic RL data. By decoupling agent runtime complexity from RL backend requirements through a unified Gateway Server and Data Pool, Claw-R1 advances both the systematization and scalability of agentic RL pipelines. The middleware’s focus on standardized, curated, and optimized data assets enables transparent monitoring, curation, and efficient downstream consumption, paving the way for scalable, robust, and reproducible research and deployment of interactive LLM-based agents.