AReaL2.0: Microservice Agentic RL

• AReaL2.0 is a microservice-oriented online agentic RL system that continuously improves agents through live production traffic without altering core logic.
• It implements a standardized Agent Trajectory Data Protocol and enterprise-grade data proxy to ensure robust, auditable, and governance-compliant deployments.
• Its modular architecture and unified evolution control plane with PPO-based updates yield measurable improvements in operator efficiency and task accuracy.

AReaL2.0 is a prototype online agentic reinforcement learning (RL) system that structurally reorganizes existing asynchronous RL infrastructure into a microservice-oriented loop. This enables deployed LLM agents to improve continuously from production traffic with near-zero alteration to their core business logic or harness. The system implements three essential pillars—standardized step-level agent trajectory data protocol, enterprise-grade data proxying, and a unified evolution control plane—facilitating robust, auditable, and minimally supervised self-evolution of agentic deployments. The approach is validated on production workloads (specifically the Hermes agent), where it delivers substantial improvements in accuracy and operator efficiency via continuous PPO-based policy updates on live interaction trajectories (Yan et al., 1 Jul 2026).

1. System Architecture and Components

AReaL2.0’s architecture is modular and service-oriented, designed to slot in front of existing agent endpoints (such as SGLang or vLLM) and capture the full learning and improvement loop. The four primary components are:

• Gateway: Replaces or fronts the agent’s standard LLM inference API, intercepting all calls for detailed logging and potential routing.
• Router: Ensures session affinity, binding all calls from a multi-turn user interaction to a single data proxy instance, aggregating atomic events into unified trajectories.
• Data Proxy: Materializes the Agent Trajectory Data Protocol (ATDP), handles data redaction, replay annotations, reward harvesting, tenant isolation, and enforces governance before emitting data into queued training buffers.
• Agent-Compute Worker: Combines real-time rollout (“actor”) and policy training (“learner”). It consumes ATDP trajectories, performs (asynchronous) RL policy updates, and exposes the latest $\theta$ parameters for continued inference.

This architecture allows the introduction of RL-driven policy improvement through a drop-in replacement of endpoints, enabling continuous online adaptation without retraining or redeployment cycles (Yan et al., 1 Jul 2026).

2. Agent Trajectory Data Protocol (ATDP)

ATDP formalizes how agent-environment interactions are recorded, enabling step-level RL across heterogeneous agent frameworks. Each trajectory $\tau$ is a sequence of typed events:

$\tau = (e_1, e_2, \ldots, e_T)$

where each step

$e_t = \langle o_t, h_t, a_t, y_t, r_t, m_t \rangle$

with:

• $o_t$: observable input (e.g., user message, tool output)
• $h_t$: hidden agent-internal state or scratchpad
• $a_t$: agent action (LLM tokens, tool invocation, memory op)
• $y_t$: observed outcome (tool return, user correction, etc.)
• $r_t$: reward (scalar, binary, natural-language critique)
• $m_t$: metadata (latency, cost, tenant, model/version, governance flags)

ATDP is cross-framework (LangChain, CrewAI, ClaudeSDK, etc.) and supports post-hoc, late-bound rewards, versioned replay, and embedding of governance signals (e.g., eligibility, redaction). Only information strictly necessary for learning and credit assignment is recorded, facilitating scalable credit assignment, replayability, and governed observability (Yan et al., 1 Jul 2026).

3. Enterprise Data Proxy and Governance

The Data Proxy mediates all live agent interactions, transforming events into ATDP records and ensuring that organizational/redaction policies are enforced before data enters any training buffer. Features include:

• Framework-agnostic interception: At LLM, tool, memory, and human-in-the-loop APIs.
• Lossless emission: Maintains all the relevant context (prompt templates, log-probs, tool schemas).
• Replay support: Stores appropriate artifacts to enable (exact or approximate) replay and to signal non-replayable trajectories.
• Reward harvesting: Extracts weak or delayed supervision from subsequent user or system events.
• Tenant and governance isolation: Enforces strong boundaries for multi-tenant environments and supports federated or split training.
• Data integrity gate: Applies retention, redaction, and access control prior to buffering for learning.

These functions ensure the system remains compliant, reproducible, and safe for enterprise workloads, with direct links from raw production signals to actionable learning substrates (Yan et al., 1 Jul 2026).

4. Unified Agent Evolution Control Plane

The evolution control plane abstracts the agent as a composite at epoch $\tau$0:

$\tau$1

where:

• $\tau$2: policy LLM with weights $\tau$3
• $\tau$4: in-context harness (prompt templates, etc.)
• $\tau$5: memory/retrieval policy
• $\tau$6: tool repertoire and schemas
• $\tau$7: governance and guardrail configuration

Given a recent window of trajectories $\tau$8, the controller selects evolution actions $\tau$9 to maximize an improvement score $\tau = (e_1, e_2, \ldots, e_T)$0:

$\tau = (e_1, e_2, \ldots, e_T)$1

Possible actions include policy updates (RLHF, PPO, DPO), harness edits, memory changes, tool/schema changes, rollbacks, or no-ops.

Decision logic utilizes metrics such as fail_rate, user_edit_rate, average return, KL divergence shift, and tool-specific error clustering to determine whether to e.g., update policy weights, patch memory, or simply monitor.

All control actions are subject to significant safety and governance gates: replay evaluation on held-out sets, regression and safety benchmarking, access controls, canary rollouts, and versioned promotion/rollback (Yan et al., 1 Jul 2026).

5. Mathematical RL Foundations

AReaL2.0 accommodates conventional on-policy RL as well as specialized reward schemes. Abstractly, for ATDP trajectories under policy $\tau = (e_1, e_2, \ldots, e_T)$2:

• Objective:

$\tau = (e_1, e_2, \ldots, e_T)$3

• Policy gradient update:

$\tau = (e_1, e_2, \ldots, e_T)$4

where $\tau = (e_1, e_2, \ldots, e_T)$5 estimates advantage.

• PPO variant: For

$\tau = (e_1, e_2, \ldots, e_T)$6

$\tau = (e_1, e_2, \ldots, e_T)$7

• Parameter update step:

$\tau = (e_1, e_2, \ldots, e_T)$8

The decoupled rollout-and-train architecture supports high-throughput, asynchronous policy evolution suitable for live production traffic (Yan et al., 1 Jul 2026).

6. End-to-End Data Flow and Control Logic

The operation proceeds through the following dataflow:

1. Agent App sends an LLM request to the Gateway.
2. Gateway tags and forwards the request to the Router.
3. Router ensures all session turns go to the same Data Proxy.
4. Data Proxy:
   • Emits a partial ATDP event into its buffer.
   • Forwards request to an Agent-Compute Worker (actor).
   • Completes the ATDP record with outcome and reward, then writes it into the training queue.
5. Learner consumes trajectory minibatches, performs RL updates, checkpoints the policy.
6. Actor periodically refreshes parameters, so updated models are instantly deployed for inference.

Control-plane decision process:

• Computes trajectory statistics: fail_rate, user_edit_rate, avg_return, KL_shift, tool_failures.
• Applies rules to select actions (policy update, harness edit, etc.).
• For policy updates, runs shadow RL, offline evaluation, and canary rollout before global deployment.

$\tau = (e_1, e_2, \ldots, e_T)$9 (Yan et al., 1 Jul 2026)

7. Hermes Use Case and Experimental Results

In the Hermes deployment, substituting a standard vLLM/SGLang endpoint with the AReaL2.0 Gateway enables the collection of live ATDP trajectories and streaming PPO-based updates at approximately 500–1000 episodes/hour. This process led to a 30% reduction in human-operator corrections and a 15% improvement in end-task completion within two days, as measured by canary A/B testing. These empirical results demonstrate that minimally invasive integration of AReaL2.0 into live agent systems yields measurable gains in agent quality and operator efficiency, with direct linkage between live workload and learning signal (Yan et al., 1 Jul 2026).

---

AReaL2.0 thus validates the practicality of an asynchronous, high-throughput agentic RL system architected via microservices, operationalizing step-level learning, governed observability, and unified evolution control for enterprise-scale self-evolving agent platforms (Yan et al., 1 Jul 2026).