---
title: 'Polar: Decoupled Agentic RL Rollout Architecture'
url: https://www.emergentmind.com/topics/polar-decoupled-agentic-rl-rollout-architecture
type: topic
---

# Polar: Decoupled Agentic RL Rollout Architecture

Polar is a decoupled, agentic reinforcement learning (RL) rollout architecture designed for scalable, asynchronous training over arbitrary large language model (LLM)-driven agent harnesses. It enables RL for language agents that depend on complex, multi-turn harnesses by abstracting the harness as a black box: LLM API calls are proxied, token-level interactions are faithfully captured, and all trainable events are reconstructed as RL trajectories. Its architecture exposes parallel, asynchronous rollout endpoints consumable by independent trainers at scale, rendering Polar agnostic to agent environments, RL algorithms, and compute substrate. This decoupling allows for increased hardware utilization, full preservation of credit assignment, and plug-and-play integration across research harnesses, accelerating RL progress for long-horizon, multi-agent, and tool-use workloads [2605.24220].

## 1. Architectural Decomposition and Core Components

Polar operates in a multi-tiered service structure with explicit separation of environment execution, proxying, trajectory reconstruction, and reward evaluation:

- **Rollout Server** acts as the central scheduler, receiving `TaskRequests`—specifying harness, sample count, runtime specs, trajectory builder, evaluator, and callback URL—splitting them into sessions, persisting state, and allocating work to gateway nodes.
- **Gateway Nodes** manage per-session runtimes (e.g., via Docker or Apptainer), launch unmodified agent harnesses with LLM endpoint redirection, and include an LLM-API proxy to intercept every model call. Execution is pipelined across worker pools (INIT: runtime prewarm; READY: buffer management; RUNNING: harness execution/proxy capture; POSTRUN: trajectory reconstruction/evaluation/teardown).
- **LLM-API Proxy** is co-located with each agent runtime, intercepting completions in diverse provider formats, normalizing to a standard schema with log-probabilities, invoking vLLM or Triton backends, and recording prompt/response tokens and log-probs. Returned data is reshaped to match the expected format, maintaining harness transparency.
- **Trajectory Reconstruction Module** constructs RL-trainable traces from the sequence of intercepted completions, supporting strategies that enable token-faithful loss masking and merging (e.g., strict prefix alignment).
- **Asynchronous APIs** allow submission (`POST /rollout/task/submit`), status, heartbeat, and results polling, decoupling the rollout service from any particular trainer implementation.

This separation aligns conceptually with other state-of-the-art rollout-as-a-service systems, enabling independent scaling and fault domains across rollout, orchestration, and training [2603.18815].

## 2. Decoupled Design Principles and Token-Faithful Rollout

Polar’s foundational principle is to treat the agent harness—including multi-turn, multi-agent, or tool-augmented code—as an opaque executable, requiring no modifications. LLM API calls are intercepted directly at the boundary between the agent and inference backend, thus:

- All sampled (behavior-policy) tokens and per-token log-probabilities are captured exactly as emitted by the rollout policy.
- Generated completions, associated context, and non-trainable scaffolding (tool calls, canonical prompt content) are normalized and (optionally) masked out in training loss via a binary `loss_mask`.
- Credit assignment is preserved at the token level, with strict avoidance of “retokenization drift” (i.e., the divergence between tokens generated in the agent and those re-encoded during RL training).
- Proxying at the LLM API layer ensures unification of heterogeneous harness stacks (CLI, SDKs, binaries, agent loops, tool-use shells) into a common rollout protocol.

Polar’s builder supports two principal trajectory strategies: **per_request**—which records each LLM invocation as a separate, short trace, and **prefix_merging**—which merges adjacent invocations wherever prompt-token alignment permits, yielding longer context-trace chains and improved hardware batching [2605.24220].

## 3. Mathematical Framework: GRPO and Trajectory Processing

Polar natively implements Generalized Rollout Policy Optimization (GRPO), with each trajectory $\tau$ encoding precise (sampled) tokens, loss masks, log-probs under the rollout policy, and scalar reward $R(\tau)$:

- For each trace, sampled response tokens $z_t$ and corresponding $m_t\in\{0,1\}$ are preserved.
- Importance weight per trajectory:
  $$
  w(\tau) = \prod_{t: m_t=1} \frac{\pi_\theta(z_t|z_{<t})}{\pi_{\theta_{\text{old}}}(z_t|z_{<t})}
  $$
- Surrogate RL objective:
  $$
  J(\theta) = \mathbb{E}_{\tau\sim\pi_{\theta_{\text{old}}}} [w(\tau) \cdot R(\tau)]
  $$
- First-order gradient estimator:
  $$
  \nabla_\theta J(\theta) \approx \mathbb{E}_\tau \left[ \sum_t m_t \nabla_\theta \log\pi_\theta(z_t|z_{<t}) \cdot R(\tau) \right]
  $$
- Practical update applies a baseline $b$ (moving average of $R$):
  $$
  \theta \leftarrow \theta + \alpha \mathbb{E}_\tau \left[ \sum_t m_t \nabla_\theta \log\pi_\theta(z_t|z_{<t}) (R(\tau) - b) \right]
  $$

After prefix-merging, the RL loss per trajectory is:
$$
L_{RL}(\theta; \tau) = -\sum_{t} m_t \cdot A(\tau) \cdot \log \pi_\theta(z_t|z_{<t})
$$
where $A(\tau)=R-b$ [2605.24220].

## 4. Workflow, Scheduling, and System Implementation

The operational cycle in Polar is designed for high-throughput, low-latency RL training:

- The trainer issues batched `TaskRequests` to the Rollout Server, specifying the policy version.
- Gateway nodes execute agent harnesses, proxy LLM calls, and reconstruct traces; upon completion, the reward evaluator assigns each trace a scalar reward.
- All intermediate results are communicated asynchronously; sessions report to registered callbacks, and completed traces are batched for RL gradient updates.
- Only the rollout service holds environment runtimes, agent containers, and proxy backends; training can proceed independently and asynchronously.

Pseudocode illustrates this procedure, with strict pipeline separation between submission, execution/proxying, trace building, reward computation, and trainer update. Key implementation optimizations (fast container startup, UDS for intra-container IPC, robust recovery from container/job failures) further improve efficiency [2605.24220, 2603.18815].  

## 5. Empirical Evaluation and Performance Metrics

Polar demonstrates substantial efficacy for agentic RL in multi-harness contexts:

- On SWE-Bench Verified (2,438 issues, Qwen3.5-4B base), Polar achieves:
  - Codex harness: 3.8% → 26.4% pass@1 (Δ = +22.6)
  - Claude Code: 29.8% → 34.6% (Δ = +4.8)
  - Qwen Code: 34.6% → 35.2% (Δ = +0.6)
  - Pi: 34.2% → 40.4% (Δ = +6.2)
- Trajectory builder ablations show a 5.4× reduction in wall-clock time using prefix_merging over per_request, and GPU utilization improved from ~20% to ~87.7%.
- System throughput scales near-linearly with node count, and decoupled deployment yields up to 1.4× higher throughput than standard coupled (in-process) designs, primarily by eliminating pod and batch-level stragglers [2605.24220, 2603.18815].

## 6. Comparison to Related Decoupled Rollout Architectures

Polar builds upon and generalizes principles found in roll-out-as-a-service and decoupled RL systems:

- **RollArt** and **ROLL Flash** disaggregate RL pipelines at the stage level (e.g., separating prefill, decoding, environment, reward, and training with hardware-affinity mapping), employing trajectory-level asynchrony to mask stragglers and achieve up to 2.05× (RollArt) and 2.72× (ROLL Flash) speedup relative to synchronous baselines [2512.22560, 2510.11345].
- **DART** employs a four-module asynchronous pipeline, highlighting per-worker model sync and dynamic data curation. Efficiency improvements are demonstrated in both GPU and environment utilization [2509.23866].
- **ProRL Agent** provides a rollout-as-a-service blueprint—with HTTP endpoints, sandboxed runtimes, and three-stage asynchronous execution—which has directly influenced Polar’s API and modularity [2603.18815].
- **Heddle** advances the trajectory-level paradigm by treating scheduling (“when”), placement (“where”), and resource allocation (“how many GPUs”) as fully orthogonal axes, optimized via learning-based prediction, dynamic programming, and simulated annealing. This offers a complementary approach to Polar's stage-level pipeline, and these systems are readily composable [2603.28101].
- **ROSE** introduces cooperative elasticity by harvesting idle serving GPU capacity for rollouts without breaching SLOs, using fine-grained VMM-based memory/page remapping and shard/sparsity-aware cross-cluster weight transfer. These strategies translate directly into best practices for robust, scalable decoupled rollout design [2605.06534].

## 7. Design Trade-Offs, Extensions, and Best Practices

Polar’s fully decoupled design entails several empirical and engineering considerations:

- *Credit assignment and batching*: Prefix_merging in trajectory reconstruction improves gradient variance and hardware utilization relative to fine-grained (per_request) traces at the expense of added logic for strict token-prefix alignment.
- *Harness-agnosticity*: Achieves broad coverage for existing agent stacks, but requires robust schema normalization and extensive format detection in the LLM proxy.
- *Isolation*: All agent/container initialization, environment I/O, and LLM proxy compute are separated from RL trainers, enabling tight packing, rootless HPC deployments, and easier cloud scaling.
- *Hardware Utilization and Scheduling*: Asynchronous service APIs, affinity routes, and independent worker pools exploit cloud and on-premises hardware efficiently, yielding up to 87.7% GPU utilization in typical runs [2605.24220].
- *Opportunities for further optimization*: Integration with trajectory-level schedulers [2603.28101], cooperative elasticity [2605.06534], or hierarchical rollout coordination are directly compatible with Polar’s abstraction boundaries.

Polar substantiates a practical blueprint for modern, large-scale agentic RL systems—demonstrating that strict decoupling, token-faithful trajectory capture, and fully asynchronous rollout services are necessary to maximize real-world throughput and fidelity in reinforcement learning for LLM-driven agents [2605.24220].

Source: https://www.emergentmind.com/topics/polar-decoupled-agentic-rl-rollout-architecture