Polar: Decoupled Agentic RL Rollout Architecture

Updated 27 May 2026

Polar is a scalable, asynchronous RL architecture designed for LLM-driven agents, abstracting complex harnesses as black boxes for efficient training.
The architecture enables effective interactions with LLM APIs, preserving credit assignment and allowing seamless integration across various RL environments.
Empirical evaluations show Polar significantly enhances pass rates and GPU utilization while reducing processing time and improving system throughput.

Polar is a decoupled, agentic reinforcement learning (RL) rollout architecture designed for scalable, asynchronous training over arbitrary LLM-driven agent harnesses. It enables RL for language agents that depend on complex, multi-turn harnesses by abstracting the harness as a black box: LLM API calls are proxied, token-level interactions are faithfully captured, and all trainable events are reconstructed as RL trajectories. Its architecture exposes parallel, asynchronous rollout endpoints consumable by independent trainers at scale, rendering Polar agnostic to agent environments, RL algorithms, and compute substrate. This decoupling allows for increased hardware utilization, full preservation of credit assignment, and plug-and-play integration across research harnesses, accelerating RL progress for long-horizon, multi-agent, and tool-use workloads (Xu et al., 22 May 2026).

1. Architectural Decomposition and Core Components

Polar operates in a multi-tiered service structure with explicit separation of environment execution, proxying, trajectory reconstruction, and reward evaluation:

Rollout Server acts as the central scheduler, receiving TaskRequests—specifying harness, sample count, runtime specs, trajectory builder, evaluator, and callback URL—splitting them into sessions, persisting state, and allocating work to gateway nodes.
Gateway Nodes manage per-session runtimes (e.g., via Docker or Apptainer), launch unmodified agent harnesses with LLM endpoint redirection, and include an LLM-API proxy to intercept every model call. Execution is pipelined across worker pools (INIT: runtime prewarm; READY: buffer management; RUNNING: harness execution/proxy capture; POSTRUN: trajectory reconstruction/evaluation/teardown).
LLM-API Proxy is co-located with each agent runtime, intercepting completions in diverse provider formats, normalizing to a standard schema with log-probabilities, invoking vLLM or Triton backends, and recording prompt/response tokens and log-probs. Returned data is reshaped to match the expected format, maintaining harness transparency.
Trajectory Reconstruction Module constructs RL-trainable traces from the sequence of intercepted completions, supporting strategies that enable token-faithful loss masking and merging (e.g., strict prefix alignment).
Asynchronous APIs allow submission (POST /rollout/task/submit), status, heartbeat, and results polling, decoupling the rollout service from any particular trainer implementation.

This separation aligns conceptually with other state-of-the-art rollout-as-a-service systems, enabling independent scaling and fault domains across rollout, orchestration, and training (Zhang et al., 19 Mar 2026).

2. Decoupled Design Principles and Token-Faithful Rollout

Polar’s foundational principle is to treat the agent harness—including multi-turn, multi-agent, or tool-augmented code—as an opaque executable, requiring no modifications. LLM API calls are intercepted directly at the boundary between the agent and inference backend, thus:

All sampled (behavior-policy) tokens and per-token log-probabilities are captured exactly as emitted by the rollout policy.
Generated completions, associated context, and non-trainable scaffolding (tool calls, canonical prompt content) are normalized and (optionally) masked out in training loss via a binary loss_mask.
Credit assignment is preserved at the token level, with strict avoidance of “retokenization drift” (i.e., the divergence between tokens generated in the agent and those re-encoded during RL training).
Proxying at the LLM API layer ensures unification of heterogeneous harness stacks (CLI, SDKs, binaries, agent loops, tool-use shells) into a common rollout protocol.

Polar’s builder supports two principal trajectory strategies: per_request—which records each LLM invocation as a separate, short trace, and prefix_merging—which merges adjacent invocations wherever prompt-token alignment permits, yielding longer context-trace chains and improved hardware batching (Xu et al., 22 May 2026).

3. Mathematical Framework: GRPO and Trajectory Processing

Polar natively implements Generalized Rollout Policy Optimization (GRPO), with each trajectory $\tau$ encoding precise (sampled) tokens, loss masks, log-probs under the rollout policy, and scalar reward $R(\tau)$ :

For each trace, sampled response tokens $z_t$ and corresponding $m_t\in\{0,1\}$ are preserved.
Importance weight per trajectory:

$w(\tau) = \prod_{t: m_t=1} \frac{\pi_\theta(z_t|z_{<t})}{\pi_{\theta_{\text{old}}}(z_t|z_{<t})}$

Surrogate RL objective:

$J(\theta) = \mathbb{E}_{\tau\sim\pi_{\theta_{\text{old}}}} [w(\tau) \cdot R(\tau)]$

First-order gradient estimator:

$\nabla_\theta J(\theta) \approx \mathbb{E}_\tau \left[ \sum_t m_t \nabla_\theta \log\pi_\theta(z_t|z_{<t}) \cdot R(\tau) \right]$

Practical update applies a baseline $b$ (moving average of $R$ ):

$\theta \leftarrow \theta + \alpha \mathbb{E}_\tau \left[ \sum_t m_t \nabla_\theta \log\pi_\theta(z_t|z_{<t}) (R(\tau) - b) \right]$

After prefix-merging, the RL loss per trajectory is:

$R(\tau)$ 0

where $R(\tau)$ 1 (Xu et al., 22 May 2026).

4. Workflow, Scheduling, and System Implementation

The operational cycle in Polar is designed for high-throughput, low-latency RL training:

The trainer issues batched TaskRequests to the Rollout Server, specifying the policy version.
Gateway nodes execute agent harnesses, proxy LLM calls, and reconstruct traces; upon completion, the reward evaluator assigns each trace a scalar reward.
All intermediate results are communicated asynchronously; sessions report to registered callbacks, and completed traces are batched for RL gradient updates.
Only the rollout service holds environment runtimes, agent containers, and proxy backends; training can proceed independently and asynchronously.

Pseudocode illustrates this procedure, with strict pipeline separation between submission, execution/proxying, trace building, reward computation, and trainer update. Key implementation optimizations (fast container startup, UDS for intra-container IPC, robust recovery from container/job failures) further improve efficiency (Xu et al., 22 May 2026, Zhang et al., 19 Mar 2026).

5. Empirical Evaluation and Performance Metrics

Polar demonstrates substantial efficacy for agentic RL in multi-harness contexts:

On SWE-Bench Verified (2,438 issues, Qwen3.5-4B base), Polar achieves:
- Codex harness: 3.8% → 26.4% pass@1 (Δ = +22.6)
- Claude Code: 29.8% → 34.6% (Δ = +4.8)
- Qwen Code: 34.6% → 35.2% (Δ = +0.6)
- Pi: 34.2% → 40.4% (Δ = +6.2)
Trajectory builder ablations show a 5.4× reduction in wall-clock time using prefix_merging over per_request, and GPU utilization improved from ~20% to ~87.7%.
System throughput scales near-linearly with node count, and decoupled deployment yields up to 1.4× higher throughput than standard coupled (in-process) designs, primarily by eliminating pod and batch-level stragglers (Xu et al., 22 May 2026, Zhang et al., 19 Mar 2026).

Polar builds upon and generalizes principles found in roll-out-as-a-service and decoupled RL systems:

RollArt and ROLL Flash disaggregate RL pipelines at the stage level (e.g., separating prefill, decoding, environment, reward, and training with hardware-affinity mapping), employing trajectory-level asynchrony to mask stragglers and achieve up to 2.05× (RollArt) and 2.72× (ROLL Flash) speedup relative to synchronous baselines (Gao et al., 27 Dec 2025, Lu et al., 13 Oct 2025).
DART employs a four-module asynchronous pipeline, highlighting per-worker model sync and dynamic data curation. Efficiency improvements are demonstrated in both GPU and environment utilization (Li et al., 28 Sep 2025).
ProRL Agent provides a rollout-as-a-service blueprint—with HTTP endpoints, sandboxed runtimes, and three-stage asynchronous execution—which has directly influenced Polar’s API and modularity (Zhang et al., 19 Mar 2026).
Heddle advances the trajectory-level paradigm by treating scheduling (“when”), placement (“where”), and resource allocation (“how many GPUs”) as fully orthogonal axes, optimized via learning-based prediction, dynamic programming, and simulated annealing. This offers a complementary approach to Polar's stage-level pipeline, and these systems are readily composable (Zhang et al., 30 Mar 2026).
ROSE introduces cooperative elasticity by harvesting idle serving GPU capacity for rollouts without breaching SLOs, using fine-grained VMM-based memory/page remapping and shard/sparsity-aware cross-cluster weight transfer. These strategies translate directly into best practices for robust, scalable decoupled rollout design (Gao et al., 7 May 2026).

7. Design Trade-Offs, Extensions, and Best Practices

Polar’s fully decoupled design entails several empirical and engineering considerations:

Credit assignment and batching: Prefix_merging in trajectory reconstruction improves gradient variance and hardware utilization relative to fine-grained (per_request) traces at the expense of added logic for strict token-prefix alignment.
Harness-agnosticity: Achieves broad coverage for existing agent stacks, but requires robust schema normalization and extensive format detection in the LLM proxy.
Isolation: All agent/container initialization, environment I/O, and LLM proxy compute are separated from RL trainers, enabling tight packing, rootless HPC deployments, and easier cloud scaling.
Hardware Utilization and Scheduling: Asynchronous service APIs, affinity routes, and independent worker pools exploit cloud and on-premises hardware efficiently, yielding up to 87.7% GPU utilization in typical runs (Xu et al., 22 May 2026).
Opportunities for further optimization: Integration with trajectory-level schedulers (Zhang et al., 30 Mar 2026), cooperative elasticity (Gao et al., 7 May 2026), or hierarchical rollout coordination are directly compatible with Polar’s abstraction boundaries.

Polar substantiates a practical blueprint for modern, large-scale agentic RL systems—demonstrating that strict decoupling, token-faithful trajectory capture, and fully asynchronous rollout services are necessary to maximize real-world throughput and fidelity in reinforcement learning for LLM-driven agents (Xu et al., 22 May 2026).