Polar: Agentic RL on Any Harness at Scale

Abstract: Reinforcement learning for language agents increasingly depends on custom harnesses that manage long-running context, multi-turn tool use and multi-agent orchestration. However, porting these harnesses into RL environment interfaces remains difficult and often loses important training signals. We bridge this gap with polar, a rollout framework for scalable asynchronous RL over arbitrary agent harnesses. Polar treats the agent harness as a black box: it proxies LLM API calls, records token-level model interactions, and reconstructs token-faithful trajectories for training. Each rollout node efficiently manages runtime prewarming, agent execution, trajectory reconstruction, and evaluation in parallel, exposing asynchronous service endpoints that can be consumed by independent trainers at scale. This decoupled design makes Polar agnostic to agent harnesses, training infrastructure, and RL algorithms while improving compute utilization for long-running agent workloads. We validate polar by training agents on software-engineering tasks with popular coding harnesses. Using simple GRPO, polar improves Qwen3.5-4B by 22.6, 4.8, 0.6 and 6.2 points on SWE-Bench Verified with the Codex, Claude Code, Qwen Code and Pi harnesses, respectively. We further demonstrate Polar for offline data generation over custom harnesses and ablate trajectory reconstruction strategies. Polar rewrites its preceding work, Prorl Agent, and has been registered as one of NeMo Gym environments.

• The paper introduces a decoupled rollout architecture to perform harness-native reinforcement learning without modifying underlying agent harnesses.
• It leverages a model API proxy to reconstruct token-faithful trajectories, reducing trainer stream size by over 80% and boosting GPU utilization.
• Empirical results show significant reward gains across diverse coding harnesses, validating enhanced policy performance in unfamiliar action protocols.

Polar: Decoupled Agentic RL Rollout Architecture for Harness-Native Training

Motivation and Problem Formulation

Reinforcement learning (RL) for LLM-based agents is rapidly shifting from atomic, single-step language tasks toward sustained agentic workloads involving multi-turn tool use, code editing, orchestration, and interaction with heterogeneous environments. The complexity of these harnesses, including closed-source binaries and containerized agents, presents significant integration barriers for RL pipelines. Prior frameworks such as SkyRL-Agent and PRIME-RL require agent-specific integration, often forcing harness reengineering, compromising token-fidelity, or missing native execution details crucial for policy adaptation.

Polar addresses the central challenge: enabling RL training across arbitrary agent harnesses without reengineering or intruding on the harness logic. It achieves this by treating the agent harness as a black-box environment and leveraging the model API boundary as the universal interface for RL data collection, thereby sidestepping internal integration complexities. This approach fundamentally decouples rollout execution from trainer implementation, supporting scalable rollout-as-a-service architectures and harness-agnostic RL policy optimization.

Figure 1: Polar architecture exposes rollout server and gateway nodes, allowing black-box harness execution with token-level trajectory capture and asynchronous evaluation.

Architectural Components and Execution Model

Polar operationalizes this design with two primary architectural abstractions:

• Rollout Server: Handles durable task scheduling, session expansion, result aggregation, and interface with trainers. Each rollout session encapsulates the unique runtime, harness specification, evaluation criteria, and callback orchestration.
• Gateway Nodes: Execute agent sessions, host provider-compatible model API proxies, reconstruct token-faithful trajectories, evaluate outcomes, and tear down resources. Gateway-level asynchronous staging decomposes INIT, RUNNING, and POSTRUN phases into isolated worker pools, maximizing GPU utilization by enabling concurrent preparation, execution, and evaluation.

This decoupling ensures that expensive CPU-bound preparation and evaluator setup proceed off the critical path, avoiding bottlenecks on the active agent GPU-bound pipeline.

Figure 2: Gateway-level asynchronous staging allows independent resource scheduling for runtime initialization, harness execution, and evaluation.

Model API Boundary and Proxy-Based RL Rollout

Polar's key innovation is the model API proxy as the rollout boundary. Rather than requiring agents to implement RL-compatible environment APIs or trace instrumentation, Polar routes all harness model calls through a proxy that auto-detects provider schemas (Anthropic, OpenAI, Google), normalizes requests for inference, captures token IDs, sampled tokens, and log probabilities, and reconstructs RL trajectories externally. This ensures harnesses remain unchanged, enabling compatibility with CLI agents, binaries, and package-managed tools.

Figure 3: Polar uses proxy boundary for rollout instead of framework-bound environment interface, maintaining deployment-native harness execution.

Token-Faithful Trajectory Reconstruction

Polar implements rigorous token-fidelity in trajectory construction, addressing retokenization drift issues prevalent in prior frameworks. Two builder strategies are provided:

• Per-Request Builder: Emits a trace for each model completion, maximally conservative but potentially fragmenting long-horizon sessions.
• Prefix-Merging Builder: Merges completions across append-only conversation chains, copying only sampled assistant tokens as trainable tokens and masking canonical interstitial context, ensuring that RL gradients are only applied to behavior-policy tokens.

The prefix-merging mechanism handles context compaction, subagent boundaries, and harness-specific prompt rewriting, supporting multi-agent orchestration without compromising token-level data integrity.

Figure 4: Trajectory reconstruction visualizes session-level conversation compaction and subagent spawning, with prefix merging preserving behavior-policy fidelity.

Scaling and Asynchronous RL Rollouts

The rollout-as-a-service architecture enables trainer-agnostic asynchronous RL. Trainers can consume completed trajectory batches without being blocked by rollouts. Polar maximizes GPU utilization by emitting merged traces, reducing the number of trainer updates and wall-clock time, thus accelerating policy optimization.

Figure 5: Polar enables efficient asynchronous RL pipelines by separating rollout computation from trainer update steps.

Empirical ablation shows prefix-merging reduces trainer stream size by over 80% and speeds training by >5$\times$ compared to per-request reconstruction, with average rollout GPU utilization of 87.7% against 20.4% for per-request, minimizing reward hacking and optimizing credit assignment.

Harness-Native RL: Experiments and Results

Polar is validated on SWE-Gym with GRPO training of Qwen3.5-4B across four coding harnesses: Codex, Claude Code, Qwen Code, and Pi. RL training via Polar improves outcome rewards across the board, most dramatically on harnesses presenting unfamiliar action protocols (Codex: 3.8% $\to$ 26.4% pass@1, 22.6 pt gain). The harnesses with native-compatible policies show smaller but consistent gains (Claude Code: +4.8 pt, Pi: +6.2 pt, Qwen Code: +0.6 pt), directly attributing improvement to harness-aware RL.

Figure 6: SWE-Gym GRPO training curves demonstrate reward gains for each harness, highlighting impact of RL on unfamiliar execution paths.

Practical Impact: Offline SFT Data Generation

Beyond online RL, Polar serves as a distributed data-generation substrate for SFT. Using Qwen3.5-122B-A10B, Polar generated 504 accepted trajectories out of 1,638 SWE-Gym instances—30.8% acceptance—at high scale and efficiency, supporting rich multi-turn conversations and agent tool calls against containerized environments. The output corpus, stratified per repository, demonstrates utility for downstream training, preference modeling, and verifier calibration without harness modification.

Implications, Limitations, and Future Directions

Polar introduces a paradigm shift in agentic RL infrastructure: RL rollouts can be decoupled from trainer logic and conducted over harness-native executions without intrusive integration. This approach enhances flexibility, reproducibility, and scaling, especially as harness complexity and diversity increase. The token-fidelity framework addresses correctness in RL credit assignment, mitigating drift and reward hacking. The asynchronous rollout-trainer architecture unlocks new efficiency frontiers for large-scale RL pipelines.

Potential future developments include extension to multimodal harnesses, richer reward assignment via process reward models, and more complex agent orchestration featuring concurrent tool calls and nested subagents. Polar's infrastructure paves the way for scalable, harness-agnostic RL and offline data generation, laying a foundation for broader agentic AI capabilities.

Conclusion

Polar delivers a robust and scalable rollout infrastructure for agentic RL, harnessing the model API boundary as a universal interface, ensuring token-faithful trajectory reconstruction, and efficiently scaling agent rollouts across arbitrary harnesses. Its architectural decoupling allows RL trainers to optimize policies aligned with deployment-native agent execution, accelerating adaptation and correctness in complex environments. Polar establishes a practical foundation for harness-agnostic RL training and large-scale offline data generation, marking a substantive advancement in agent-level reinforcement learning systems.