LLMs Can Learn to Reason Via Off-Policy RL

Abstract: Reinforcement learning (RL) approaches for LLMs frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

• The paper introduces OAPL, an off-policy RL algorithm that overcomes on-policy constraints to improve stability and efficiency in LLM training.
• Empirical results show that OAPL achieves superior Pass@k metrics on math reasoning and code generation tasks even under significant policy lag.
• By using KL regularization with a lagged inference policy, OAPL circumvents high variance issues typical of importance sampling corrections.

Off-Policy RL for Reasoning in LLMs: An Analysis of OAPL

Introduction

The paper "LLMs Can Learn to Reason Via Off-Policy RL" (2602.19362) systematically interrogates the strict on-policy assumption that underlies prominent RL frameworks for LLM alignment, particularly Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO). The authors introduce an alternative, fully off-policy approach: Optimal Advantage-based Policy Optimization with Lagged Inference Policy (OAPL). In contrast to recent efforts that attempt to patch off-policy behavior in LLM RL pipelines with elaborate importance sampling corrections or inference engine modifications, OAPL is explicitly designed to operate under substantial policy lag and log-probability mismatches. The paper provides both a closed-form derivation of the OAPL loss and empirical analyses across competitive math reasoning and code generation tasks.

Motivation and Problem Statement

Contemporary LLM RL post-training infrastructures inherently introduce off-policyness due to asynchronous data pipelines and implementation discrepancies between trainers and inference engines. These discrepancies generate mismatched log-probabilities, causing deviations from the on-policy premise foundational to REINFORCE, PPO, and GRPO. Prior works frequently attempt to correct for this via importance sampling (IS) at either the token or sequence level, or by architecting inference engines for stricter policy-token synchrony. These approaches are challenged by increased variance (in the case of IS) or computational overhead and incomplete alignment (in the case of inference engine tweaks).

The core questions addressed are:

• Whether strictly on-policy algorithms are necessary for effective LLM RL post-training.
• Whether robust, scalable, fully off-policy RL algorithms can be devised and empirically validated given the architecture and operational constraints of current LLM infrastructures.

The OAPL Framework

The OAPL algorithm centralizes KL-regularized RL with respect to the lagged inference policy:

$$\max_{\pi} \mathbb{E}_{x, y\sim \pi(\cdot | x)} r(x,y) - \beta \mathrm{KL}( \pi \| \pi_{vLLM} )$$

where, crucially, $\pi_{vLLM}$ denotes the potentially asynchronous, inference engine-based policy. The closed-form solution yields an optimal policy $\pi^*$ in the exponential family of $\pi_{vLLM}$ weighted by exponentiated normalized reward. Critically, the optimal value function $V^*(x)$ is computed as a log-sum-exp (softmax) over rewards on rollouts sampled from $\pi_{vLLM}$. This structure enables estimation of the optimal advantage $A^*(x, y)$ and leads to a squared regression objective:

$$\min_\pi \sum_{x} \sum_{i=1}^G \left( \beta \ln \frac{ \pi(y_i|x) }{\pi_{vLLM}(y_i|x)} - ( r(x,y_i) - \hat V^*(x) ) \right)^2$$

Training proceeds asynchronously: the trainer is updated using off-policy rollouts from the inference engine and periodically synchronizes with it, with data buffering handled carefully to ensure consistency of policy references within each OAPL optimization phase.

Comparison to Prior RL Approaches

OAPL provides a marked departure from both IS-corrected GRPO and other off-policy RL approaches in this context.

• Against GRPO/IS methods: OAPL avoids the high variance and bias management issues induced by IS, eliminating the need for aggressive ratio clipping, outlier deletion, or heavy reliance on on-policy data refresh. Instead, stability is achieved through explicit KL regularization to the lagged inference policy and a least-squares regression structure.
• Versus on-policy $A^*PO$: While $A^*PO$ assumes synchronized policy references throughout, OAPL directly incorporates policy lag, updating the inference policy only after many (potentially hundreds) of gradient steps.
• Sample Efficiency: OAPL supports substantial reuse of prior samples and tolerates high degrees of off-policyness in the data buffer, circumventing the limits imposed by strict on-policy learning.

Experimental Results

Math Reasoning Benchmarks

On competition mathematics datasets (AIME 25, HMMT 25 Feb/Nov, BRUMO 25), OAPL surpasses GRPO across all Pass@k metrics ($k=1,5,10$), demonstrating both superior average maximum accuracy and enhanced stability during training.

Figure 1: Benchmark accuracy comparison of OAPL and GRPO on three math datasets across Pass@1, @5, @10 metrics.

Training curves further reveal that OAPL maintains stable convergence and higher accuracy under asynchronous regimes, outperforming GRPO throughout the optimization trajectory.

Figure 2: Mean accuracy across benchmarks over training iterations for OAPL and GRPO (Pass@1, @5, @10).

OAPL's critical advantage is further amplified as the synchronization lag increases. Experimental results show that even at substantial lag intervals ($K=100$), OAPL remains robust, stably improving policy performance without collapse or instability.

Figure 3: (Left) Policy entropy remains stable under OAPL but collapses under GRPO; (Right) OAPL accuracy is resilient to policy lag (large $K$).

Entropy and Pass@k Scaling

OAPL-trained policies preserve and even enhance entropy, in contrast to the rapid entropy collapse observed with GRPO. This entropy preservation correlates with improved Pass@k scaling for large $k$, contradicting claims that RL post-training sharpens distributions at the expense of diversity and higher-k performance.

Code Generation

On LiveCodeBench, OAPL matches or slightly outperforms DeepCoder—a public baseline reliant on GRPO and multiple training heuristics—on Pass@k for all measured values of $k$. Crucially, OAPL achieves this with approximately one-third of the training samples required by DeepCoder, highlighting improved sample efficiency.

Figure 4: (Left) Pass@k scaling for OAPL, DeepCoder, and the base model; (Right) OAPL achieves equivalent accuracy with substantially fewer training generations.

Theoretical and Practical Implications

OAPL provides a principled route to scalable, stable off-policy RL in LLM post-training, a regime that is increasingly relevant as training is distributed, asynchronous, and architecturally heterogeneous. By removing the IS bottleneck and not requiring on-policy rollouts, OAPL enables:

• Efficient exploitation of stale or buffered experience,
• Fully asynchronous optimization,
• Stable learning under significant trainer–inferencer policy divergence,
• Significant sample/computational cost reductions relative to policy-gradient approaches reliant on re-weighting.

The explicit KL regularization with respect to the inference policy addresses policy drift far more directly and robustly than PPO/GRPO-style clipped ratio constraints.

The empirical findings also challenge prevailing narratives regarding the limitations of RL post-training, particularly the notion that RL simply redistributes probability mass without improving higher-order metrics such as Pass@k for large $k$.

Future Directions

The OAPL approach opens multiple avenues for future research, including:

• Off-policy value function learning for enhanced credit assignment,
• Leveraging large-scale offline datasets (e.g., human preference data or synthetic reward-labeled corpora) without synchronous rollout refresh,
• Integration with other RL techniques (e.g., distributional RL, entropy maximization via alternative regularization forms),
• Detailed theoretical exploration of stability boundaries and optimal choices of synchronization intervals for diverse architectures and reward structures.

Conclusion

The study demonstrates that deterministic on-policy RL approaches are non-essential for effective LLM alignment in complex reasoning tasks. OAPL’s off-policy objective, rooted in KL-regularized RL theory, provides both theoretical soundness and substantial empirical gains in stability, sample efficiency, and final performance—without the complexity or drawbacks of importance sampling. This work paves the way for broader adoption of off-policy RL in LLM training and alignment pipelines.