Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

Abstract: Large scale reinforcement learning has become a central tool for improving reasoning in LLMs. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.

• The paper’s main contribution is its formulation of Lambert-tempered target distributions to embed inherent pessimism in off-policy RL for LLM reasoning.
• The methodology introduces a β-shifted mean advantage update that stabilizes policy updates and prevents entropy collapse under policy lag.
• Empirical evaluations demonstrate that the modified Lambert objective consistently achieves higher rewards and stable entropy compared to traditional OAPL objectives.

Off-Policy Learning-to-Reason: Pessimism via Lambert Policies in LLM Reinforcement Learning

Introduction

This work presents a rigorous theoretical and empirical investigation into off-policy reinforcement learning (RL) objectives for LLM reasoning. The central insight is that the empirical success of ratio-free off-policy objectives, such as OAPL and $A^\star$-PO, is attributable to their implicit pessimistic bias, which arises from their advantage normalization and target density. This pessimism is formalized via the emergence of Lambert-tempered target distributions, whose regime (pessimistic, exponential, or unstable) is governed by population-level properties of the chosen advantage. The authors propose a principled $\beta$-shifted mean advantage construction that guarantees the policy update remains in the desired pessimistic regime, conferring stability and robustness absent in nominal OAPL objectives. Their analysis provides a new lens for understanding and designing robust RL algorithms for LLM reasoning.

Regimes of Off-Policy Policy Optimization

Modern RL pipelines for LLMs are inherently off-policy, predominantly due to stale or asynchronously generated samples from lagged models. Despite this, common approaches rely on trust-region or importance-weighted objectives originally designed for near-on-policy data (e.g., PPO, GRPO). These approaches often struggle under large policy lags, as high-variance importance weights and aggressive clipping induce instability and premature entropy collapse.

Recent empirical work demonstrates that dispensing with importance sampling—i.e., training directly on behavior policy samples via ratio-free objectives—can yield much stronger and more robust RL for LLM reasoning. Canonical examples are $A^\star$-PO and OAPL, which instantiate a regression objective over the log-ratio of the target and behavior policy, aligning the log-likelihood ratio to an advantage (usually constructed from rewards). However, the precise mechanism underlying their stability (or instability) has, until now, lacked a precise theoretical explanation.

Lambert-Tempered Targets: Implicit Pessimism in Off-Policy Objectives

The authors analyze ratio-free off-policy objectives within the class of regularized weighted maximum log-likelihoods, penalized by a quadratic term in the sentence log-ratio from the behavior policy. The optimal policy under this objective arises from solving a self-consistency equation, whose solution is expressed via the principal branch of the Lambert $W$ function (see Proposition~\ref{prop:mopo-lambert-sentence}).

The critical insight is that the normalization (centering) of the advantage controls the regime of the induced target distribution:

• Pessimistic regime ($\tau_s > 0$): The policy is conservative; high-advantage completions are upweighted sub-exponentially, preventing over-commitment to outliers. The density ratio grows at most linearly for large advantages.
• Exponential boundary ($\tau_s=0$): The policy coincides with the exponential tilt familiar from KL-regularized RL. This boundary is fragile; small baseline shifts move the regime into pessimistic or unstable territory.
• Unstable regime ($\tau_s < 0$): The target is more aggressive than exponential, highly sensitive to noise in the advantage, and induces entropy collapse.

Notably, it is shown that nominal implementations of $A^\star$-PO/OAPL fall into the unstable regime due to the bias in the log-sum-exp advantage estimator for finite group sizes. This instability is not benign: small deviations in advantage normalization drive the population target into the regime where high-reward samples are over-amplified, destabilizing learning.

Implications for Advantage Construction

From this analysis, temperature-decoupling heuristics—where the advantage and regularization temperatures are set separately—are vindicated: increasing the advantage temperature relative to regularization (as done in practice) moves the update toward the pessimistic regime, thereby stabilizing RL. However, heuristic tuning introduces new hyperparameters and does not offer formal robustness guarantees.

The authors' key proposal is to guarantee pessimism by construction: shift the mean group advantage upward by $\beta$, setting $A_i = r_i - \bar{r} + \beta$. This guarantees $\beta$0, which, by Jensen's inequality, places the induced Lambert multiplier in the conservative ($\beta$1) regime for both population and empirical objectives. This practically eliminates the regime sensitivity observed in standard OAPL, leading to more robust and reliable off-policy LLM RL.

Empirical Evaluation

Empirical results demonstrate substantial gains in stability and robustness conferred by the Lambert-advantage modification, compared to OAPL. Across a range of regularization strengths (varying $\beta$2) and under increased policy staleness (larger lag $\beta$3), the Lambert objective:

• Maintains stable entropy, avoiding the collapse that plagues OAPL at low $\beta$4 or high lag.
• Achieves higher and more consistent evaluation rewards on compositional math reasoning benchmarks.

These findings are summarized in the results below.

Figure 1: Lambert improves off-policy stability. Top: Lambert remains stable while OAPL collapses for small $\beta$5. Bottom: Lambert maintains higher reward and stable entropy under stale rollouts, while OAPL becomes unstable.

Crucially, the Lambert regime is less sensitive to hyperparameter misspecification, as it is designed to be robust to both aggressive regularization and significant distributional shift between the behavior and target policy.

Theoretical and Practical Implications

Theoretical implications include a precise understanding of how advantage normalization determines the shape and stability of self-normalized off-policy learning objectives. The work aligns the observed empirical “pessimism” in stable RL procedures with the existence of a Lambert-tempered regime, and formally connects regularization, advantage centering, and off-policy robustness.

Practically, this yields an actionable advantage estimator that obviates the need for fragile temperature tuning and offers greater reliability, especially as lag and dataset size grow. This insight can inform both future advances in off-policy preference optimization and the general design of RL algorithms for LLMs—especially for domains where high-variance or multi-modal rewards are present.

Looking forward, the theoretical lens provided here suggests that similar pessimism-inducing modifications or principled advantage baselining should be explicitly considered for both sentence-level and token-level off-policy RL. Extensions to token-level objectives, connection to $\beta$6-regularized and robust RL, and application to more complex or high-dimensional reasoning tasks represent promising avenues.

Conclusion

This work advances the understanding of why ratio-free off-policy RL objectives succeed or fail for LLM reasoning. By characterizing the induced distribution as Lambert-tempered and tying its regime directly to advantage normalization, the authors illuminate both the strengths and fragility of OAPL-style objectives. Their $\beta$7-shifted mean advantage update targets the conservative regime by construction, leading to improved stability and sample efficiency. These insights provide a principled basis for future developments in robust and scalable RL for LLMs.