---
title: Tapered Off-Policy REINFORCE (TOPR)
url: https://www.emergentmind.com/topics/tapered-off-policy-reinforce-topr
type: topic
---

# Tapered Off-Policy REINFORCE (TOPR)

Tapered Off-Policy REINFORCE (TOPR) is an algorithm designed for stable and efficient reinforcement learning-based fine-tuning of large language models (LLMs) in fully offline, off-policy settings, where training data $\tau = (x,y)$ are collected from a fixed reference or behavior policy $\mu$ and not from the current optimized policy $\pi_\theta$. TOPR introduces an asymmetric, “tapered” variant of importance sampling to balance rapid supervised-style learning for positive examples with bounded, low-variance updates for negative examples—achieving robust training without the need for explicit Kullback-Leibler (KL) regularization. This methodological innovation enables joint utilization of both positive and negative samples, enhancing data efficiency and empirical performance while maintaining implementational simplicity characteristic of Monte Carlo algorithms [2503.14286].

## 1. Foundations and Motivation

The standard reinforcement learning objective for LLM fine-tuning seeks to maximize expected reward:
\[
J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).
\]
On-policy REINFORCE estimates $\nabla_\theta J(\pi_\theta)$ using trajectories sampled from the current policy. However, in practical LLM pipelines, data collection is asynchronous and typically performed with a fixed reference model $\mu$, leading to an off-policy setup. Naive off-policy gradient estimation, which disregards the behavior-target mismatch, yields an unbounded surrogate objective subject to collapse when negative rewards are present.

The canonical off-policy remedy, importance sampling (IS), produces unbiased estimates but incurs excessive variance due to the product of per-token probability ratios in long sequences. Truncated importance sampling (TIS) mitigates this by clipping the IS weight to $[0,1]$, yet applies this symmetrically to all samples, slowing the learning of rare high-reward sequences. These tradeoffs motivate the development of TOPR, which introduces an asymmetric tapering of importance weights to achieve both rapid positive learning and bounded, stable updates for negatives.

## 2. The TOPR Algorithm and Surrogate Objective

Define $T^+ = \{\tau : R(\tau) \ge 0\}$ and $T^- = \{\tau : R(\tau) < 0\}$. The TOPR update utilizes the general taper function:
\[
\rho(x; a, b) = 
\begin{cases}
a\left(1 + \ln\frac{x}{a}\right), & x < a \\
x, & a \le x \le b \\
b\left(1 + \ln\frac{x}{b}\right), & x > b
\end{cases}
\]
with $\rho(x; a, b) \le x$ and $\rho(x; a, b) = x$ for $x \in [a, b]$. The TOPR surrogate objective is:
\[
J_{\rm TOPR}(\pi) =
\sum_{\tau \in T^+} \mu(\tau) \, \rho(\pi(\tau)/\mu(\tau); a^+, b^+) R(\tau) +
\sum_{\tau \in T^-} \mu(\tau) \, \rho(\pi(\tau)/\mu(\tau); a^-, b^-) R(\tau)
\]
The canonical configuration sets $a^+ = b^+ = 1$ (full SFT on positives) and $a^- = 0, b^- = 1$ (TIS on negatives), yielding an update of the form:
\[
\hat{\nabla}_\theta J_{\rm TOPR} =
\sum_{\tau \in T^+} \mu(\tau) R(\tau) \nabla \log \pi(\tau)
+
\sum_{\tau \in T^-} \mu(\tau) \left[\frac{\pi(\tau)}{\mu(\tau)}\right]_0^1 R(\tau) \nabla \log \pi(\tau)
\]
where $[\cdot]_a^b = \min\{\max(\cdot, a), b\}$. This asymmetric weighting induces rapid, supervised-style learning on positives (regardless of importance ratio), while using clipped importance weights to provide stable, vanishing contributions from negatives as $\pi(\tau)\to0$, closing the possibility of unbounded collapse.

## 3. Tapering Mechanism, Handling of Positives and Negatives

TOPR’s tapering mechanism is characterized by its asymmetric assignment of importance weights:
- For $R(\tau) \ge 0$, the weight is set to $1$. This ensures that even rare positive samples drive learning with full supervised cross-entropy, countering the vanishing gradients of IS for unlikely positives.
- For $R(\tau) < 0$, the weight is $[\pi(\tau)/\mu(\tau)]_{0}^{1}$, meaning it adheres to TIS for negatives. As $\pi(\tau)$ decreases, the update contribution vanishes, preventing instability.

All generations—positive and negative—are incorporated, forming a dataset $D = \{(x_i, y_i^j, R(x_i, y_i^j))\}$. Each $(x, y, R)$ is weighted according to the aforementioned rule. This contrasts with positive-only fine-tuning that discards negatives, thereby increasing “training data efficiency” by extracting learning signal from every inference. TOPR reduces the prevalence of invalid-format outputs and enhances pass@1 and majority-vote self-consistency metrics compared to baselines.

## 4. Baseline Parameter and Dataset Composition

The baseline parameter $c$ plays a distinct role in off-policy REINFORCE relative to its variance-reducing use on-policy. Adjusting the baseline $R' = R - c$ not only shifts the relative weighting of positive and negative samples but also can rebalance the effective dataset composition post hoc:
\[
\tilde{p} = \frac{p(1-c)}{1 + (1-2p)c}
\]
where $p$ is the proportion of positives. Tuning $c$ thus allows compensation for imbalanced or suboptimal data distributions without resampling; $c<0$ increases positive influence, $c>0$ amplifies negatives. Within TOPR, the baseline introduces a soft KL regularization when $b^- = b^+ > 0$, further stabilizing the learning when the policy deviates markedly from the data distribution.

Empirically, the optimal baseline is not the mean reward, but one that induces $\tilde{p} \approx 10\%$–$20\%$ positives. Both an excess and a paucity of positives degrade accuracy. This effect is consistent for both raw positive splits of 10% and 50% [2503.14286 (Fig. 4)].

## 5. Pseudocode and Implementation

A single offline iteration of TOPR proceeds as follows:

```python
# TOPR pseudocode
for i in 1..m:
    for j in 1..n:
        y_ij ~ mu(.|x_i)
        r_ij = R(x_i, y_ij)    # +1 or -1
dataset D = {(x_i, y_ij, r_ij)}

for (x, y, r) in D:
    if r >= 0:
        alpha = 1
    else:
        alpha = clip(pi_theta(y|x)/mu(y|x), lower=0, upper=1)
    loss = -alpha * r * log pi_theta(y|x)
    # take gradient step w.r.t. theta on loss
```

Key differences vis-à-vis vanilla off-policy REINFORCE (always $\alpha=1$) and OPR/TIS (symmetric clipping) are the asymmetric weight assignment and the inclusion of all samples. This setup allows exploitation of both easy and hard examples from the reference distribution.

## 6. Empirical Results: GSM8K and MATH Benchmarks

Experiments on GSM8K and MATH reasoning benchmarks use Llama 3 8B as the base model (with comparisons to Llama 3 70B and DeepSeek-R1 8B). Data generation employs vLLM (T=1, top_p=1, top_k=500, max_len=512). For each prompt, $n=16$ CoT generations (GSM8K, 8-shot) or $n=32$ (MATH, 4-shot) are drawn; rewards are $+1$ for exact matches and $-1$ otherwise. Optimization uses Adafactor, learning rate $5\times10^{-7}$, per-token loss averaging, no KL penalty, and gradient clipping norm $=1.0$.

Performance is measured via pass@1 (single-sample accuracy) and maj@K (majority voting). Bootstrap confidence intervals are reported.

| Method         | GSM8K pass@1 | SFT-only | PPO | DPO | TOPR (8B) | TOPR vs. 70B |
|----------------|--------------|----------|-----|-----|-----------|--------------|
| Baseline       | ~54%         | ~60%     | ~58%| ~61%| ~65%      | Matches      |

Key findings:
- TOPR with 8B matches or exceeds 70B maj@16 GSM8K performance after a few iterations when paired with “Anna Karenina sampling.”
- Negative samples are critical; omitting them reduces overall and self-consistency accuracy.
- Optimal dataset positive fractions ($\sim10\%$–$20\%$) are robust to the original data distribution.
- Training an 8B generative verifier on MATH increases verifier accuracy to 71% (base: 32%) and boosts self-consistency from 56.7% (no verifier) to 61.5%.

## 7. Stability Properties and Theoretical Guarantees

With $a^-=0$, the surrogate objective $J_{\rm TOPR}(\pi)$ is bounded above for all reward functions. This stands in contrast to naive off-policy REINFORCE, which is unbounded and vulnerable to collapse, and PPO, whose surrogate has zero gradient outside a narrow importance ratio range. Empirical ablations demonstrate:
- Naive off-policy REINFORCE collapses when $\pi$ drifts from $\mu$.
- PPO training saturates quickly off-policy.
- DPO improves moderately but is surpassed by TOPR.
- OPR without clipping suffers from exploding gradients, sensitive to the gradient norm hyperparameter.
- Noncanonical $(a^+, a^-, b^+, b^-)$ choices sacrifice either stability or sample efficiency.

TOPR’s asymmetric tapering yields a distinctive balance, offering stable learning in the absence of a KL penalty, rapid acquisition of rare successful trajectories, and full utilization of all generated supervisory data [2503.14286].

Source: https://www.emergentmind.com/topics/tapered-off-policy-reinforce-topr