Tapered Off-Policy REINFORCE (TOPR)

• The paper introduces TOPR, which employs an asymmetric tapering of importance weights to balance rapid positive updates with bounded negative updates.
• TOPR integrates both positive and negative samples into a unified objective, enhancing data efficiency without explicit KL regularization.
• Empirical results on GSM8K and MATH benchmarks demonstrate that TOPR achieves competitive performance with simpler, stable training dynamics.

Tapered Off-Policy REINFORCE (TOPR) is an algorithm designed for stable and efficient reinforcement learning-based fine-tuning of LLMs in fully offline, off-policy settings, where training data $\tau = (x,y)$ are collected from a fixed reference or behavior policy $\mu$ and not from the current optimized policy $\pi_\theta$. TOPR introduces an asymmetric, “tapered” variant of importance sampling to balance rapid supervised-style learning for positive examples with bounded, low-variance updates for negative examples—achieving robust training without the need for explicit Kullback-Leibler (KL) regularization. This methodological innovation enables joint utilization of both positive and negative samples, enhancing data efficiency and empirical performance while maintaining implementational simplicity characteristic of Monte Carlo algorithms (Roux et al., 18 Mar 2025).

1. Foundations and Motivation

The standard reinforcement learning objective for LLM fine-tuning seeks to maximize expected reward: $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$ On-policy REINFORCE estimates $\nabla_\theta J(\pi_\theta)$ using trajectories sampled from the current policy. However, in practical LLM pipelines, data collection is asynchronous and typically performed with a fixed reference model $\mu$, leading to an off-policy setup. Naive off-policy gradient estimation, which disregards the behavior-target mismatch, yields an unbounded surrogate objective subject to collapse when negative rewards are present.

The canonical off-policy remedy, importance sampling (IS), produces unbiased estimates but incurs excessive variance due to the product of per-token probability ratios in long sequences. Truncated importance sampling (TIS) mitigates this by clipping the IS weight to $[0,1]$, yet applies this symmetrically to all samples, slowing the learning of rare high-reward sequences. These tradeoffs motivate the development of TOPR, which introduces an asymmetric tapering of importance weights to achieve both rapid positive learning and bounded, stable updates for negatives.

2. The TOPR Algorithm and Surrogate Objective

Define $T^+ = \{\tau : R(\tau) \ge 0\}$ and $T^- = \{\tau : R(\tau) < 0\}$. The TOPR update utilizes the general taper function: $$\rho(x; a, b) = \begin{cases} a\left(1 + \ln\frac{x}{a}\right), & x < a \ x, & a \le x \le b \ b\left(1 + \ln\frac{x}{b}\right), & x > b \end{cases}$$ with $\mu$0 and $\mu$1 for $\mu$2. The TOPR surrogate objective is: $\mu$3 The canonical configuration sets $\mu$4 (full SFT on positives) and $\mu$5 (TIS on negatives), yielding an update of the form: $\mu$6 where $\mu$7. This asymmetric weighting induces rapid, supervised-style learning on positives (regardless of importance ratio), while using clipped importance weights to provide stable, vanishing contributions from negatives as $\mu$8, closing the possibility of unbounded collapse.

3. Tapering Mechanism, Handling of Positives and Negatives

TOPR’s tapering mechanism is characterized by its asymmetric assignment of importance weights:

• For $\mu$9, the weight is set to $\pi_\theta$0. This ensures that even rare positive samples drive learning with full supervised cross-entropy, countering the vanishing gradients of IS for unlikely positives.
• For $\pi_\theta$1, the weight is $\pi_\theta$2, meaning it adheres to TIS for negatives. As $\pi_\theta$3 decreases, the update contribution vanishes, preventing instability.

All generations—positive and negative—are incorporated, forming a dataset $\pi_\theta$4. Each $\pi_\theta$5 is weighted according to the aforementioned rule. This contrasts with positive-only fine-tuning that discards negatives, thereby increasing “training data efficiency” by extracting learning signal from every inference. TOPR reduces the prevalence of invalid-format outputs and enhances pass@1 and majority-vote self-consistency metrics compared to baselines.

4. Baseline Parameter and Dataset Composition

The baseline parameter $\pi_\theta$6 plays a distinct role in off-policy REINFORCE relative to its variance-reducing use on-policy. Adjusting the baseline $\pi_\theta$7 not only shifts the relative weighting of positive and negative samples but also can rebalance the effective dataset composition post hoc: $\pi_\theta$8 where $\pi_\theta$9 is the proportion of positives. Tuning $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$0 thus allows compensation for imbalanced or suboptimal data distributions without resampling; $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$1 increases positive influence, $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$2 amplifies negatives. Within TOPR, the baseline introduces a soft KL regularization when $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$3, further stabilizing the learning when the policy deviates markedly from the data distribution.

Empirically, the optimal baseline is not the mean reward, but one that induces $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$4–$$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$5 positives. Both an excess and a paucity of positives degrade accuracy. This effect is consistent for both raw positive splits of 10% and 50% [(Roux et al., 18 Mar 2025) (Fig. 4)].

5. Pseudocode and Implementation

A single offline iteration of TOPR proceeds as follows:

$\mu$0

Key differences vis-à-vis vanilla off-policy REINFORCE (always $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$6) and OPR/TIS (symmetric clipping) are the asymmetric weight assignment and the inclusion of all samples. This setup allows exploitation of both easy and hard examples from the reference distribution.

6. Empirical Results: GSM8K and MATH Benchmarks

Experiments on GSM8K and MATH reasoning benchmarks use Llama 3 8B as the base model (with comparisons to Llama 3 70B and DeepSeek-R1 8B). Data generation employs vLLM (T=1, top_p=1, top_k=500, max_len=512). For each prompt, $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$7 CoT generations (GSM8K, 8-shot) or $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$8 (MATH, 4-shot) are drawn; rewards are $$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)], \quad R(\tau) \in \mathbb{R}, \; \tau = (x, y).$$9 for exact matches and $\nabla_\theta J(\pi_\theta)$0 otherwise. Optimization uses Adafactor, learning rate $\nabla_\theta J(\pi_\theta)$1, per-token loss averaging, no KL penalty, and gradient clipping norm $\nabla_\theta J(\pi_\theta)$2.

Performance is measured via pass@1 (single-sample accuracy) and maj@K (majority voting). Bootstrap confidence intervals are reported.

Method	GSM8K pass@1	SFT-only	PPO	DPO	TOPR (8B)	TOPR vs. 70B
Baseline	~54%	~60%	~58%	~61%	~65%	Matches

Key findings:

• TOPR with 8B matches or exceeds 70B maj@16 GSM8K performance after a few iterations when paired with “Anna Karenina sampling.”
• Negative samples are critical; omitting them reduces overall and self-consistency accuracy.
• Optimal dataset positive fractions ($\nabla_\theta J(\pi_\theta)$3–$\nabla_\theta J(\pi_\theta)$4) are robust to the original data distribution.
• Training an 8B generative verifier on MATH increases verifier accuracy to 71% (base: 32%) and boosts self-consistency from 56.7% (no verifier) to 61.5%.

7. Stability Properties and Theoretical Guarantees

With $\nabla_\theta J(\pi_\theta)$5, the surrogate objective $\nabla_\theta J(\pi_\theta)$6 is bounded above for all reward functions. This stands in contrast to naive off-policy REINFORCE, which is unbounded and vulnerable to collapse, and PPO, whose surrogate has zero gradient outside a narrow importance ratio range. Empirical ablations demonstrate:

• Naive off-policy REINFORCE collapses when $\nabla_\theta J(\pi_\theta)$7 drifts from $\nabla_\theta J(\pi_\theta)$8.
• PPO training saturates quickly off-policy.
• DPO improves moderately but is surpassed by TOPR.
• OPR without clipping suffers from exploding gradients, sensitive to the gradient norm hyperparameter.
• Noncanonical $\nabla_\theta J(\pi_\theta)$9 choices sacrifice either stability or sample efficiency.

TOPR’s asymmetric tapering yields a distinctive balance, offering stable learning in the absence of a KL penalty, rapid acquisition of rare successful trajectories, and full utilization of all generated supervisory data (Roux et al., 18 Mar 2025).