ReMix: Reincarnating Mix-policy PPG

• The paper demonstrates that ReMix reduces rollout costs by reusing off-policy trajectories, achieving SOTA-level performance with significant efficiency gains.
• ReMix combines mix-policy proximal policy gradients with an increased update-to-data ratio and a KL-convex constraint to balance stability and adaptability.
• The approach introduces a policy reincarnation stage that shifts from biased off-policy updates to stable on-policy refinement, preserving asymptotic performance.

Reincarnating Mix-policy Proximal Policy Gradient (ReMix) is an off-policy reinforcement finetuning method for LLMs that enables on-policy RFT methods such as PPO and GRPO to reuse trajectories generated earlier in training. Introduced in "Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for LLM" (Liang et al., 9 Jul 2025), ReMix combines three elements: mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio, a KL-Convex policy constraint that jointly anchors the policy to the original pretrained base model and the most recent policy, and a policy reincarnation mechanism that shifts from efficient early-stage off-policy learning to steadier asymptotic on-policy improvement. The method is motivated by the observation that standard on-policy RFT discards past trajectories after each policy update, which raises rollout cost substantially for long-chain reasoning.

1. Problem setting and design rationale

ReMix is situated in reinforcement finetuning for reasoning-oriented LLMs, where rollout generation dominates compute and time. The motivating claim is that most existing RFT methods are on-policy in nature and therefore do not fully utilize data generated during the past learning process. In the formulation associated with ReMix, this produces a stringent bottleneck for economic and efficient scaling, especially when large models perform long, multi-step reasoning (Liang et al., 9 Jul 2025).

The central design choice is to revive off-policy RL for LLM finetuning without abandoning the optimization structures of PPO- or GRPO-style training. The method therefore mixes data from the current policy and several recent policies in each update, increases the UTD ratio so that more gradient steps are taken per rollout, regularizes policy updates with a convex KL construction, and then switches back to an on-policy regime at a designated reincarnation point. This suggests that ReMix is best understood not as a replacement for PPO or GRPO, but as a wrapper that augments on-policy RFT with controlled historical data reuse.

A key contrast with standard RFT is the treatment of policy anchoring. Standard RFT adds a KL penalty $D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{base}})$ to prevent catastrophic drift, whereas ReMix explicitly treats a static base-model anchor as potentially over-restrictive for specialization. Its alternative is to interpolate between the original pretrained model and the latest policy, thereby balancing stability and flexibility during training.

2. Mix-policy proximal policy gradient

Let $\pi_k$ denote the policy at iteration $k$, and let $\{\pi_{k-i}\}_{i=0\ldots N}$ denote the set of recent policies, with $\pi_k$ the current policy. ReMix samples a minibatch of size $B$ from a mixture consisting of a fraction $p$ of off-policy trajectories from older policies $\pi_{k-i}$ for $i>0$, and a fraction $1-p$ of fresh on-policy data from $\pi_k$0. The importance ratio is defined as

$\pi_k$1

Using a clipped surrogate akin to PPO, the Mix-PPG objective at step $\pi_k$2 is

$\pi_k$3

Here $\pi_k$4, $\pi_k$5, and $\pi_k$6 is the advantage under the current policy (Liang et al., 9 Jul 2025).

To improve sample efficiency further, ReMix performs $\pi_k$7 gradient-update steps on each minibatch, so that the UTD ratio equals $\pi_k$8. With V-trace-corrected advantages, the reported gradient estimator is

$\pi_k$9

with

$k$0

Within the method’s internal logic, the mix-policy update is the primary source of rollout-efficiency gains. The data explicitly attributes accelerated early learning to the combination of Mix-PPG and higher UTD, while also noting that this comes at the cost of off-policy bias.

3. KL-Convex policy constraint and policy reincarnation

The KL-Convex policy constraint is defined as

$k$1

where $k$2. Because $k$3 is convex in its first argument, this mixture remains convex in $k$4 (Liang et al., 9 Jul 2025).

The intended effect is to anchor the updated policy simultaneously to the original base model $k$5 and the most recent policy $k$6. Early in training, this enforces stronger inheritance from the pretrained model; later, as $k$7 decays, the constraint permits more local adaptation around the current trajectory of optimization. The Stage 1 decay schedule is

$k$8

ReMix then introduces policy reincarnation to address the claim that even clipped off-policy updates accumulate bias and can limit final performance. The method uses two stages. In Stage 1, for steps $k$9, it runs Mix-PPG with off-policy reuse, UTD $\{\pi_{k-i}\}_{i=0\ldots N}$0, and KL-Convex regularization toward $\{\pi_{k-i}\}_{i=0\ldots N}$1. In Stage 2, for steps $\{\pi_{k-i}\}_{i=0\ldots N}$2, it reverts to on-policy PPO or GRPO with fresh data only, while resetting the base anchor to $\{\pi_{k-i}\}_{i=0\ldots N}$3 so that $\{\pi_{k-i}\}_{i=0\ldots N}$4, again with KL-Convex regularization to $\{\pi_{k-i}\}_{i=0\ldots N}$5. The method describes this reset as eliminating accumulated off-policy bias and restoring on-policy stability for asymptotic improvement.

4. Algorithmic workflow and experimental configuration

The algorithmic procedure follows a two-stage schedule. During Stage 1, ReMix samples fresh prompts of size $\{\pi_{k-i}\}_{i=0\ldots N}$6 and generates rollouts under the current policy, samples past rollouts of size $\{\pi_{k-i}\}_{i=0\ldots N}$7 from a history buffer containing data from recent policies, forms a mixed training batch, stores the current batch in history while dropping the oldest entries beyond the history window, and repeats the policy update $\{\pi_{k-i}\}_{i=0\ldots N}$8 times using $\{\pi_{k-i}\}_{i=0\ldots N}$9. At the reincarnation point, it resets the anchor with $\pi_k$0 and clears history. Stage 2 then samples only fresh rollouts and optimizes a standard PPO or GRPO objective plus the KL-Convex term (Liang et al., 9 Jul 2025).

The reported experimental setup uses Qwen-1.5B and Qwen-7B, both distilled by DeepSeek-R1, and trains on DeepScaler Preview, described as approximately 40K math problems. Evaluation is conducted on five math reasoning benchmarks: AIME’24 validation with 195 problems, AMC’23 with 25, Minerva with approximately 7K, OlympiadBench with 100, and MATH500 with 1,000. Rollouts use temperature $\pi_k$1, top-$\pi_k$2, and maximum context length of 8,192 tokens.

The principal hyperparameters reported for ReMix are $\pi_k$3, $\pi_k$4, $\pi_k$5, and $\pi_k$6 for PPO or $\pi_k$7 for GRPO, together with the $\pi_k$8 decay schedule above. The reported compute budget is 50 hours on $\pi_k$9 GPUs for Qwen-1.5B and 75 hours on $B$0 GPUs for Qwen-7B.

5. Empirical performance and rollout efficiency

The primary evaluation statistic is average Pass@1 over the five benchmarks. For the 1.5B setting, the paper reports ReMix-PPO at 52.10% with 79K rollouts and 350 training steps, compared with DeepScaleR (GRPO) at 53.35% with 2,519K rollouts, PPO at 50.60% with 230K rollouts and 900 steps, GRPO at 48.64% with 410K rollouts and 200 steps, and ReMix-GRPO at 50.53% with 368K rollouts and 200 steps. For the 7B setting, it reports AceReason-Nemotron at 63.24% with 3,584K rollouts, PPO at 60.35% with 51K rollouts and 200 steps, and ReMix-PPO at 64.39% with 11K rollouts and 75 steps. The abstract further reports 63.27% and 64.39% for the 7B model with 0.007M and 0.011M response rollouts, respectively, using 50 and 75 training steps (Liang et al., 9 Jul 2025).

Setting	Average Pass@1	Rollout volume
DeepScaleR (GRPO), 1.5B	53.35%	2,519K
ReMix-PPO, 1.5B	52.10%	79K
AceReason-Nemotron, 7B	63.24%	3,584K
ReMix-PPO, 7B	64.39%	11K

From these results, the paper characterizes ReMix as achieving SOTA-level performance with over $B$1 to $B$2 reduction in training cost in terms of rollout data volume. The numerical basis for that claim is explicit in the 1.5B comparison against DeepScaleR and the 7B comparison against AceReason-Nemotron. A plausible implication is that ReMix’s principal contribution is not merely incremental improvement in final Pass@1, but a change in the cost-performance frontier for reasoning-oriented RFT.

6. Analytical findings, failure modes, and adaptation guidelines

The paper reports two analytical findings that function as warnings against a simplistic reading of off-policy reuse. The first is the Whipping Effect on Response Length. Under off-policy training, most advantages $B$3 are reported to be negative, so minimizing the approximate loss

$B$4

drives the policy to reduce the importance ratio $B$5, which in turn shrinks trajectories toward shorter sequences. Empirically, higher $B$6 is reported to produce shorter responses, a faster early accuracy boost, and worse later performance (Liang et al., 9 Jul 2025).

The second is the collapse mode of self-reflection. Vanilla PPO is reported to maintain a high self-reflection rate, expressed through use of the "> ..." subroutine, throughout training. By contrast, Mix-PPG initially improves accuracy rapidly but suppresses reflective tokens and response length, which degrades final performance. ReMix’s reincarnation stage is reported to restore both length and reflection, thereby recovering asymptotic accuracy. This suggests that efficient off-policy reuse can distort reasoning style as well as optimization dynamics.

The reported component analysis assigns distinct roles to the three major elements. Mix-PPG plus UTD accelerates early learning by a factor of $B$7 to $B$8 in the first 100 steps, while introducing off-policy bias. KL-Convex guards against collapse by anchoring updates between $B$9 and $p$0, balancing exploration against over-specialization. Policy reincarnation switches from an efficient but biased regime to a stable on-policy regime and is presented as the mechanism that preserves final performance.

For adaptation to new tasks, the paper gives explicit operational ranges rather than a universal prescription: choose $p$1 to trade efficiency against stability, set UTD $p$2 to $p$3, use $p$4 to $p$5 recent policies in the mix, choose $p$6 so that Stage 1 ends when off-policy bias begins to dominate as measured by validation accuracy, and decay $p$7 from $p$8 to $p$9 over Stage 1. These guidelines frame ReMix as a tunable methodology rather than a fixed training recipe.