Policy Reincarnation in Reinforcement Learning

• Policy reincarnation is a reinforcement learning design pattern that resets the learned policy mid-training to balance sample efficiency and stability.
• It employs a two-stage approach where an off-policy phase is followed by a mid-training reset that enables pure on-policy refinement.
• Empirical evidence from ReMix demonstrates improved performance and cost reduction by mitigating off-policy bias and leveraging historical trajectories.

Searching arXiv for the supplied paper and closely related work on policy reincarnation / reuse / consolidation. arXiv search query: all:"policy reincarnation" OR ti:"Reincarnating Reinforcement Learning" OR ti:"Policy Consolidation" OR ti:"Efficient Off-policy Reinforcement Finetuning for LLM" Policy reincarnation is a reinforcement-learning design pattern in which a learned policy is not treated as a terminal artifact but as an intermediate object to be reset, regularized, selected, or re-expressed so that subsequent optimization can proceed from a better behavioral reference. In its most explicit recent formulation, the term denotes a two-stage training scheme inside ReMix for reinforcement finetuning of LLMs: an early off-policy, mix-policy phase is followed by a mid-training “reincarnation” event that replaces the original base reference by the current policy and restarts purely on-policy training (Liang et al., 9 Jul 2025). In a broader sense, closely related work treats reincarnation as reuse of prior computation, regularization by a policy’s own history, selection from a library of learned policies, or preservation of multiple policy variants for future transfer (Agarwal et al., 2022).

1. Canonical definition in ReMix

An explicit definition is given by ReMix, where policy reincarnation is an explicit two-stage training scheme inside Reincarnating Mix-policy Proximal Policy Gradient. The algorithm first trains a policy with an off-policy, mix-policy proximal PG objective using historical trajectories and a KL constraint to the original base model. It then “reincarnates” the policy at a predefined mid-training step $T$ by resetting the KL reference from the original base model $\pi_{\text{base}}$ to the current policy $\pi_T$, and by switching from mixed on/off-policy updates back to a purely on-policy proximal method such as PPO or GRPO (Liang et al., 9 Jul 2025).

Formally, the reincarnation event is captured by

$$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$

together with the objective transition

$$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$

or the GRPO equivalent. ReMix describes this as replacing the base model with the mix-policy RFT model in the mid way of training and restarting on-policy training, to achieve a seamless transition from efficient early-stage learning to steady asymptotic improvement (Liang et al., 9 Jul 2025).

This definition is narrower than generic warm-starting. Standard on-policy PPO or GRPO continually collects fresh trajectories, discards old data, and keeps the original base model as KL reference. Standard off-policy RL keeps learning from old data with high Update-To-Data ratios but can suffer from off-policy bias and instability in the limit. Policy reincarnation, by contrast, uses an off-policy flavored phase only for a finite early window, then resets the optimization problem around the learned policy and resumes a clean on-policy regime (Liang et al., 9 Jul 2025).

2. Two-stage objective, constraints, and schedule

Within ReMix, policy reincarnation operates as the bridge between three components: mix-policy proximal policy gradient with increased UTD ratio, a KL-Convex policy constraint, and the reincarnation event itself. In Stage 1, training uses a mixture of on-policy and historical off-policy data from recent policies $\{\pi_{k-i}\}$, with a higher UTD ratio $m$, so that each sampled batch is reused for multiple gradient updates. In Stage 2, training drops all historical replay and continues with pure on-policy PPO or GRPO around the new anchor $\pi_T$ (Liang et al., 9 Jul 2025).

The KL-Convex constraint regularizes the current policy against both a global anchor and a local anchor:

$$L_{\text{KLC}}(\theta;\pi_{\text{base}}) = \mathbb{E}_s\!\left[ \lambda\, D_{\mathrm{KL}}\!\big(\pi_\theta(\cdot\mid s)\parallel \pi_{\text{base}}(\cdot\mid s)\big) + (1-\lambda)\, D_{\mathrm{KL}}\!\big(\pi_\theta(\cdot\mid s)\parallel \pi_{k-1}(\cdot\mid s)\big) \right].$$

During Stage 1, $\pi_{\text{base}}$ is the original base LLM; after reincarnation, the base reference becomes $\pi_{\text{base}}$0. The schedule

$\pi_{\text{base}}$1

gradually shifts weight from global base alignment toward iterative refinement (Liang et al., 9 Jul 2025).

Algorithmically, the procedure is direct. Training starts with $\pi_{\text{base}}$2 and $\pi_{\text{base}}$3. In Stage 1, each step mixes fresh trajectories of size $\pi_{\text{base}}$4 with historical responses of size $\pi_{\text{base}}$5, stores the current policy and its rollouts in $\pi_{\text{base}}$6, and applies Mix-PPG with KLC for $\pi_{\text{base}}$7 updates. At the boundary between steps $\pi_{\text{base}}$8 and $\pi_{\text{base}}$9, the method sets $\pi_T$0, clears $\pi_T$1, and from then on uses no off-policy data. Stage 2 samples only fresh responses of size $\pi_T$2 and optimizes standard on-policy PPO or GRPO plus $\pi_T$3 (Liang et al., 9 Jul 2025).

The reported default choices make the intended operating regime concrete: $\pi_T$4, $\pi_T$5, $\pi_T$6, $\pi_T$7 for ReMix-PPO 1.5B, and $\pi_T$8 for ReMix-GRPO (Liang et al., 9 Jul 2025).

3. Motivation: sample efficiency, off-policy bias, and failure modes

The motivation for policy reincarnation in ReMix is a tension between early-stage efficiency and late-stage stability. Mix-PPG with high UTD reuses trajectories aggressively and gives very fast gains in Pass@1 accuracy in the first 50–100 steps, but large off-policy usage or very high UTD makes training unstable and degrades or collapses asymptotic performance. Purely on-policy PPO or GRPO shows stable long-term behavior and monotone-ish improvement, but is sample-inefficient because each iteration requires new rollouts and discards rich historical data (Liang et al., 9 Jul 2025).

Reincarnation separates these regimes. Stage 1 “squeeze[s] the soaked sponge”: it harvests signal from early trajectories, reuses them many times, and moves the policy far from the original base quickly. Stage 2 recenters optimization around $\pi_T$9, discards old experience, and continues with on-policy updates to obtain steady asymptotic improvement without accumulated off-policy bias (Liang et al., 9 Jul 2025).

The empirical evidence isolates the importance of the switch. On five math reasoning benchmarks—AIME’24, AMC’23, Minerva, OlympiadBench, and MATH500—ReMix reports an average Pass@1 accuracy of $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$0 for the 1.5B model with $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$1M response rollouts and 350 training steps, and $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$2 for the 7B model with $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$3M/$$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$4M response rollouts and 50/75 training steps. Compared with 15 recent advanced models, it reports SOTA-level performance with an over $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$5 to $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$6 reduction in training cost in terms of rollout data volume (Liang et al., 9 Jul 2025).

Ablations indicate that the mid-training reset is not optional. Full ReMix-PPO at 350 steps reaches $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$7 average score, whereas PPO at 500 steps reaches $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$8; ReMix-PPO without policy reincarnation reaches only $$\pi_{\text{base}} \Leftarrow \pi_T,\qquad \mathbb{H} \Leftarrow \emptyset,$$9, and “Mix-PPG only” reaches $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$0. Training curves show that Mix-PPG yields fast early gains but stalls or degrades, while ReMix preserves the early surge and then continues to improve steadily like PPO, a pattern described as “merged learning behavior” (Liang et al., 9 Jul 2025).

The same paper links policy reincarnation to specific off-policy pathologies. Under strong off-policyness, the “Whipping Effect” induces an implicit preference for shorter responses, importance ratios tend to grow along long trajectories, and optimization pressures the model toward shorter completions. As the off-policy fraction $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$1 increases, the method observes shorter responses, lower self-reflection rate, and eventually accuracy degradation or collapse. Reincarnation mitigates this by limiting the time spent under off-policy Mix-PPG: Stage 1 can learn a concise-reasoning tendency, while Stage 2 lets the model re-expand chain-of-thought length and reflection rate to the level that maximizes final accuracy (Liang et al., 9 Jul 2025).

4. Related formulations of policy self-reuse and transfer

Although ReMix provides the most explicit mid-training definition, several adjacent literatures implement closely related mechanisms. They differ mainly in what is “reincarnated”: the current policy’s own history, a teacher policy, a library entry, or a neighborhood of archived variants.

Work	Reused policy object	Mechanism
ReMix	$$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$2 learned in Stage 1	Reset KL reference and restart on-policy training
Policy Consolidation	Own past policies at multiple timescales	Cascade of hidden policies with KL regularization
Reincarnating RL / QDagger	Teacher policy and teacher replay	Offline pretraining plus online weaning
Bayesian Policy Reuse	Policy library $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$3	Belief over task types selects a reusable policy
TeLAPA	Per-task archive of elites	Latent-aligned policy neighborhoods and few-shot selection

Policy Consolidation regularizes a visible policy $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$4 by a cascade of hidden policies $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$5 that remember the agent’s policy at a range of timescales. The policy-space loss replaces parameter-space distances with KL divergences between adjacent depths, and deeper policies are constrained more strongly through $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$6. This is a continual, multi-timescale form of reincarnation: the current policy is regularized by its own history without task boundaries (Kaplanis et al., 2019).

“Reincarnating Reinforcement Learning” generalizes the reuse-centric view. Its policy-to-value setting treats a teacher policy $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$7 and teacher replay $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$8 as prior computational work to be reused by a standalone value-based student. QDagger combines offline pretraining on $$L^{\text{Mix-PPG}}(\theta)\longrightarrow L^{\text{PPO}}(\theta),$$9 with online training on student data and a decaying distillation term, thereby weaning the student off the teacher while aiming to exceed teacher performance (Agarwal et al., 2022).

Bayesian Policy Reuse offers a more episodic interpretation. Instead of learning a new policy for each short-lived task instance, an agent maintains a policy library $\{\pi_{k-i}\}$0, a belief $\{\pi_{k-i}\}$1 over latent task types, and updates that belief by Bayes’ rule using observed signals $\{\pi_{k-i}\}$2. Policy reincarnation here is selection and reuse rather than mid-training reset: the best library policy for the inferred type is revived quickly to minimize regret relative to the best policy in the library in hindsight (Rosman et al., 2015).

TeLAPA reframes continual RL around “skill-aligned neighborhoods” rather than single-model preservation. It stores per-task archives of behaviorally diverse but competent policies, aligns them in a shared latent space, and selects candidates for future tasks by few-shot transfer probes. Its results show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative (Lillo et al., 16 Apr 2026).

5. Reincarnation in LLM alignment and agentic RL

In a broader sense, the idea has migrated from classical RL policy reuse to alignment and agentic systems where “policies” may be business rules, expert trajectories, or modular skills. The common pattern is that external or historical control signals are not merely appended to context; they are transformed into reusable objects that the learner can internalize, reactivate, or revise.

$\{\pi_{k-i}\}$3 treats long business-policy documents as knowledge to be internalized and recalled at inference time without including the full policy in context. Its three-stage pipeline—continual SFT with policies in context, CoT-augmented SFT with no policies in context, and GRPO with a PolicyRecall reward $\{\pi_{k-i}\}$4 plus hallucination and length penalties—teaches the model to recall and apply relevant policies during chain-of-thought reasoning. Its best model outperforms the baseline by 16 points, surpasses comparable in-context baselines of similar model size by 3 points, and uses 40% fewer words (Dipta et al., 15 Mar 2026).

RePO addresses off-policy knowledge absorption in RL for LLMs by prompting the policy to comprehend external knowledge and rephrase it into trajectories that conform to its own stylistic and parametric distribution. It dynamically replaces low-reward rollouts with these rephrased, high-quality trajectories while preserving a standard GRPO objective. In reincarnation terms, the external policy’s reasoning is projected into the learner’s own distribution before optimization, rather than forcing the learner to match alien trajectories directly (Xia et al., 11 Feb 2026).

ReSkill extends the idea to modular policy components in agentic RL. It embeds an assertion-driven skill creator, within-group rollout sampling, and Thompson Sampling with adaptive discounting into the GRPO loop, so that skills are created, tested, refined, and pruned as the policy improves. Here, what is reincarnated is not a single frozen policy but reusable strategy modules whose survival depends on whether they continue to help the evolving policy under current rewards (He et al., 1 Jun 2026).

6. Limitations, misconceptions, and open problems

A recurrent misconception is that policy reincarnation is equivalent to simple warm-starting. The surveyed formulations are stricter. In ReMix, reincarnation changes both the KL reference and the data regime; in Policy Consolidation, the current policy is tied to a hierarchy of its own past behaviors; in TeLAPA, reuse depends on preserving neighborhoods rather than a single checkpoint (Liang et al., 9 Jul 2025).

A second misconception is that preserving the best past policy is sufficient. TeLAPA explicitly reports that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood. Policy Consolidation likewise notes that hidden policies are often not coherent standalone policies after task switches, implying that what is preserved may be a distributed behavioral trace rather than a directly executable controller (Lillo et al., 16 Apr 2026, Kaplanis et al., 2019).

Practical fragilities are also consistent across formulations. In ReMix, overly large $\{\pi_{k-i}\}$5 behaves like pure off-policy RL and can trigger shortening of responses, drop in reflection rate, and performance collapse under high $\{\pi_{k-i}\}$6 or $\{\pi_{k-i}\}$7; overly small $\{\pi_{k-i}\}$8 loses most efficiency gains. In Reincarnating RL, the behavior of the student depends on teacher quality, $\{\pi_{k-i}\}$9-step choice, $m$0, and temperature $m$1, and several baselines collapse when imitation pressure is removed (Liang et al., 9 Jul 2025, Agarwal et al., 2022).

For LLM-alignment variants, further complications appear. $m$2 reports residual hallucinations, substantial upfront CoT generation cost, and brittleness when policies change. RePO depends on reliable off-policy data and adds the cost of generating rephrased trajectories. ReSkill increases prompt length, relies on trigger design, and leaves open how to scale from a small skill bank to much larger libraries (Dipta et al., 15 Mar 2026, Xia et al., 11 Feb 2026, He et al., 1 Jun 2026).

These limitations suggest a broader research direction rather than a closed recipe. One plausible implication is that policy reincarnation is best understood not as a single algorithm but as a family of reuse-centric optimization strategies: some reset the objective around a newly learned policy, some regularize against multiple past selves, some select among reusable policies under uncertainty, and some preserve a neighborhood or skill library so that future optimization remains plastic rather than merely stable.