- The paper introduces On-Policy Replay, which fills cross-task buffers with high-scoring rollouts from the previous model and reduces forgetting without modifying the standard supervised fine-tuning objective.
- Experiments on TRACE show that OPR-RU cuts backward-transfer degradation by 42โ46% versus tuned vanilla replay at a 1% replay budget, while reaching โ0.65 percentage points at a 10% budget.
- A low-score ablation confirms that both on-policy sampling and reward filtering are essential, while the label-free OPR-SC variant performs comparably overall but can preserve confidently incorrect outputs.
Motivation and problem statement
Sequential supervised fine-tuning (SFT) remains the default mechanism for adapting instruction-tuned LLMs to a stream of downstream tasks, yet it produces severe catastrophic forgetting: on the TRACE benchmark, plain sequential SFT degrades backward transfer (BWT) by 10.85โ17.07 percentage points across three backbones. Recent analyses converge on a mechanistic account in which forgetting at each update is upper-bounded by the per-update KL divergence between consecutive policies evaluated on historical prompts. Because training on samples drawn from the previous policy ฯฮธtโ1โโ shrinks this KL by construction, on-policy data is the natural lever against forgetting.
Existing methods deliver this on-policy signal through the objective: SDFT, OPSD, and GKD add an explicit per-token KL term against a frozen teacher copy of the model, paying a teacher forward pass per gradient step and introducing schedule-sensitive hyperparameters; RazorSFT rewrites current-task targets with verifier-passing self-samples. This paper takes a different position: the constraint is on the data distribution the loss sees, not on the loss itself. The cross-task replay bufferโthe one pipeline object that already carries information across task boundariesโis repurposed as the delivery vehicle for the on-policy signal, leaving the SFT training loop entirely unchanged.
Method
On-Policy Replay (OPR) operates as follows. At the end of stage i, the checkpoint ฯฮธiโโ rolls out one response per historical prompt under low-temperature decoding (temperature 0.1 via vLLM). A scorer ranks the resulting (prompt, response) pairs, and the top-b entriesโwith b=ฯni+1โ allocated evenly across prior tasksโform the replay buffer Bi+1โ, which is mixed with next-task data under standard token-level cross-entropy. There is no teacher, no auxiliary loss, and no target rewrite.
Two scorers are considered. OPR-RU uses each task's own evaluation metric (exact match, ROUGE-L F1, edit similarity, or SARI), aligning buffer construction with the evaluation criterion by design. OPR-SC scores rollouts by the model's own length-normalized log-probabilityโa free by-product of sampling that removes any dependence on ground-truth labels, making the method applicable when reliable task annotations are unavailable.
Theoretical account
The paper grounds OPR in a KL-shrinkage analysis. Building on the bound from prior work, the loss change on historical tasks satisfies, to leading order,
ฮiโโคCโ
Exโผphistโโ[KL(ฯฮธtโโโฅฯฮธtโ1โโ)]+o(โฅฮธtโโฮธtโ1โโฅ2).
For the self-confidence scorer, top-b selection is exactly a per-sample upper bound on the loss at ฮธtโ1โ: every buffer item lies in a high-likelihood region of the previous checkpoint. Under smoothness assumptions, this yields a batch-gradient norm bound, and the identity Eqโ[โฮธโโ]=โฮธโKL(qโฅฯฮธโ) shows that SGD on the replay portion descends i0, pulling the policy toward a high-reward restriction of its previous self. Combining these gives
i1
so the reward filter directly controls the leading-order forgetting term. Two caveats are stated plainly: the derivation assumes the rule-based scorer correlates positively with log-likelihood (an empirical assumption, not an identity), and the bound covers only the replay portion of each batchโthe new-task gradient remains unconstrained, making the mechanism intrinsically one-sided.
This analysis makes a falsifying prediction: replacing high-score rollouts with low-score ones from the same pool preserves on-policy sampling but breaks the loss bound, and should degrade BWT below even vanilla off-policy replay. The experiments confirm both parts.
Experimental results
The evaluation uses TRACE's canonical eight-task sequence with 5,000 examples per task, full-parameter fine-tuning of Qwen2.5-7B-Instruct, Qwen3-8B, and Llama3.1-8B-Instruct under identical hyperparameters across all methods, averaged over three seeds. At the strictest budget (i2):
| Method |
Backbone |
ACC (%) |
BWT (pp) |
| Sequential SFT |
Qwen2.5-7B |
55.02 |
โ13.93 |
| Vanilla Replay |
Qwen2.5-7B |
62.57 |
โ4.27 |
| SDFT |
Qwen2.5-7B |
61.23 |
โ1.00 |
| OPR-RU |
Qwen2.5-7B |
65.45 |
โ2.29 |
| OPR-SC |
Qwen2.5-7B |
65.11 |
โ2.08 |
| Sequential SFT |
Llama3.1-8B |
50.78 |
โ17.07 |
| OPR-RU |
Llama3.1-8B |
61.89 |
โ2.89 |
OPR-RU cuts i3 by 42โ46% relative to a tuned Vanilla Replay baseline across all three backbones at i4, and by 46โ60% across budgets i5. Notably, OPR-RU's overall accuracy is nearly flat across budgets (range 0.47 pp) while Vanilla Replay swings by 3.51 ppโonce the buffer is on-policy, replay quality dominates replay quantity. At i6, OPR-RU reaches BWT i7, versus i8 for sequential SFT.
SDFT retains a marginal BWT edge (within ~1.5 pp on every backbone) but loses 3โ5 pp of overall accuracy, driven almost entirely by a collapse on MeetingBank (โ23 to โ32 pp relative to Vanilla Replay). Loss-dynamics analysis explains this: on the three tasks requiring large distributional shifts from the base model (MeetingBank summarization, Py150 code completion, 20Minuten German simplification), SDFT's reverse-KL anchor causes its training loss to plateau an order of magnitude above OPR-SC's, while on classification tasks the curves are indistinguishable. The teacher anchor prevents the cross-entropy from descending precisely where it must.
Low-score ablation. Holding prompts, sampler, and code fixed and swapping high-score rollouts for low-score ones degrades BWT from i9 to ฯฮธiโโ0 at ฯฮธiโโ1โworse than Vanilla Replay (ฯฮธiโโ2). This rules out "replay-of-anything" explanations and demonstrates that the active ingredient is the conjunction of on-policy sampling and the high-reward filter, not response quality alone nor on-policy distribution alone.
Scorer comparison. OPR-SC matches OPR-RU in aggregate (65.11 vs. 65.45 ACC), confirming that the on-policy-plus-filter structure, rather than the specific scorer, drives the gains. However, OPR-SC underperforms on NumGLUE-cm, where fluently stated incorrect rollouts pass the confidence filterโthe buffer anchors what the model is already confident in rather than manufacturing new capability.
Limitations and open questions
The authors are explicit about scope. Claims rest on three 7โ8B instruction-tuned backbones and TRACE's canonical English task ordering; the KL-shrinkage account is unvalidated on base checkpoints, models above ~10B parameters, alternative orderings, or non-English streams. OPR-RU depends on task-specific evaluators, which limits applicability where such metrics do not exist; OPR-SC removes this dependence but fails predictably on inputs where the model is confidently wrong. The theoretical bound is one-sidedโit constrains only the replay portion of each batchโand rests on the empirical assumption that rule-based rewards correlate with likelihood. The paper leaves open whether the scorer can be learned end-to-end in place of handcrafted metrics or raw likelihoods.
Conclusion
This paper demonstrates that the on-policy signal known to mitigate catastrophic forgetting can be delivered through the training data source rather than the objective. By filling the cross-task replay buffer with reward-filtered rollouts of the most recent checkpoint, OPR enforces per-sample loss bounds at the previous policy and descends a KL toward it as an algebraic identity, all within an unmodified SFT loop. It matches or exceeds teacher-based distillation on overall accuracy without a teacher forward pass, and a controlled ablation isolates the high-reward filter as decisive. The result reframes loss-side and data-side continual-learning methods as two encodings of the same surrogate constraint, with the data-side route avoiding the acquisition cost that reverse-KL anchoring imposes on large-distribution-shift tasks.