Papers
Topics
Authors
Recent
Search
2000 character limit reached

On-Policy Replay for Continual Supervised Fine-Tuning

Published 28 May 2026 in cs.LG | (2605.29495v1)

Abstract: Continual supervised fine-tuning (SFT) is the de facto recipe for adapting LLMs to a stream of downstream tasks, but it suffers from catastrophic forgetting of earlier capabilities. Recent work shows that on-policy signals -- training on the model's own outputs -- reduce forgetting more reliably than off-policy supervision. Existing on-policy methods route this signal through a new training objective (e.g., self-distillation losses with a teacher copy), inheriting an extra forward pass, schedule sensitivity, and stylistic drift from the teacher.We instead route the on-policy signal through the training data source. Our method, On-Policy Replay (OPR), rolls out the most recent checkpoint on a small budget of historical prompts, filters the generations by a task reward, and replays the surviving (prompt, model response) pairs as ordinary SFT examples. There is no teacher, no auxiliary loss, and no on-the-fly distillation. Across three 7--8B instruction-tuned backbones (Qwen2.5-7B-Instruct, Qwen3-8B, Llama3.1-8B-Instruct) on the TRACE continual-learning benchmark, OPR consistently reduces forgetting; on the sharpest stress test (Qwen2.5-7B-Instruct, Sequential SFT BWT -13.93), OPR lifts BWT to -0.65 at a 10% replay budget and to -2.29 at a 1% budget -- a 46% reduction in |BWT| over a tuned Vanilla Replay baseline, with 42--46% reductions observed across all three backbones. We give a KL-shrinkage interpretation that places OPR and prior on-policy distillation methods on a single axis, and we present a counterintuitive finding that explains why Vanilla Replay is already a strong baseline: low-score replay is uniformly worse than Vanilla Replay, demonstrating that the active ingredient in OPR is the on-policy distribution, not the response quality alone.Our code is available at https://github.com/Yancey2024/OnPolicyReplay.

Summary

  • The paper introduces On-Policy Replay, which fills cross-task buffers with high-scoring rollouts from the previous model and reduces forgetting without modifying the standard supervised fine-tuning objective.
  • Experiments on TRACE show that OPR-RU cuts backward-transfer degradation by 42โ€“46% versus tuned vanilla replay at a 1% replay budget, while reaching โˆ’0.65 percentage points at a 10% budget.
  • A low-score ablation confirms that both on-policy sampling and reward filtering are essential, while the label-free OPR-SC variant performs comparably overall but can preserve confidently incorrect outputs.

Motivation and problem statement

Sequential supervised fine-tuning (SFT) remains the default mechanism for adapting instruction-tuned LLMs to a stream of downstream tasks, yet it produces severe catastrophic forgetting: on the TRACE benchmark, plain sequential SFT degrades backward transfer (BWT) by 10.85โ€“17.07 percentage points across three backbones. Recent analyses converge on a mechanistic account in which forgetting at each update is upper-bounded by the per-update KL divergence between consecutive policies evaluated on historical prompts. Because training on samples drawn from the previous policy ฯ€ฮธtโˆ’1\pi_{\theta_{t-1}} shrinks this KL by construction, on-policy data is the natural lever against forgetting.

Existing methods deliver this on-policy signal through the objective: SDFT, OPSD, and GKD add an explicit per-token KL term against a frozen teacher copy of the model, paying a teacher forward pass per gradient step and introducing schedule-sensitive hyperparameters; RazorSFT rewrites current-task targets with verifier-passing self-samples. This paper takes a different position: the constraint is on the data distribution the loss sees, not on the loss itself. The cross-task replay bufferโ€”the one pipeline object that already carries information across task boundariesโ€”is repurposed as the delivery vehicle for the on-policy signal, leaving the SFT training loop entirely unchanged.

Method

On-Policy Replay (OPR) operates as follows. At the end of stage ii, the checkpoint ฯ€ฮธi\pi_{\theta_i} rolls out one response per historical prompt under low-temperature decoding (temperature 0.1 via vLLM). A scorer ranks the resulting (prompt, response) pairs, and the top-bb entriesโ€”with b=ฯโ€‰ni+1b = \rho\, n_{i+1} allocated evenly across prior tasksโ€”form the replay buffer Bi+1B_{i+1}, which is mixed with next-task data under standard token-level cross-entropy. There is no teacher, no auxiliary loss, and no target rewrite.

Two scorers are considered. OPR-RU uses each task's own evaluation metric (exact match, ROUGE-L F1, edit similarity, or SARI), aligning buffer construction with the evaluation criterion by design. OPR-SC scores rollouts by the model's own length-normalized log-probabilityโ€”a free by-product of sampling that removes any dependence on ground-truth labels, making the method applicable when reliable task annotations are unavailable.

Theoretical account

The paper grounds OPR in a KL-shrinkage analysis. Building on the bound from prior work, the loss change on historical tasks satisfies, to leading order,

ฮ”iโ€…โ€Šโ‰คโ€…โ€ŠCโ‹…Exโˆผphist[KL(ฯ€ฮธtโˆฅฯ€ฮธtโˆ’1)]+o(โˆฅฮธtโˆ’ฮธtโˆ’1โˆฅ2).\Delta_i \;\le\; C \cdot \mathbb{E}_{x \sim p_{\text{hist}}}\bigl[\mathrm{KL}(\pi_{\theta_t} \| \pi_{\theta_{t-1}})\bigr] + o(\|\theta_t - \theta_{t-1}\|^2).

For the self-confidence scorer, top-bb selection is exactly a per-sample upper bound on the loss at ฮธtโˆ’1\theta_{t-1}: every buffer item lies in a high-likelihood region of the previous checkpoint. Under smoothness assumptions, this yields a batch-gradient norm bound, and the identity Eq[โˆ‡ฮธโ„“]=โˆ‡ฮธโ€‰KL(qโˆฅฯ€ฮธ)\mathbb{E}_q[\nabla_\theta \ell] = \nabla_\theta\,\mathrm{KL}(q \| \pi_\theta) shows that SGD on the replay portion descends ii0, pulling the policy toward a high-reward restriction of its previous self. Combining these gives

ii1

so the reward filter directly controls the leading-order forgetting term. Two caveats are stated plainly: the derivation assumes the rule-based scorer correlates positively with log-likelihood (an empirical assumption, not an identity), and the bound covers only the replay portion of each batchโ€”the new-task gradient remains unconstrained, making the mechanism intrinsically one-sided.

This analysis makes a falsifying prediction: replacing high-score rollouts with low-score ones from the same pool preserves on-policy sampling but breaks the loss bound, and should degrade BWT below even vanilla off-policy replay. The experiments confirm both parts.

Experimental results

The evaluation uses TRACE's canonical eight-task sequence with 5,000 examples per task, full-parameter fine-tuning of Qwen2.5-7B-Instruct, Qwen3-8B, and Llama3.1-8B-Instruct under identical hyperparameters across all methods, averaged over three seeds. At the strictest budget (ii2):

Method Backbone ACC (%) BWT (pp)
Sequential SFT Qwen2.5-7B 55.02 โˆ’13.93
Vanilla Replay Qwen2.5-7B 62.57 โˆ’4.27
SDFT Qwen2.5-7B 61.23 โˆ’1.00
OPR-RU Qwen2.5-7B 65.45 โˆ’2.29
OPR-SC Qwen2.5-7B 65.11 โˆ’2.08
Sequential SFT Llama3.1-8B 50.78 โˆ’17.07
OPR-RU Llama3.1-8B 61.89 โˆ’2.89

OPR-RU cuts ii3 by 42โ€“46% relative to a tuned Vanilla Replay baseline across all three backbones at ii4, and by 46โ€“60% across budgets ii5. Notably, OPR-RU's overall accuracy is nearly flat across budgets (range 0.47 pp) while Vanilla Replay swings by 3.51 ppโ€”once the buffer is on-policy, replay quality dominates replay quantity. At ii6, OPR-RU reaches BWT ii7, versus ii8 for sequential SFT.

SDFT retains a marginal BWT edge (within ~1.5 pp on every backbone) but loses 3โ€“5 pp of overall accuracy, driven almost entirely by a collapse on MeetingBank (โˆ’23 to โˆ’32 pp relative to Vanilla Replay). Loss-dynamics analysis explains this: on the three tasks requiring large distributional shifts from the base model (MeetingBank summarization, Py150 code completion, 20Minuten German simplification), SDFT's reverse-KL anchor causes its training loss to plateau an order of magnitude above OPR-SC's, while on classification tasks the curves are indistinguishable. The teacher anchor prevents the cross-entropy from descending precisely where it must.

Low-score ablation. Holding prompts, sampler, and code fixed and swapping high-score rollouts for low-score ones degrades BWT from ii9 to ฯ€ฮธi\pi_{\theta_i}0 at ฯ€ฮธi\pi_{\theta_i}1โ€”worse than Vanilla Replay (ฯ€ฮธi\pi_{\theta_i}2). This rules out "replay-of-anything" explanations and demonstrates that the active ingredient is the conjunction of on-policy sampling and the high-reward filter, not response quality alone nor on-policy distribution alone.

Scorer comparison. OPR-SC matches OPR-RU in aggregate (65.11 vs. 65.45 ACC), confirming that the on-policy-plus-filter structure, rather than the specific scorer, drives the gains. However, OPR-SC underperforms on NumGLUE-cm, where fluently stated incorrect rollouts pass the confidence filterโ€”the buffer anchors what the model is already confident in rather than manufacturing new capability.

Limitations and open questions

The authors are explicit about scope. Claims rest on three 7โ€“8B instruction-tuned backbones and TRACE's canonical English task ordering; the KL-shrinkage account is unvalidated on base checkpoints, models above ~10B parameters, alternative orderings, or non-English streams. OPR-RU depends on task-specific evaluators, which limits applicability where such metrics do not exist; OPR-SC removes this dependence but fails predictably on inputs where the model is confidently wrong. The theoretical bound is one-sidedโ€”it constrains only the replay portion of each batchโ€”and rests on the empirical assumption that rule-based rewards correlate with likelihood. The paper leaves open whether the scorer can be learned end-to-end in place of handcrafted metrics or raw likelihoods.

Conclusion

This paper demonstrates that the on-policy signal known to mitigate catastrophic forgetting can be delivered through the training data source rather than the objective. By filling the cross-task replay buffer with reward-filtered rollouts of the most recent checkpoint, OPR enforces per-sample loss bounds at the previous policy and descends a KL toward it as an algebraic identity, all within an unmodified SFT loop. It matches or exceeds teacher-based distillation on overall accuracy without a teacher forward pass, and a controlled ablation isolates the high-reward filter as decisive. The result reframes loss-side and data-side continual-learning methods as two encodings of the same surrogate constraint, with the data-side route avoiding the acquisition cost that reverse-KL anchoring imposes on large-distribution-shift tasks.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.