A Quantitative Characterization of Forgetting in Post-Training

Abstract: Continual post-training of generative models is widely used, yet a principled understanding of when and why forgetting occurs remains limited. We develop theoretical results under a two-mode mixture abstraction (representing old and new tasks), proposed by Chen et al. (2025) (arXiv:2510.18874), and formalize forgetting in two forms: (i) mass forgetting, where the old mixture weight collapses to zero, and (ii) old-component drift, where an already-correct old component shifts during training. For equal-covariance Gaussian modes, we prove that forward-KL objectives trained on data from the new distribution drive the old weight to zero, while reverse-KL objectives converge to the true target (thereby avoiding mass forgetting) and perturb the old mean only through overlap-gated misassignment probabilities controlled by the Bhattacharyya coefficient, yielding drift that decays exponentially with mode separation and a locally well-conditioned geometry with exponential convergence. We further quantify how replay interacts with these objectives. For forward-KL, replay must modify the training distribution to change the population optimum; for reverse-KL, replay leaves the population objective unchanged but prevents finite-batch old-mode starvation through bounded importance weighting. Finally, we analyze three recently proposed near-on-policy post-training methods, SDFT (arxiv:2601.19897), TTT-Discover (arxiv:2601.16175), and OAPL (arxiv:2602.19362), via the same lens and derive explicit conditions under which each retains old mass and exhibits overlap-controlled drift. Overall, our results show that forgetting can by precisely quantified based on the interaction between divergence direction, geometric behavioral overlap, sampling regime, and the visibility of past behavior during training.

• The paper rigorously quantifies catastrophic forgetting by distinguishing mass forgetting from old-component drift in generative models.
• The study employs forward- and reverse-KL objectives to reveal how replay impacts the data distribution and prevents old-mode collapse.
• Application to methods like SDFT and TTT-Discover shows that the choice of objective critically influences retention and controls parameter drift.

A Quantitative Characterization of Forgetting in Post-Training

Introduction and Motivation

The study rigorously investigates catastrophic forgetting in post-training phases of generative models using a mixture model abstraction. Contending that prior work lacks a unified, quantitative theoretical account for when and why forgetting occurs, the authors formalize two distinct but often conflated phenomena: mass forgetting (collapse of prior-mode mixture weights) and old-component drift (parameter shift within the retained component). The analytic lens is grounded in the dichotomy between forward and reverse Kullback–Leibler (KL) divergence objectives and their population-level optima, especially in the context of continued post-training with data from distributions representing new behavior.

Formal Model and Definitions

Modeling the learning target as a two-mode mixture $$p_\alpha(y) = \alpha\,p_{\mathrm{o}}(y) + (1-\alpha)p_{\mathrm{n}}(y)$$ for $\alpha \in [0,1]$, the learner constructs a model mixture $$q_\beta(y) = \beta\,q_{\mathrm{o}}(y) + (1-\beta)q_{\mathrm{n}}(y)$$ with explicit mixture coefficient $\beta$ and component distributions. The central definitions are:

• Mass Forgetting: Occurs if the population-optimal $\beta^\star = 0$ under the training objective, with $q_\beta(y)$ assigning zero probability to old behavior even when $q_{\mathrm{o}}(y) = p_{\mathrm{o}}(y)$.
• Old-Component Drift: Refers to the deviation of the old-component parameters from $p_{\mathrm{o}}$, even when mixture mass is retained.

The minimality of the setup (only two well-modeled Gaussian modes with shared covariance) ensures the separability of loss-induced forgetting from representational limitations.

Forward-KL versus Reverse-KL: Analytical Results

Forward-KL (SFT, New-Data-Only):

• The forward-KL objective is equivalent to maximum likelihood with respect to the data distribution, often only the new data ($p = p_{\mathrm{n}}$).
• The authors prove that $$L_{\mathrm{SFT}}(\beta) = \mathrm{KL}(p_{\mathrm{n}}\,\|\,q_\beta)$$ is strictly increasing in $\beta$; its unique minimizer is $\beta^\star = 0$, regardless of component accuracy. Therefore, the model loses all old behavior. Gradient flow in logit space ($\beta = \sigma(\phi)$) confirms exponential decay in $\beta$, with update dynamics governed by exponentially small overlap-dependent terms for well-separated modes.

Reverse-KL (KL-Regularized RL or On-Policy):

• The reverse-KL population optimum, $\min_{\theta}\mathrm{KL}(q_\theta\,\|\,p_\alpha)$, is uniquely aligned—$$(\beta^\star, m^\star) = (\alpha, \mu_{\mathrm{n}})$$ achieves zero KL. At this stationary point, component drift is exponentially suppressed in the Mahalanobis separation $\delta$, with all update signals on $m_{\mathrm{o}}$ (the old mean) gated by overlap (Bhattacharyya coefficient).
• The local geometry near the optimum is strongly convex with explicit lower bounds, ensuring exponential convergence of simple gradient flow methods.

The analysis is extended to $f$-divergences, finite-mixture, and log-concave component families, where analogous forgetting and retention regimes are observed.

Replay and Data Exposure: Disparate Roles under Different Objectives

Significant theoretical distinctions are drawn regarding the effect of replay:

• For forward-KL, only replay that modifies the data distribution (numerator) alters the population optimum and can prevent $\beta$-collapse. Mixing a $\lambda$ fraction of old samples in the data moves the optimum to $\beta^\star = \lambda$. By contrast, denominator (model-side) replay serves only as a hard floor, not driven by optimization.
• For reverse-KL, replay via mixture of old samples in batch construction does not alter the population objective but prevents “old-mode starvation” in finite-batch settings. Bounded importance weights ensure low-variance, unbiased gradient estimation and persistent sampling from the old mode.

Application to Recent Near-On-Policy Post-Training Methods

The authors apply the mixture analysis to recently proposed algorithms:

• SDFT (Self-Distillation Fine-Tuning): Behaves like a reverse-KL iteration towards a demonstration-anchored teacher with EMA smoothing. Prevents mass forgetting if the anchor exerts persistent influence, and drift remains bounded (and exponentially small with mode separation).
• TTT-Discover: Uses an entropic, reward-tilted objective with a KL anchor. If the anchor coefficient is insufficient, old-mode collapse can still occur. However, when mass is retained, old mean drift is again overlap-gated and exponentially small.
• OAPL: Constructs improvement targets via an exponential tilt of a frozen reference policy. Only modes present in this reference can be reweighted; cross-mode influence is localized and suppressed by small geometric overlap.

The quantitative bounds and necessary retention conditions for each algorithm are articulated with explicit dependence on mixture parameters, separation $\delta$, and overlap statistics.

Theoretical and Practical Implications

This analysis establishes that catastrophic forgetting in post-training is not merely a limitation of representation but, often, a direct population-level consequence of the chosen loss function and data regime. Off-policy objectives (e.g., forward-KL on new-only data) intrinsically induce forgetting by mass collapse, while on-policy or KL-regularized RL objectives can robustly preserve old behaviors—with any drift controlled by the exponentially decaying tail interaction of components.

Replay serves fundamentally different purposes depending on the underlying objective. For forward-KL, it is an essential ingredient for achieving any form of population-level retention; for reverse-KL and related on-policy methods, stochastic replay is a practical mechanism for addressing pathologies arising from poor coverage in finite-sample optimization.

By quantifying exactly when forgetting is unavoidable and when it is negligible (or summable), this work provides an analytic platform for designing and evaluating continual learning methods in generative modeling scenarios.

Conclusion

The paper delivers a mathematically precise account of forgetting dynamics in post-training, dissecting the roles of objective choice, mixture geometry, and sampling regime. Forward-KL–based SFT exhibits unavoidable mass forgetting on new-only data; reverse-KL–based methods are naturally aligned to retention, with locality and exponential convergence. Replay’s function is context-dependent: it is requisite for retention in forward-KL, and an optimization stabilizer for reverse-KL. Analysis of modern post-training algorithms via this lens clarifies which mechanisms guarantee retention and under what quantitative thresholds. These insights elucidate which methodological attributes are necessary for robustness to catastrophic forgetting, and suggest principled directions for algorithmic development in future work on continual post-training of large generative models.

---

Reference:

"A Quantitative Characterization of Forgetting in Post-Training" (2603.12163)