Predictive Forgetting in Forecasting & Continual Learning

• Predictive forgetting is defined as tailoring memory retention based on its impact on future predictions, employing adaptive temporal weighting and confidence-based diagnostics.
• It is formalized through techniques such as weighted empirical risk minimization, predictive self-consistency, and context-conditional suppression across varied AI domains.
• This approach offers actionable insights for mitigating catastrophic forgetting and improving predictive performance in forecasting, continual learning, and language model unlearning.

Searching arXiv for recent and directly relevant papers on predictive forgetting and related formulations. Predictive forgetting denotes a family of mechanisms, metrics, and theories in which forgetting is defined, induced, or measured through its relation to prediction rather than treated solely as generic memory loss. Across recent work, the term is used in several technically distinct senses: adaptive temporal weighting for forecasting under distribution shift, calibrated uncertainty-based assessment of catastrophic forgetting, context-conditional suppression of otherwise available knowledge, and algorithm-agnostic predictive self-consistency over future experience (Bennett et al., 2022, Pitsiorlas et al., 15 May 2025, Takashiro et al., 2024, Sanati et al., 6 Nov 2025). This suggests that predictive forgetting is best understood not as a single method, but as a unifying perspective in which the central question is what information should be retained, attenuated, or monitored in order to preserve or improve future predictive performance.

1. Conceptual scope

The recent literature uses predictive forgetting in at least four non-equivalent ways. In time-series forecasting, forgetting is a learned weighting policy over historical observations, introduced to improve one-step-ahead prediction when the conditional law $Y_t \mid X \sim \pi_t(\cdot \mid X)$ drifts over time (Bennett et al., 2022). In continual learning, forgetting can be assessed through predictive uncertainty, as in the Conformal Prediction Confidence Factor (CPCF), where confidence-set inflation on old tasks is treated as a calibrated signature of catastrophic forgetting (Pitsiorlas et al., 15 May 2025). In LLMs, forgetting can be context-conditional at inference time rather than permanent, so that a model suppresses a target answer when prompted with explicit unlearning tags while preserving unrelated responses (Takashiro et al., 2024). In a more general theoretical formulation, forgetting is defined as a lack of self-consistency in the learner’s predictive distribution over future experiences, with the resulting loss of predictive information measured by a divergence $\Gamma_k(t)$ (Sanati et al., 6 Nov 2025).

Formulation	Core predictive object	Representative paper
Adaptive temporal weighting	Next-step risk under drift	(Bennett et al., 2022)
Confidence-based diagnosis	Prediction set size on past tasks	(Pitsiorlas et al., 15 May 2025)
Context-conditional suppression	Output behavior given prompt context	(Takashiro et al., 2024)
Predictive self-consistency	Future-experience distribution	(Sanati et al., 6 Nov 2025)
Replay-risk monitoring	Class-wise or sample-wise forgetting risk	(Serra et al., 2024, Tamajo et al., 14 May 2026)

The survey literature does not use the exact phrase “predictive forgetting,” but it places closely related ideas under selective forgetting, active forgetting, hybrid forgetting, and feedback-driven adaptive memory control (Sha et al., 2024). That broader framing is important because it separates beneficial, controlled forgetting from catastrophic forgetting. In this view, forgetting is adaptive when it selectively removes obsolete, noisy, misleading, privacy-sensitive, or low-utility information while preserving predictive structure that matters for future tasks or future data.

2. Formalizations

A first formalization appears in distribution-shifted forecasting as weighted empirical risk minimization. The next-step risk is written as

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$

and is estimated by the weighted empirical risk

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$

Here forgetting is the map from temporal lag to sample weight, $\alpha(i;\eta)=\alpha_i$, with exponential decay $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$ and mixed decay $$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$ as two explicit parameterizations (Bennett et al., 2022). The governing trade-off is between adaptivity to distribution shift and effective sample size.

A second formalization treats forgetting as predictive self-consistency failure. In the general stochastic interaction framework of "Forgetting is Everywhere" (Sanati et al., 6 Nov 2025), the learner state $Z_t$ is characterized operationally by the induced future distribution $q(H^{t+1:\infty}\mid Z_t,H_{0:t})$. One-step non-forgetting requires that updating on learner-consistent targets leave this induced future unchanged in expectation: $$q(H^{t+1:\infty}\mid Z_{t-1},H_{0:t-1}) = \mathbb{E}_{Y_t,X_t,Z_t} \left[ q(H^{t+1:\infty}\mid Z_t,H_{0:t}) \right].$$ The resulting $\Gamma_k(t)$0-step propensity to forget is

$\Gamma_k(t)$1

In this account, forgetting is neither primarily a parameter-level nor an accuracy-level concept; it is a divergence between predictive distributions over future experience.

A third formalization is diagnostic and confidence-based. CPCF defines forgetting after task $\Gamma_k(t)$2 as the average conformal prediction set size on old-task samples: $\Gamma_k(t)$3 Prediction sets are formed by a calibrated cumulative-softmax rule using a split-conformal threshold $\Gamma_k(t)$4, and larger sets indicate greater uncertainty on previous tasks (Pitsiorlas et al., 15 May 2025). The underlying claim is that catastrophic forgetting is visible not only through accuracy degradation but through confidence erosion.

A fourth formalization is conditional output suppression. In "Answer When Needed, Forget When Not: LLMs Pretend to Forget via In-Context Knowledge Unlearning" (Takashiro et al., 2024), a pre-trained autoregressive model is trained so that, given a target $\Gamma_k(t)$5 and a query $\Gamma_k(t)$6, the response $\Gamma_k(t)$7 becomes "forgot" when $\Gamma_k(t)$8 is related to $\Gamma_k(t)$9, while remaining normal when $$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$0 is unrelated. The training loss is

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$1

with

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$2

This formalizes forgetting as a context-sensitive readout policy rather than guaranteed deletion from internal representations.

3. Algorithmic mechanisms

In forecasting under drift, the most explicit predictive-forgetting optimizer is differentiable bi-level learning of the forgetting mechanism. The lower-level objective fits predictor parameters with lag-dependent weights,

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$3

while the upper-level objective evaluates the fitted model on a recent validation block,

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$4

The forgetting parameters are then learned by

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$5

with implicit differentiation used to obtain $$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$6 and update $$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$7 by gradient descent (Bennett et al., 2022). The main methodological claim is that gradient-based optimization makes richer forgetting profiles practical.

Replay-based approaches instantiate predictive forgetting differently. In continual predictive learning from videos, CPL combines a mixture world model, predictive experience replay, and non-parametric task inference (Chen et al., 2022). Prior trajectories are regenerated by sampling an initial frame and rolling it forward with the world model, rather than storing full videos. "Predictive Experience Replay for Continual Visual Control and Forecasting" extends this idea to Dreamer-style visual control, using an initial-frame generator, frozen old world model, and frozen old policy to generate replayed trajectories of observations, actions, and rewards (Zhang et al., 2023). In both papers, replay is not generic rehearsal but rehearsal through prediction.

Streaming replay can also be analyzed as a stochastic approximation to a joint objective. "Mitigating Catastrophic Forgetting in Streaming Generative and Predictive Learning via Stateful Replay" writes the ideal risk as

$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$8

and compares sequential fine-tuning against replay, whose expected phase-$$R_t(\theta) = \mathbb{E}_{Y_t \sim \pi_t(\cdot \mid X_t)} \left[ L\!\left(f(X_t;\theta),Y_t\right) \,\middle|\, \{(X_\tau,Y_\tau)\}_{\tau=1}^{t-1} \right],$$9 gradient is

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$0

The sign of $$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$1 determines whether current updates help or harm old predictive performance, so gradient conflict becomes a predictor of forgetting severity (Du, 22 Nov 2025). A crucial complication is that replay is not universally beneficial: in an over-parameterized continual linear regression setting, replay can increase forgetting in worst-case and distributional settings, and forgetting can be non-monotonic with respect to the number of replay samples (Mahdaviyeh et al., 4 Jun 2025). This makes replay-span geometry, task-subspace alignment, and principal angles between task null spaces plausible predictive signals of when rehearsal will help or hurt.

Uncertainty-aware memory management introduces another algorithmic direction. In online class-incremental learning, several uncertainty scores are used to rank candidate memory samples—Least Confidence, Smallest Margin, Ratio of Confidence, Entropy, Rainbow Memory agreement score, and Bregman Information (BI)—under top-$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$2, bottom-$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$3, and step-size storage rules (Serra et al., 2024). The proposed BI estimate is

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$4

and is interpreted as a logit-space generalized variance induced by the negative log-likelihood. The main empirical conclusion is that low-uncertainty, representative samples are more effective replay anchors than high-uncertainty, boundary-like samples.

4. Domain-specific realizations

Time-series forecasting is the most direct predictive-forgetting application. In synthetic settings—FixedRegime, RandomWalk, RandomRegime, and Stat—series length is $$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$5, with

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$6

and 192 Monte Carlo runs per setting (Bennett et al., 2022). Real-data experiments use a factor model for 50 NYSE stocks and a volatility-forecasting task for 15 ETFs with expanding-window cross-validation. The learned mixed-decay rule, GradMixedDecay, is best on 4 of 6 datasets and near-best on the other two, supporting the claim that predictive forgetting can be learned rather than hand-tuned.

In continual video prediction and model-based RL, predictive forgetting is the loss of old environment dynamics. CPL shows that standard predictors trained through non-stationary streams begin to generate old tasks with last-task dynamics, whereas predictive experience replay preserves prior motion patterns and scene structure (Chen et al., 2022). In Dreamer-style continual visual control, the same problem appears in latent rollouts, reward prediction, and value estimation; the proposed mixture world model and predictive replay substantially improve retention on DeepMind Control and Meta-World, while also alleviating forgetting of spatiotemporal dynamics on RoboNet and KTH (Zhang et al., 2023).

Knowledge tracing treats predictive forgetting as a latent-state transition that directly controls next-response prediction. In the Concept-driven Personalized Forgetting model, the student’s interaction history is

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$7

and the knowledge state evolves under personalized learning gain and personalized forgetting gate. The model introduces a student-specific ability representation

$$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$8

a prerequisite matrix $$\hat{R}_t(\theta) = \sum_{\tau=1}^{t-1} \alpha_{|t-1-\tau|} L\!\left(f(X_\tau;\theta),Y_\tau\right).$$9, a forgetting weight

$\alpha(i;\eta)=\alpha_i$0

and the state update

$\alpha(i;\eta)=\alpha_i$1

Here forgetting is neither post hoc nor separate from prediction; it is embedded in the recurrent update used to predict whether a student will answer the next exercise correctly (Wang et al., 2024).

LLMs exhibit two additional variants. In pre-training, forgetting appears as declining entity-related factual memory despite continuing improvement in aggregate perplexity. The paper on pre-training forgetting introduces entity-centered metrics $\alpha(i;\eta)=\alpha_i$2 and $\alpha(i;\eta)=\alpha_i$3, showing that PPL and $\alpha(i;\eta)=\alpha_i$4 can mask forgetting of relatively frequent entities in early corpora that become relatively infrequent later (Liao et al., 2024). In contrast, in-context knowledge unlearning studies conditional suppression at inference time. Its main empirical claim is that the method reaches up to 95% forget accuracy while retaining 80% of unrelated knowledge, but logit-lens analysis indicates that the correct factual answer often remains decodable in middle layers and is only vetoed at the final layer, hence the characterization that “LLMs pretend to forget” (Takashiro et al., 2024).

Human-like forgetting-curve work shifts the focus from mechanistic erasure to retention monitoring. Recall probability is defined from prototype similarity as

$\alpha(i;\eta)=\alpha_i$5

and is used to trigger adaptive review when retention falls below $\alpha(i;\eta)=\alpha_i$6 of the original prototype score (Kline, 22 May 2025). This is not a long-horizon forecast of future forgetting, but it is a predictive-risk monitor in the operational sense that low current retention identifies knowledge that should be rehearsed before performance collapses.

5. Measurement and diagnostics

Several papers recast forgetting as a measurable predictive quantity rather than a purely retrospective accuracy drop. CPCF is the clearest calibrated example. Using split-conformal calibration, the model computes conformal scores

$\alpha(i;\eta)=\alpha_i$7

forms a threshold

$\alpha(i;\eta)=\alpha_i$8

and then tracks average prediction set size on old-task test samples (Pitsiorlas et al., 15 May 2025). Across MNIST, CIFAR-10, FashionMNIST, and KMNIST, CPCF shows moderate to strong distance correlation with the standard previous-task accuracy metric, with the strongest reported value $\alpha(i;\eta)=\alpha_i$9 on MNIST with EWC at calibration ratio $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$0 and $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$1.

Pre-training work on entity memory introduces more task-specific forgetting diagnostics. $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$2 counts top-1 next-token correctness, but the paper argues that even entity-filtered versions remain dominated by easy contexts. It therefore introduces $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$3, an entity-conditioned continuation accuracy over 32 generated tokens, and $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$4, which tests whether the gold entity appears in the generated continuation when the entity itself is removed from the prompt (Liao et al., 2024). These metrics reveal strong degradation at corpus transitions that PPL fails to show.

Predictive uncertainty can also be used prospectively. In online continual learning with replay, the choice between top-$\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$5 uncertain samples and bottom-$\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$6 certain samples becomes a question about which examples are easiest to forget or easiest to remember. The reported result is that bottom-$\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$7 selection is consistently stronger than top-$\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$8, and BI is often the strongest uncertainty measure, particularly for lowering last forgetting $\alpha(\tau;\eta)=\exp(-\eta_1 \tau)$9 on CIFAR-10, CIFAR10-LT, CIFAR100-LT, and BloodCell (Serra et al., 2024). This turns predictive forgetting into a memory-population problem.

The most explicit within-step forecasting of future forgetting appears in rehearsal-based class-incremental learning. The paper on imbalanced forgetting defines per-class end-of-step forgetting

$$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$0

and constructs three last-layer coefficients: the Self-Induced Bias Interference Coefficient,

$$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$1

the Cross-Class Bias Interference Coefficient,

$$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$2

and the New-Dataset Interference Coefficient,

$$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$3

These coefficients predict how previously learned classes will rank in terms of forgetting at the end of the current step, with SIC emerging as the strongest single predictor and the joint linear model improving rank prediction further (Tamajo et al., 14 May 2026). This is one of the clearest demonstrations that predictive forgetting can mean forecasting class-wise forgetting risk before end-of-step evaluation.

6. Debates, limitations, and open directions

A central debate is whether predictive forgetting refers to forecasting future forgetting, measuring current forgetting through predictive quantities, or intentionally modifying predictive behavior. The literature supports all three uses. CPCF is primarily diagnostic rather than future-predictive (Pitsiorlas et al., 15 May 2025). Prototype-based recall scores are predictive only in an operational monitoring sense, not as explicit trajectory forecasts (Kline, 22 May 2025). In-context knowledge unlearning changes predictions conditionally at test time, but the mechanistic evidence indicates late-layer suppression rather than genuine internal deletion (Takashiro et al., 2024). This suggests that the term covers both predictive diagnostics and prediction-shaping mechanisms.

Another major issue is whether forgetting should be regarded as a pathology to eliminate or a regulated process that can improve generalization. "Fortuitous Forgetting in Connectionist Networks" argues that partial forgetting can be favorable to learning through a forget-and-relearn cycle, provided the forgetting operator reduces training accuracy without erasing all task information and thereby disproportionately removes undesirable information (Zhou et al., 2022). The 2026 consolidation theory pushes this further: predictive forgetting is defined as reducing $$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$4 while approximately preserving $$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$5, with consolidation acting as outcome-conditioned compression that improves information-theoretic generalization bounds on stored representations (Fountas et al., 5 Mar 2026). Together with the survey account of active and hybrid forgetting, this suggests a normative reading in which beneficial forgetting removes outcome-irrelevant detail while retaining predictive sufficiency (Sha et al., 2024).

Theoretical unification remains incomplete. The task-agnostic theory of predictive self-consistency is broad, but practical estimators of $$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$6 require Monte Carlo rollout approximations and assume that learner state induces a coherent predictive distribution over future interactions (Sanati et al., 6 Nov 2025). In forecasting under drift, differentiable forgetting relies on smooth lower-level optimization and can face unstable Hessian inversion in deep models (Bennett et al., 2022). In continual learning, uncertainty-based methods often require retained calibration or rehearsal data, which may violate strict no-memory assumptions (Pitsiorlas et al., 15 May 2025, Serra et al., 2024). Replay is simultaneously one of the most effective and one of the most fragile tools: stateful replay is a strong baseline on heterogeneous streaming tasks (Du, 22 Nov 2025), yet replay can provably increase forgetting when replayed samples induce unfavorable task-subspace geometry (Mahdaviyeh et al., 4 Jun 2025).

A plausible synthesis is that predictive forgetting is becoming a general design principle: forget what is not predictively useful, preserve what is needed for stable future inference, and monitor the learner through predictive quantities rather than retrospective correctness alone. The literature supports this view, but it also indicates that the crucial unresolved problems are identifying the right predictive target $$\alpha(\tau;\eta)=\exp(-\eta_1\tau-\eta_2\tau^2-\eta_3\log(\tau+1))$$7, estimating future utility without full access to past data, distinguishing suppression from real unlearning, and turning strong diagnostics into closed-loop mitigation policies (Liao et al., 2024, Sha et al., 2024, Tamajo et al., 14 May 2026).