Evidence Forgetting: Models and Mechanisms

• Evidence Forgetting is the systematic decay or suppression of acquired information, observed in both biological memory and artificial neural systems.
• Quantitative models, including exponential and power-law decay functions, reveal the role of noise, interference, and order effects in memory retention.
• Practical applications range from optimizing training curricula and mitigating catastrophic forgetting in continual learning to reducing privacy risks in data-driven models.

Evidence forgetting encompasses the systematic loss, decay, or suppression of previously acquired information—whether in biological memory, artificial neural systems, machine learning workflows, or LLMs. The phenomenon ranges from the adaptive filtering of irrelevant traces (human memory) to the abrupt loss of prior task competence (catastrophic forgetting in continual learning), and from the fading of individual training examples to measurable reductions in memorized content even in over-parameterized models. Research across neuroscience, psychology, and machine learning has elucidated diverse mechanisms, measurement protocols, mathematical formalisms, and mitigation techniques related to evidence forgetting.

1. Mathematical and Computational Models of Forgetting

Quantitative modeling of forgetting spans analytical models for human memory, formalizations in deep learning, and mechanistic descriptions for LLMs:

• Human forgetting curves: A unified retention probability function based on Poisson arrivals of “noise” and the regularized upper incomplete Gamma function describes the retrieval probability of an item seen $n_0+1$ times over time $t$:

$$R_{n_0}(t) = \frac{\Gamma(n_0+1, k t)}{\Gamma(n_0+1)}$$

Special cases with $n_0=0$ reproduce exponential decay, while mixtures of such forms allow simultaneous modeling of short-term and long-term memory (Yu et al., 2018).

• Power-law forgetting: Competing retroactive-interference models with n-dimensional valence predict that memory retention follows a power-law:

$R_n(t) \approx \Bigl(1+\frac{t}{n}\Bigr)^{-n}$

with empirical evidence globally rejecting exponential forms in favor of power-law fits in both human experiments and recognition-probe data (Georgiou et al., 2019).

• Order-dependent example forgetting: Empirically, the learning speed of an example (mean correctness across epochs) almost perfectly predicts its probability of subsequent forgetting, with Pearson correlations $\rho=0.995$ between early learning and long-term retention across standard vision data (Hacohen et al., 2024).
• Latent-state inference in LLMs: Sequence models such as LLMs can be described by discounted (non-exact) Bayesian filters:

$$p_t(\theta|\mathcal{D}_{1:t}) \propto p(D_t|\theta)\bigl[p_{t-1}(\theta|\mathcal{D}_{1:t-1})\bigr]^\gamma$$

where $\gamma=e^{-\lambda \Delta t}$, and $\lambda$ encodes the exponential decay rate of evidence (Tran et al., 28 Dec 2025).

• Forgetting metrics in continual learning: Task-level forgetting is operationalized as

$F_i = \max_{j \geq i} (A_{j,i}) - A_{N,i}$

or via backward transfer:

$t$0

capturing the decrement in performance on earlier tasks due to later updates (Ororbia et al., 2019, Pandey, 29 Mar 2026).

2. Empirical Evidence and Measurement Protocols

Large-scale experiments have confirmed evidence forgetting across neural architectures, domains, and task protocols:

• Single-task forgetting events: In deep network classifiers, specific examples are repeatedly forgotten (transition from correct to incorrect), with a minority being “unforgettable” (never lost after being first learned). The distribution of forgetting events is strongly bimodal across MNIST, CIFAR-10, and their variants. For example, 31.3% of CIFAR-10 training images are unforgettable (Toneva et al., 2018).
• Task-level catastrophic forgetting: In continual learning setups, standard multi-layer perceptrons trained using backpropagation or regularization (EWC, SI, MAS) lose up to 70 points or more in backward transfer on Split-MNIST and custom streams. Predictive-coding models and local Hebbian learning significantly mitigate this loss (e.g., BWT $t$1 and accuracy near 0.98 with S-NCN) (Ororbia et al., 2019).
• Example-level order effects: When grouping by learning speed, the fraction of “remembered” examples after task switch correlates nearly linearly with early learning, a strong “last-in, first-out” forgetting pattern (Hacohen et al., 2024).
• Pre-training memory in LLMs: Standard metrics such as perplexity mask fact-level forgetting, while entity-focused metrics (insertion accuracy, extraction success) reveal pronounced, slowly-recovering declines after dataset switches. Episodic replay mitigates forgetting cost-effectively (Intensive Focused Stochasticity reduces entity extraction loss at about 5% extra compute) (Liao et al., 2024).
• Membership-inference decay: Adversarial privacy attacks demonstrate that previously memorized examples become harder to extract as further SGD steps accumulate. For instance, in ImageNet, MI precision for injected canaries decays from 100% to ∼65% over ten epochs after their removal; LLMs trained on C4 show exposure returning to baseline within roughly 10,000 steps (Jagielski et al., 2022).

3. Mechanisms and Theoretical Explanations

• Interference suppression: Forgetting in both human and artificial systems can act as an adaptive feature that manages interference. In unified memory models, the rate of “noise” item arrival increases the likelihood that older traces are outranked and discarded, thus reducing retrieval interference and minimizing the expected number of probes required during recall (Yu et al., 2018).
• Retroactive interference: Competing memories with higher valence supplant earlier ones. This mechanism yields both the power-law retention characteristic and age-dependent stabilization: older traces are harder to erase due to their extremal property in at least one dimension of valence (Georgiou et al., 2019).
• Exponential decay in LLMs: Transformer-based LLMs implement evidence forgetting that matches an exponential decay kernel, with the effective decay rate ($t$2) empirically decreasing with model size. This decay aligns numerically with the half-lives observed for human recall across a range of experimental manipulations (Tran et al., 28 Dec 2025).
• Remediating mechanisms: Freezing large fractions of parameters, as in Low-Rank Adaptation (LoRA) or shallow-head fine-tuning, sharply reduces catastrophic forgetting. Mechanistic probes demonstrate that the preservation of a stable feature backbone explains the majority of forgetting suppression, rather than the low-rank adapters per se (Pandey, 29 Mar 2026).
• Contingencies in replay-based mitigation: Sample replay, while widely regarded as a forgetting remedy, may both help and harm depending on the choice and geometry of replayed samples. In high-dimensional regression and classification, adversarial or even random replay can increase forgetting, especially when the new task's nullspace and replay subspaces are not well-aligned (Mahdaviyeh et al., 4 Jun 2025).

4. Metrics, Benchmarks, and Methodologies

A diversity of metrics and experimental methodologies are deployed to quantify and analyze evidence forgetting:

Metric/Protocol	Formalization or Key Insight	Primary Context
Forgetting event count	$t$3	Example-level, single-task
Task-level forgetting $t$4	$t$5	Continual learning
Backward transfer (BWT)	$t$6	Continual learning
Learning speed correlation	$t$7	Forgetting risk prediction
Privacy exposure decay	Exposure(s,f) via log-rank	Privacy/SGD unlearning
Entity extraction/Insertion	$t$8 on entity-containing samples	LLM memory

Novel replay buffer techniques such as the Goldilocks protocol explicitly leverage learning-speed metrics to select examples likely to be forgotten, optimizing limited buffer capacity for maximal retention (Hacohen et al., 2024).

5. Practical Implications and Applications

• Data efficiency and pruning: Forgetting-based importance scores enable pruning of training datasets with negligible loss in test accuracy. On CIFAR-10, removing up to 30% of the least-forgotten examples does not measurably degrade generalization (Toneva et al., 2018).
• Curriculum design and robust training: By identifying forgettable examples (e.g., difficult or noisy outliers), it is possible to refine curricula, focus training on informative support sets, or cleanse mislabeled data.
• Adaptive memory strategies in LLMs: Implementing probabilistic memory prompting (sampling context windows according to exponential decay kernels) allows fine-tuning the trade-off between recency bias and long-range dependency, yielding measurable gains in multi-step reasoning, chain-of-thought arithmetic, and nonstationary streaming tasks (Tran et al., 28 Dec 2025).
• Privacy risk mitigation: Forgetting of early-seen or subsequently removed examples decreases the success of membership-inference and canary extraction attacks, offering passive privacy amplification. Practically, recent examples are the most vulnerable, and continued SGD steps without sensitive data actively diminish risk (Jagielski et al., 2022).
• Continual learning practice: Strict backbone freezing, as in LoRA, nearly eliminates catastrophic forgetting. For instance, sequential fine-tuning on four GLUE tasks with full adaptation yields 19.9% average forgetting, while LoRA ($t$9, query/value) reduces this to 0.6%, a 97% reduction (Pandey, 29 Mar 2026).

6. Controversies, Limitations, and Open Directions

• Sample replay pitfalls: Contrary to conventional wisdom, replay can often be deleterious. Theoretical and empirical studies show that under certain task sequences and replay sample choices, forgetting increases non-monotonically or remains bounded below regardless of buffer size (Mahdaviyeh et al., 4 Jun 2025).
• Metric inadequacy: Common optimization/evaluation metrics such as overall accuracy, perplexity, or memorization fraction can fail to detect substantial forgetting at the fact or entity level. Entity-focused recall and insertion metrics, or targeted privacy attacks, are more reliable for tracking persistent memory loss in both LLMs and discriminative models (Liao et al., 2024, Jagielski et al., 2022).
• Architectural generality: While freezing-based mitigation and replay strategies generalize well within Transformer and convolutional backbones, the universality across less-explored architectures (e.g., graph neural networks, memory-augmented models) remains underexamined.
• Cognitive alignment: Decay rates of evidence forgetting in state-of-the-art LLMs approach those measured in human subjects—a result that both invites deeper mechanistic parallels and raises new questions over optimality and adaptability in artificial agents (Tran et al., 28 Dec 2025).
• Active vs. passive mitigation: Methods such as continual replay, adaptive sample selection, and parameter-efficient adaptation promote retention, but privacy and security guarantees require additional mechanisms (e.g., differential privacy, randomization) for robust protection against adversarial extraction.

7. Broader Impact and Theoretical Synthesis

Evidence forgetting is a central phenomenon uniting diverse strands of neuroscience, cognitive science, and machine learning. Far from being a mere deficiency, forgetting—at the level of items, facts, or internal mechanisms—acts as a critical feature for efficient memory search, interference suppression, robust adaptation, and passive privacy enhancement. Modern unifying theories frame it as an emergent property of rational systems trading off retention against computational and statistical interference, and as a substrate for continual learning and life-long adaptation in both biological and artificial systems (Yu et al., 2018, Georgiou et al., 2019, Tran et al., 28 Dec 2025).

Research continues to explore the interplay between representation geometry, optimization protocol, memory-limited replay, and task structure in shaping the quantitative and qualitative dynamics of forgetting and retention across artificial and biological agents.