---
title: Evidence Forgetting Rate
url: https://www.emergentmind.com/topics/evidence-forgetting-rate
type: topic
---

# Evidence Forgetting Rate

Evidence Forgetting Rate is a quantitatively defined parameter that captures the rate at which previously acquired, observed, or encoded information loses influence, salience, or retrievability within a given system. The term arises across disciplines—from cognitive psychology, where it describes human memory decay, to machine learning, where it governs data retention in streams, determines neural-network recall over time, and structures the dynamics of continual and large language model (LLM) learning. Theoretical and empirical approaches rigorously model forgetting rates using exponential, power-law, and convolutional decay forms, and operationalize measurement through both accuracy and privacy-motivated audit metrics.

## 1. Mathematical Formalizations of Forgetting Rate

Forgetting rate is instantiated mathematically by specifying a functional decay law for retention or influence of evidence over time or sequence steps.

- **Exponential Decay:** In both human and machine memory frameworks, the retention of an item, $R(t)$, often follows $R(t) = e^{-\lambda t}$, where $\lambda$ is the evidence forgetting rate [2601.09726][1812.05668].
- **Power-Law Decay:** Empirical and classic cognitive studies frequently fit $R(t) = A t^{-\beta}$, where $\beta$ is the decay exponent. This form captures the typical “fast-then-plateau” characteristic of human memory [2506.12034][1812.05668].
- **Convolutional Models:** For repeated learning or spaced rehearsal, forgetting is modeled as the superposition (convolution) of impulse response kernels, each with their own decay rate, $a_2$, yielding $y(t) = \sum_{n=1}^N a_1 e^{-a_2 (t-T_n)} + a_3$; instantaneous forgetting rate is $r(t) = \frac{a_2 a_1 \sum_n e^{-a_2 (t-T_n)}}{\sum_n (a_1 e^{-a_2 (t-T_n)} + a_3)}$ [1901.08114].
- **Markovian Absorption (Collective Forgetting):** In social or cultural-semantic memory, forgetting is parameterized as the mean absorption rate $\lambda = \frac{q (p + r)}{r + q}$, where $p,q,r$ are rates of transfer/decay between communicative memory, cultural memory, and oblivion [2008.06592].
- **Resource-Constrained Bayesian Updating:** LLM in-context memory can be expressed as a discounted Bayesian update,
  $$ p_t(\theta|D_{1:t}) \propto p(D_t|\theta) [p_{t-1}(\theta|D_{1:t-1})]^\gamma, $$
  where $\gamma = e^{-\lambda}$ (discount factor), or equivalent token-weighting $w_i \propto e^{-\lambda \Delta_i}$ [2601.09726].

## 2. Measurement in Machines: Empirical and Theoretical Metrics

Operationalizing evidence forgetting rate requires extracting decay parameters from observed trajectories, audit tests, or performance metrics.

- **Recall Probability:** In neural networks, recall is measured as the probability that the network’s hidden state aligns with a target prototype, tracking this probability over time/epochs since last exposure [2506.12034].
- **Attack Success Rate:** In privacy studies, the metric is the success probability of a membership-inference or canary-extraction attack after removal of a sensitive datum; the forgetting rate is the drop in attack success per step [2207.00099].
- **Cross-Entropy Loss Relative to Base Model:** Following fine-tuning, forgetting can be quantified as $L_f := \mathbb{E}_x H(p_M(\cdot|x), p_{M'}(\cdot|x))$, the cross-entropy from pre-trained to fine-tuned model prediction distributions [2401.05605].
- **Task Error or Loss Curves:** In continual learning frameworks, the mean squared error on prior tasks (or classification error) as a function of new task iterations directly measures rate of forgetting [2506.04377].
- **Information Attention Decay:** In collective memory, normalized citation attention ($S(t)$) fit to an exponentially-mixed model, with $\lambda$ estimated via ODE parameter regression [2008.06592].

## 3. Psychological, Sociotechnical, and Algorithmic Drivers

Forgetting rate exhibits deep connections to cognitive constraints, cultural information load, task drift, and learning-systems architecture.

- **Information Volume Pressure:** In communities (e.g., physics, invention), the forgetting rate $\lambda$ of works doubles over multi-decade intervals as collective output volume rises, validating the “forgetting as annulment” effect [2008.06592].
- **Stability–Plasticity Trade-Off:** In LLM and cognitive modeling, rate $\lambda$ governs the balance between retaining old evidence (low $\lambda$), and rapid adaptation to new data (high $\lambda$) [2601.09726].
- **Concept Drift Adaptation:** In online learning with non-stationary distributions, optimal performance requires tuning forgetting rate proportional to detected drift magnitude—a central tenet of the “sweet path” hypothesis [1801.09354].
- **Replay and Interference:** Intensive replay or review reduces effective forgetting rates and can produce nonlinear (non-monotonic) effects, with small or misaligned replay sometimes increasing global forgetting via geometric task interference [2506.04377][1901.08114].
- **Randomness vs. Determinism:** Stochasticity in training induces $1/k$ decay (“forgetting law”) in audit attack success; deterministic optimization can preclude forgetting entirely for specific memorized points [2207.00099].

## 4. Empirical Scaling Laws and Quantitative Values

Several studies provide fitted values for forgetting-rate parameters and scaling relationships.

- **LLMs:** Empirical optimal forgetting rates $\lambda$ span roughly $0.02$ to $0.5$ per token, corresponding to “half-lives” from dozens to hundreds of tokens; optimal Bayesian discount factors $\gamma^* \in [0.6, 0.9]$ [2601.09726].
- **Power-Law Decay in MLPs:** Single-task MLP classification without review fits a power curve $R(t) \approx 0.32 t^{-0.47}$ ($R^2\approx0.96$), with half-life $t_{1/2}\approx 2$ epochs [2506.12034].
- **Collective Memory:** APS papers (1950–1999) show $\lambda$ increases from $\approx0.06\,\mathrm{yr}^{-1}$ to $0.13\,\mathrm{yr}^{-1}$; USPTO patent forgetting rate rises from $0.12\,\mathrm{yr}^{-1}$ to $0.28\,\mathrm{yr}^{-1}$ [2008.06592].
- **Fine-Tuning Scaling Laws:** Forgetting loss obeys $L_f(P,N) = -c_{f,ft}c_{ft} \left[\left(a_{ft}/P\right)^{\alpha_{ft}}+\left(b_{ft}/N\right)^{\beta_{ft}}\right]^{\rho} + (s_{f,ft} - c_{f,ft}s_{ft})$, with fitted exponents and strong $R^2$ on benchmark LLMs [2401.05605].
- **Ebbinghaus-style Human Curves:** Human and MLP memory decay exponents $\beta \simeq 0.47$ consistently emerge, with overlearning and spaced repetition flattening the decay [2506.12034][1812.05668].

## 5. Practical Implications and Interventions

Strategic manipulation of the forgetting rate has significant consequences for learning efficiency, memory safety, and system adaptability.

- **Review Scheduling:** Spaced or intensive review resets, implemented according to the monitored forgetting-curve, robustly flatten decay and increase recall even past initial capacity—mirroring human overlearning [1901.08114][2506.12034].
- **Replay in LLM Pre-Training:** Very low-cost focused stochastic replay can reduce entity forgetting rate by $\sim$40–50% (relative on extraction metrics), yielding persistent improvements even in vanilla models and boosting zero-shot accuracy by $\sim0.8$ points [2410.17018].
- **Optimal Matching to Drift:** Online and incremental learners perform best when forgetting rate is adaptively matched to measured drift rates, following the sweet path: higher drift $\rightarrow$ higher forgetting, lower model variance [1801.09354].
- **Cultural Selectivity:** Knowledge communities adapt to rising forgetting rates by increasing selectivity at cultural-memory transfer ($r/(p+r)$), preferentially “buffering” the most valuable artifacts into durable memory reservoirs [2008.06592].
- **Privacy and Unlearning:** Natural forgetting induced by SGD randomness offers passive privacy amplification; privacy-preserving mechanisms benefit by focusing on “fresh” rather than long-trained data [2207.00099].

## 6. Limitations, Nuances, and Open Questions

Forgetting rate is not universally beneficial or monotonic in its effects; system structure and process details produce complex dependencies.

- **Non-monotonicity in Replay:** Sample replay can worsen forgetting, with effect size and direction tightly linked to choice of samples and mutual geometry of task subspaces; monotonic benefit emerges only with sufficiently large or well-aligned replay [2506.04377].
- **Metric Sensitivity and Misestimation:** Popular metrics like PPL and binary memorization scores fail to detect substantial fact-level forgetting, urging the adoption of entity-focused metrics in LLM assessments [2410.17018].
- **Interference-Limited Capacity:** In human models, the noise-driven retention boundary naturally recovers Miller's “seven plus or minus two” item short-term memory constraint [1812.05668].
- **Non-convexity and Determinism:** In specific non-convex systems or with fully deterministic training, memorized evidence may be indefinitely persistent regardless of elapsed steps [2207.00099].

## 7. Cross-Domain Convergence and Theoretical Synthesis

Across domains, evidence forgetting rate emerges as a central parameter structuring adaptive memory processes, whether in human cognition, artificial networks, knowledge communities, or large-scale LLMs. The trade-off between stability and adaptability, operationalized via exponential or power-law decay, is reflected in both the quantitative dynamics of learning systems and their practical regulatory mechanisms for memory management. Decades of work in both human and artificial domains converge on the finding that forgetting is not merely a deficit but a principled mechanism for efficiency, adaptability, and safe, scalable information integration.

Source: https://www.emergentmind.com/topics/evidence-forgetting-rate