Factual Forgetting in Memory Systems

• Factual Forgetting is the decline of accessible facts due to interference in high-dimensional memory representations rather than simple temporal decay.
• Statistical and geometric models quantify forgetting using power-law retention curves that mirror the dynamics observed in human memory.
• Mitigation strategies like data interleaving, targeted unlearning, and memory-augmenting architectures help preserve factual knowledge in language models.

Factual forgetting refers to the phenomenon whereby concrete facts—such as semantic associations, entity relationships, or domain-specific knowledge—stored in biological or artificial memory systems degrade or become inaccessible over time or after updating. It arises not merely from temporal decay, but predominantly from competitive interference in high-dimensional representational spaces, resource constraints, and the intrinsic trade-offs between stability (retaining old knowledge) and plasticity (acquiring new knowledge). Factual forgetting plays a central role in human memory dynamics, artificial neural systems, LLMs, and machine unlearning, and its mechanistic underpinnings are now substantially clarified by geometric, statistical, and operational frameworks.

1. Geometric and Interference-Based Mechanisms

The geometric theory of factual forgetting posits that memory traces are encoded as vectors in a high-dimensional semantic space. Retrieval operates by proximity (e.g., cosine similarity), and competition among stored traces leads to power-law forgetting. The core retrieval score over time $t$ is formalized as

$$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$

with $\psi=0.5$ mirroring human memory exponents. In the absence of competitors (i.e., no distractor embeddings), time-dependent decay alone yields a negligible forgetting exponent ($b \approx 0.009$), producing near-ceiling retention. Introducing interference, by accumulating $N(t)$ distractors, yields the empirical power law

$R(t) \propto t^{-b}$

where $b=0.460 \pm 0.183$ (95% CI: $[0.354, 0.644]$), matching the human benchmark $b\approx 0.5$. Core results show that interference, rather than decay, is necessary and sufficient to produce the observed forgetting dynamics. Production embedding models, although nominally high-dimensional (e.g., $d=384$–$$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$0), concentrate semantic variance into $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$1 dimensions, moving the system into a regime highly susceptible to angular interference and spontaneous false memory formation (Barman et al., 27 Mar 2026).

2. Statistical and Theoretical Models of Forgetting

Multiple mathematical frameworks formalize forgetting as the outcome of statistical interference, search efficiency, or memory management under capacity constraints. The rearrangement inequality and Poisson statistics yield an explicit retention curve:

$$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$2

where $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$3 is the upper incomplete Gamma function, $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$4 is the number of exposures, and $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$5 is the noise (interference) rate. Superposing terms of different $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$6 and $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$7 captures empirically observed multi-stage forgetting curves (short term, consolidated, and long term), and fitting yields excellent agreement with Ebbinghaus’s data. This framework also reveals the efficiency role of forgetting: as more items are stored, interference rises and optimal memory capacity converges to Miller's “magic number seven,” optimizing the speed–accuracy trade-off (Yu et al., 2018).

A complementary interference-based model uses multidimensional “valence” to describe memory stability. Each new fact wipes out all prior facts it strictly dominates in all $$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$8 dimensions. The resulting analytic retention function is

$$S(q, m; t) = \cos(q(t), m) \cdot (1 + \beta t)^{-\psi}$$9

with base case $\psi=0.5$0. For $\psi=0.5$1, corresponding to empirical word recognition data, the decay fits a power-law exponent close to $\psi=0.5$2 and exhibits stabilization of older facts—a counterintuitive but observed property (Georgiou et al., 2019).

3. Operationalization and Quantification in LLMs

In contemporary LLMs, factual forgetting is measured as the relative drop in accuracy or likelihood assigned to previously correct facts after continued pretraining, post-training, or knowledge editing. For a benchmark $\psi=0.5$3, if $\psi=0.5$4 is the set of items correctly answered pre-update and $\psi=0.5$5 post-update, then

$\psi=0.5$6

These metrics are standard in factual QA benchmarks and adaptation studies (Kunz, 24 Oct 2025, Wallat et al., 2021). Factual forgetting is particularly pronounced in naive single-task continued pretraining and after instruction fine-tuning (IFT), with typical factual loss of 4–12% on broad benchmarks, exceeding 35% in some language adaptation settings (Thede et al., 19 Feb 2026, Kunz, 24 Oct 2025). Preference optimization (e.g., DPO) and data mixture strategies offer partial mitigation by preserving a larger fraction of prior knowledge.

For continual model updates or knowledge editing, the average probability drop and explicit “forget rate” are tracked:

$\psi=0.5$7

In-context knowledge editing (IKE) demonstrates notably reduced forgetting (average forget rate $\psi=0.5$8) relative to direct parameter editing (up to $\psi=0.5$9 forgetting), primarily by avoiding parameter overwrite and by explicit “retain” demonstrations (Zheng et al., 2023).

4. Functional and Adaptive Perspectives

Forgetting is increasingly understood as functional rather than pathological. Rational memory process theories formalize it as continual statistical compression under storage constraints. Incoming facts are first stored as high-resolution summaries, then merged, compressed, or pruned by heuristic worth-to-cost criteria (recency, recurrence, surprisal, task relevance) to optimize for anticipated future use under a hard memory budget:

$b \approx 0.009$0

Forgetting emerges as a “graceful” process, with concrete details lost first and abstracted representations persisting longer—a direct consequence of multi-level, time-scale-aware statistical encoding and utility-driven store management (Cheveigné, 16 Feb 2025). This perspective subsumes both biological and artificial systems, predicting that factual recall decays naturally even in the absence of explicit decay mechanisms.

Cognitively inspired models of in-context LLM reasoning recapitulate exponential-decay forgetting kernels:

$b \approx 0.009$1

with decay rates $b \approx 0.009$2, quantitatively matching human memory’s range ($b \approx 0.009$3). Interventions such as probabilistic memory prompting (PMP) exploit this property for improved deliberative reasoning and extended context utilization (Tran et al., 28 Dec 2025).

5. Mitigation Strategies and Architectural Interventions

The literature reports a spectrum of mitigation approaches. For LLMs, interleaving random or generic data during fine-tuning (REMIX) robustly decreases gradient interference and factual forgetting, outperforming standard replay and baseline strategies (e.g., up to 90.6% factual retention vs. 39.8% for no mixing) (Chen et al., 2024). Preference optimization (DPO) curtails the drift to below $b \approx 0.009$4 while also partially recovering losses induced by instruction fine-tuning (Thede et al., 19 Feb 2026). Circuit-level interpretability further enables surgical “unlearn-then-learn” knowledge editing, localizing updates to critical modules and suppressing catastrophic forgetting of unrelated facts (control accuracy rises from $b \approx 0.009$5 in direct LoRA to $b \approx 0.009$6 with targeted $b \approx 0.009$7 PEFT) (Ngugi, 9 Aug 2025). These techniques reveal that soft, reversible suppression (“soft forgetting”)—rather than destructive erasure—predominantly governs safe, auditable knowledge modulation.

For memory-augmented architectures, latent-retrieval autoencoders (e.g., NextMem) combine compact, reversible memory with robust retrieval and quantization, preserving nearly $b \approx 0.009$8 reconstruction F1 under noise and resisting performance collapse under both quantization and synthetic forgetting pressures (Zhang et al., 26 Feb 2026).

In pretraining, deduplication of the training corpus slows power-law forgetting and enhances factual retention, especially for long-tail facts, by minimizing micro-acquisition dilution. Empirically, duplication accelerates forgetting (decay constants $b \approx 0.009$9 for duplication, $N(t)$0 for paraphrase exposures, OLMo-7B), and larger batch sizes modestly improve robustness (Chang et al., 2024).

6. Future Directions, Open Questions, and Analytical Landscapes

Ongoing research highlights several non-universalities: forgetting curves and stability–plasticity trade-offs are model size and family dependent, with no architectural or data-level fix eliminating the loss of factual knowledge without cost to adaptation (Thede et al., 19 Feb 2026). Catastrophic forgetting remains more prevalent in models lacking preliminary instruction fine-tuning, with SFT+unlearning paradigms yielding smoother and more targeted forgetting (Anna et al., 23 Feb 2026). Open problems include extending continual-memory preservation to streaming (“many-stage”) regimes, refining data mixing ratios by domain and model scale, and formalizing the interaction between popularity/salience and internal knowledge localization (Anna et al., 23 Feb 2026, Chen et al., 2024).

Theoretical challenges remain in unifying the probabilistic, geometric, and process-centric perspectives on forgetting, modeling the dynamics under the constraints imposed by bounded memory, and devising information-theoretic or statistical optimality criteria for memory retention vs. utilization. The ongoing integration of interpretability-guided, cognitively-motivated, and data-driven approaches is defining the frontier of robust factual memory management in both biological and artificial agents.