---
title: Forgetting Measure (FM) Overview
url: https://www.emergentmind.com/topics/forgetting-measure-fm
type: topic
---

# Forgetting Measure (FM) Overview

A forgetting measure (FM) quantifies the degree to which an agent, model, or information system loses previously acquired information as a result of learning, updating, or passage of time. It formalizes the notion of information loss, either at the granular (sample-wise, token-wise, class-wise) or structural (theoretical, inferential, model-theoretic) level. Across domains—language models, federated learning, knowledge representation, human memory—forgetting measures serve both as diagnostic metrics and as benchmarks guiding the development of retention and rehearsal strategies.

## 1. Mathematical Formulations of Forgetting Measures

Forgetting measures span a wide array of formal definitions, each grounded in the specifics of the domain and the questions of interest.

### Sample-wise Forgetting in Language Model Post-Training

The framework of "Mapping Post-Training Forgetting in Language Models at Scale" defines FM as the fraction of items for which a model transitions from correct (pre-training) to incorrect (post-training):

\[
F = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\left[a_i^{pre} = 1 \wedge a_i^{post} = 0\right]
\]

A companion metric, backward transfer (BT), tracks 0→1 transitions. Chance-adjusted variants correct for random guessing by subtracting analytically derived baselines [2510.17776]:

\[
F_{true} = \max(F - F_{chance}, 0)
\]
\[
BT_{true} = \max(BT - BT_{chance}, 0)
\]

### Federated Learning: Round-wise Class-Granular FM

Flashback introduces a per-round loss-based FM capturing negative changes in per-class accuracy:

\[
\mathcal{F}_t = -\frac{1}{C}\sum_{c=1}^C \min(0,\, A^c_t - A^c_{t-1})
\]

This strictly aggregates losses, ignoring gains, ensuring that net knowledge loss is visible even when overall accuracy improves in some classes [2402.05558].

### Privacy-oriented Example-wise FM in Supervised ML

In the context of privacy attacks, FM is mechanism-agnostic and attack-calibrated:

\[
FM_z(k) := S_{\mathcal{A}}(\theta^{+}_{t_0 + k},\; \theta^{-}_{t_0 + k};\; z)
\]

Here $S_{\mathcal{A}}$ is the adversarial success rate distinguishing model inclusion of $z$ after $k$ further training steps; $z$ is (A, α, k)-forgotten if $FM_z(k) \leq \alpha$ [2207.00099].

### Information-structural FM via Model Counting and Probability

In probabilistic logic and knowledge representation, Doherty and Szałas define three interlocked FM loss functions quantifying inferential strength drop following variable forgetting:

\[
\begin{aligned}
loss_{m}^{NC}(Σ,\bar p) &= \#\left[F^{NC}(Σ;\bar p)\right] - \#\left[Σ\right] \\
loss_{m}^{SC}(Σ,\bar p) &= \#\left[Σ\right] - \#\left[F^{SC}(Σ;\bar p)\right] \\
loss_{m}^{T}(Σ,\bar p)  &= \#\left[F^{NC}(Σ;\bar p)\right] - \#\left[F^{SC}(Σ;\bar p)\right]
\end{aligned}
\]

Their probabilistic analogues replace counts with measures $\mu$ [2404.02454].

### Human Memory: Fractional Dynamics and Power-law FM

In chunk-memory models, forgetting is parameterized by the tail exponent $d$ in

\[
F(t) \sim t^{-d}
\]

The FM here is precisely $d$, also expressible as $FM = 1 - (\text{order of Caputo derivative})$ [1402.4058].

### Retroactive Interference Models

Here, $FM(t) = 1 - R_n(t)$, where $R_n(t)$ is the retention function computed analytically, and the exponent $k$ of the power law is fit from empirical memory data [1907.08946].

### Long-context LM Memorization: Forgetting Curve Gap

The FM for memorization length is operationalized as

\[
F(k) = R_{copy}(k) - R_{LM}(k)
\]

where $R_{copy}$ gives token-wise copy accuracy and $R_{LM}$ the baseline LM accuracy under unrelated contexts. $L_{fine}$ and $L_{coarse}$ are derived memory lengths [2410.04727].

## 2. Domains of Application

Forgetting measures have been developed and deployed in several key areas:

- **Language models**: quantifying knowledge loss and backward transfer during post-training, surcharge (RL, SFT), and model merging [2510.17776, 2410.17018, 2410.04727].
- **Federated learning**: monitoring loss of information in highly heterogeneous data aggregation across rounds and classes [2402.05558].
- **Privacy and data-removal**: empirically bounding remaining attack surface after data is passively "forgotten" in large models [2207.00099].
- **Knowledge representation**: comparing various logical forgetting operators by inferential strength loss [2404.02454].
- **Human and animal memory**: fitting recognition curves and characterizing age-stabilization via power-law decay exponents [1402.4058, 1907.08946].
- **Semantic desktops**: continuous relevance estimation via exponentially decaying Memory Buoyancy [1811.12177].

## 3. Theoretical Insights and Foundations

Forgetting is not a defect but an adaptive, information-theoretic process. The measure $\Gamma_k(t)$ in "Forgetting is Everywhere" quantifies divergence between baseline induced futures and simulated post-update mixtures:

\[
\Gamma_k(t) = D\left(
q(H^{t+k:\infty} \mid Z_{t-1}, H_{0:t-1}) \, \| \, q_k^*(H^{t+k:\infty} \mid Z_{t-1}, H_{0:t-1})
\right)
\]

Here, full self-consistency ($\Gamma_k=0$) is achieved only by ideal Bayesian learners; practical algorithms generally violate it. Moderate forgetting is not universally harmful—it often correlates with accelerated adaptation in nonstationary or class-incremental settings [2511.04666].

FM in memory models exposes the independence between learning and forgetting exponents, supporting dissociation and flexible tuning of retention versus learning speed [1402.4058]. Retroactive interference models explain age-dependent stabilization quantitatively [1907.08946].

Model-counting FM establishes that inferential loss is additive for disjoint theories and monotonic in the forgotten variable set, giving formal guarantees for reasoning system modularity and robustness [2404.02454].

## 4. Practical Computation and Benchmarking

Forgetting measures typically require fine-grained bookkeeping:

- **Sample-wise FM**: comparison of correctness states pre- and post-update; counting transitions.
- **Class-wise FM**: computation of negative deltas per class accuracy; aggregation over rounds or stages.
- **Privacy FM**: parallel training runs with and without probe examples (canaries); measurement of attack success rates such as membership inference or canary exposure.
- **Information-structural FM**: model counting or probability assignment via logic-program translation (ProbLog), quantifier elimination, and explicit counting/valuation queries [2404.02454].
- **Power-law FM**: fitting exponents to the empirical decay or retention curve, via log-log regression or closed-form maximum-likelihood estimation [1402.4058, 1907.08946].
- **Semantic Desktop MB**: event-triggered update loop with exponentially decaying "buoyancy" plus contextual fusion [1811.12177].
- **Long-context LM FM**: iterative measurement of copy vs. LM accuracy at increasing context lengths, automated sample extraction [2410.04727].

## 5. Empirical Observations Across Learning Settings

Multiple studies confirm key behaviors:

- **Low-to-moderate forgetting and backward transfer** are typical in large-scale language model post-training; larger model scales consistently mitigate these effects [2510.17776].
- **Roundwise FM** in federated learning reveals that knowledge acquired by clients is readily lost during aggregation, especially under data heterogeneity; techniques such as dynamic distillation (Flashback) markedly reduce FM and accelerate convergence [2402.05558].
- **Privacy FM** shows that deterministic training leaves all injected probes vulnerable indefinitely, while stochastic SGD enables gradual forgetting; examples injected early in training are forgotten faster [2207.00099].
- **Model-counting FM** supports rapid, automated comparison of logic-reduction policies with formal additivity and monotonicity [2404.02454].
- **Power-law FM exponents** in human and animal memory settle robustly around that observed in recognition data (approx. 0.8 for n=5-dimensional valence models) [1907.08946].
- **Semantic Desktop MB** scores decay as expected, triggering graduated forgetting actions; local/global/group MB layers respect stability under context switches [1811.12177].
- **Long-context LM FM curves** reveal plateau, steep decay, and amnesia phases, with fine memory length under 1–4K tokens for transformers and RNN/SSM architectures dropping rapidly below transformer baselines [2410.04727].

## 6. Research Impact and Future Directions

Forgetting measures enable controlled, systematic evaluation of memory loss, offering actionable diagnostics and tuning criteria:

- In language modeling, FM highlights not only what is forgotten but which transfer (new gains) occurs; straightforward reporting of FM and BT avoids conflation typical of accuracy averages [2510.17776].
- FM frameworks are vital for privacy compliance, allowing empirical confirmation of data removal and attack resilience [2207.00099].
- Knowledge representation leverages FM for theory abstraction, rule-base pruning, and constraint manipulation, providing a unified loss-calculation interface [2404.02454].
- Federated and continual learning benefit from real-time FM diagnosis and targeted rehearsal or distillation protocols [2402.05558, 2511.04666].
- Future work aims to integrate forgetting penalties within training objectives, enforce retention bursts via synthetic rehearsal, leverage external retrieval systems to offset in-weight knowledge loss, and extend forgetting measures to more complex, relational fact and semantic memory domains [2510.17776, 2410.17018].

## 7. Comparative Summary Table

| Domain/Metric                        | FM Definition/Formula                                                        | Key Use                         |
|--------------------------------------|-----------------------------------------------------------------------------|---------------------------------|
| LM Post-Training [2510.17776]        | $F = \text{frac}(1\to 0 \text{ transitions}),\,F_{true}=F-F_{chance}$        | Knowledge loss/BT diagnostics   |
| Federated Learning [2402.05558]      | $\mathcal{F}_t = -\frac{1}{C}\sum_c \min(0,\, A^c_t-A^c_{t-1})$             | Roundwise loss tracking         |
| Privacy [2207.00099]                 | $FM_z(k)=S_{\mathcal{A}}(\theta^+,\theta^-; z)$                             | Data removal, attack decay      |
| Knowledge Represent. [2404.02454]    | $loss_m^{NC/SC/T};\,loss_p$ (model count/probability loss)                  | Theory inferential loss         |
| Human memory [1402.4058,1907.08946]  | $F(t) \sim t^{-d}$, FM=$d$, FM$(t)=1-R_n(t)$                                | Recognition, stabilization      |
| Semantic Desktop [1811.12177]        | $MB^{decay}(t)$, $MB_r(t)$ (exponential+activation bump)                     | Info relevance, auto-forgetting |
| Long-context LM [2410.04727]         | $F(k) = R_{copy}(k)-R_{LM}(k)$, $L_{fine},L_{coarse}$                       | Memory length quantification    |

Forgetting measures—by focusing on granular, interpretive, domain-adjusted quantification—have become central instruments in the analysis and control of information retention in both artificial and biological learning systems.

Source: https://www.emergentmind.com/topics/forgetting-measure-fm