Forgetting-Aware Pruning Metric

• The paper introduces FAPM as a method that integrates forgetting dynamics into pruning, effectively retaining important parameters while reducing catastrophic forgetting.
• FAPM is defined through diverse formulations, from decoder KV-cache adjustments to multi-agent consensus, by using temporal decay and relative importance scores.
• Empirical evaluations demonstrate improved retention accuracy and reduced memory footprint, underscoring FAPM’s practical benefits for model efficiency and unlearning.

Forgetting-Aware Pruning Metric (FAPM) denotes a class of pruning criteria that explicitly model forgetting when deciding which tokens, memories, parameters, channels, or task-vector updates to retain or remove. In the literature, the term appears both as an explicit method name for catastrophic-forgetting mitigation in LLM fine-tuning and as a broader abstraction covering time-decayed attention accumulation, consensus-weighted memory deletion, retain-versus-forget importance ratios, and topology-aware sparse-model unlearning. Taken together, these formulations suggest that FAPM is best understood as a family of metrics that couple pruning decisions to some notion of recency, obsolescence, retained utility, or forgetting risk rather than as a single canonical equation (Jo et al., 2024, Huang et al., 10 Sep 2025).

1. Conceptual scope and common mathematical structure

Across domains, FAPM is used to solve a shared optimization problem: remove state or parameters while preserving the behavior that should remain. In decoder KV-cache pruning, the state is a token’s cached keys and values. In multi-agent memory systems, it is a stored memory item. In machine unlearning, it is a layer, block, or parameter group whose contribution to deleted data should be attenuated. In sparse-model unlearning, it extends to the mask itself, because the pruned topology is treated as a data-dependent object that may also need to forget deleted data (Jo et al., 2024, Bach, 19 Jun 2025, Xiao et al., 24 Jul 2025).

One general formulation appears as

$$\text{FAPM}_t(i) = \sum_{\tau=1}^{t} \gamma(t,\tau) \, g\big(\text{attn}_\tau(i)\big),$$

where $\gamma(t,\tau)$ is a decay or forgetting kernel and $g(\cdot)$ aggregates a relevance signal. In this view, a forgetting-aware metric is not merely an importance score; it is an importance score with explicit temporal discounting or comparative weighting against a retained reference (Jo et al., 2024).

Other instantiations make the same structure more explicit. In multi-agent pruning, the per-agent score is

$$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$

combining temporal decay and semantic relevance. In selective forgetting for agent memory, the corresponding retained-importance score is

$$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$

with pruning priority induced by low importance or, equivalently, by negating that score (Bach, 19 Jun 2025, Gu et al., 22 Apr 2026).

In machine unlearning, the same idea appears as a retain-versus-forget comparison. DFMU defines forget-set and complete-set importance scores and uses their ratio to damp blocks that are disproportionately important for deleted classes, while FaLW measures per-sample forgetting state by comparing predicted probability to a classwise unseen-data distribution. This suggests a recurring FAPM design principle: pruning should be guided by a score that is not only large or small in absolute terms, but large or small relative to what should be retained (U et al., 24 Jun 2026, Yu et al., 26 Jan 2026).

2. Decoder-only transformer pruning

In decoder-only transformers, FAPM emerged from the observation that raw Accumulative Attention Score (A2S or AAS) is biased by causal masking. For token $k$ at generation step $n$,

$A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$

so early tokens accumulate attention over more steps than later tokens. Because softmax outputs are non-negative, old tokens systematically obtain larger A2S values simply by existing longer, which makes token-importance comparisons unfair across token ages. A2SF addresses this by introducing a forgetting factor: $$A^h_{n,k} = \sum_{q=1}^{n} \alpha^{\,n-q} \, S^h_{q,k}, \quad 0 < \alpha < 1,$$ or recursively,

$A^h_{n,k} = \alpha \, A^h_{n-1,k} + S^h_{n,k}.$

Recent attention is weighted by $\gamma(t,\tau)$0, while old attention is exponentially suppressed (Jo et al., 2024).

This construction was proposed for KV-cache pruning in autoregressive decoders, where the cache grows linearly with sequence length and can dominate memory and bandwidth. The paper frames A2SF as an exponentially weighted moving sum of attention and argues that it yields fairness among different ages of tokens because each token’s score reflects a finite effective history of length $\gamma(t,\tau)$1, regardless of absolute age. The practical pruning rule is head-wise: maintain $\gamma(t,\tau)$2, retain the top $\gamma(t,\tau)$3 tokens for cache ratio $\gamma(t,\tau)$4, and evict tokens with the smallest scores. The method uses only “selective cache” and no additional local window cache, because exponential decay already emphasizes recency (Jo et al., 2024).

Empirically, A2SF was evaluated on LLaMA-2-7B, LLaMA-7B, OPT-6.7B, and OPT-2.7B over OpenBookQA, Winogrande, PiQA, COPA, MathQA, ARC-easy, and ARC-challenge in 1-shot and 0-shot settings. For LLaMA-2-7B at cache ratio 0.2, 1-shot average accuracy was 0.458 for H2O and 0.536 for A2SF, while 0-shot average accuracy was 0.399 for H2O and 0.450 for A2SF, corresponding to improvements of approximately $\gamma(t,\tau)$5 absolute and $\gamma(t,\tau)$6 absolute. Mask-similarity analysis against an “Ideal” pruning mask reported average cosine similarity of about 0.318 for Local Attention, about 0.967 for H2O, about 0.991 for A2SF with $\gamma(t,\tau)$7, and about 0.989 for A2SF with $\gamma(t,\tau)$8. Best accuracy was typically observed for $\gamma(t,\tau)$9 on many datasets at cache ratio 0.3, although MathQA could benefit from higher $g(\cdot)$0 (Jo et al., 2024).

3. Multi-agent and agent-memory formulations

In multi-agent systems, FAPM is operationalized as a joint function of temporal decay, semantic relevance, and consensus. The Co-Forgetting Protocol defines per-scale temporal decay

$g(\cdot)$1

and multi-scale decay

$g(\cdot)$2

Each agent then computes a combined score

$g(\cdot)$3

where $g(\cdot)$4 is a DistilBERT-based relevance score, and votes forget if $g(\cdot)$5, otherwise keep. Forgetting is elevated from a local score to a system decision through the weighted forgetting score

$g(\cdot)$6

and quorum

$g(\cdot)$7

combined with PBFT conditions $g(\cdot)$8, at least $g(\cdot)$9 identical PREPARE votes, and at least $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$0 identical COMMIT votes. Experimental evaluations in a simulated four-agent environment reported a 52% reduction in memory footprint over 500 epochs, 88% voting accuracy against human-annotated benchmarks, a 92% PBFT consensus success rate under simulated Byzantine conditions, and an 82% cache hit rate (Bach, 19 Jun 2025).

For single-agent or vector-database memory systems, FSFM provides a closely related but more explicitly score-centric formulation. It combines Content Quality Assessment, Business Value Evaluation, Temporal Relevance Scoring, and Security Risk Classification into

$$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$1

with

$$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$2

in the experimental design and $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$3 for dangerous data, $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$4 for sensitive data, and $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$5 otherwise. Pruning occurs under a 70% capacity cap by sorting ascending in importance score and deleting the bottom segment; the paper reports an empirical best of 10% batch pruning per cycle. FSFM frames forgetting as passive decay-based, active deletion-based, safety-triggered, and adaptive reinforcement-based, and reports access efficiency $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$6, content quality $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$7 signal-to-noise ratio, and 100% elimination of security risks (Gu et al., 22 Apr 2026).

These systems expand the scope of FAPM beyond resource pruning alone. A plausible implication is that in memory-centric agents, forgetting awareness must encompass not only obsolescence and utility but also coordination and safety, so the pruning score becomes inherently multi-objective.

4. Machine unlearning, retain-forget comparison, and sparse topology

In machine unlearning, FAPM is often built by contrasting importance for deleted data with importance for retained or overall data. DFMU is class-wise and computes a layer-wise importance score

$$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$8

from one forward and backward pass. Applied to the forget set $$\text{FAPM}_i(m) := C_i(m) = \omega_D D_m(t) + \omega_R R_i(m),$$9 and complete set $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$0, these yield $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$1 and $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$2, which are converted into a scaling factor

$$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$3

The resulting rescaling rule,

$$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$4

implements knowledge-preserving pruning without retraining. On CIFAR-100 and CIFAR-20 with a ViT classifier, DFMU is reported to achieve 40% more retain-accuracy with just 13% of data samples and averages 88% faster processing time for forgetting a given class; the best trade-off in the ablation was observed at $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$5, $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$6 (U et al., 24 Jun 2026).

FaLW addresses long-tailed forget sets by defining a per-sample forgetting state

$$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$7

where $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$8 and $$\text{Importance Score}_i = \alpha \times \text{CQA}_i + \beta \times \text{BVE}_i + \gamma \times \text{TRS}_i + \delta \times \text{SRC}_i,$$9 are classwise statistics estimated from unseen-data proxies. The forgetting-aware weight is

$k$0

with balance factor

$k$1

This produces instance-wise dynamic loss reweighting: under-forgotten samples are up-weighted, over-forgotten samples are down-weighted, and tail classes receive stronger correction. Extensive experiments report lower Avg. Gap to retraining than several approximate unlearning baselines across CIFAR-100, CIFAR-10, and Tiny-ImageNet, while the balance factor improves tail-class forgetting behavior at the cost of minor degradation in head-class forgetting accuracy under strong skew (Yu et al., 26 Jan 2026).

Sparse-model unlearning introduces a further shift: the mask itself is treated as something that can remember deleted data. The “un-pruning” paper reports that with unstructured pruning using LTH on ResNet-18 at 60% sparsity and only 5% data deleted, there is only around a 58% overlap of pruned indexes across two sparse models trained on $k$2 and $k$3. It therefore defines un-pruning as updating both weights and topology so that the new mask $k$4 becomes distinguishable from the original mask $k$5 while $k$6 remains indistinguishable from retraining + repruning on $k$7. For evaluation, it introduces topology-aware metrics such as IoM,

$k$8

UoM,

$k$9

and IoU,

$n$0

together with a KL divergence over masked parameter distributions. The paper also argues that MIA accuracy is unreliable in this setting and shows that ImageNet un-pruning with WoodFisher can reach IoU 84.22%, IoM 85.06%, TA 57.42%, and UA 55.42% at 19.03% sparsity for ResNet-18, close to retraining + repruning (Xiao et al., 24 Jul 2025).

5. Explicit FAPM for catastrophic forgetting and related adaptive settings

The paper that explicitly names the Forgetting-Aware Pruning Metric proposes it for catastrophic forgetting in LLM fine-tuning. Let $n$1 be pre-trained weights, $n$2 fine-tuned weights, and $n$3 the task vector. The central claim is that catastrophic forgetting is tied not just to $n$4, but to the relative change $n$5. For a linear layer $n$6, FAPM defines the score matrix

$n$7

where $n$8 rescales the penalty term to the same numerical order as the magnitude term. This score rewards downstream-task-important updates while penalizing updates whose relative deviation from pre-trained parameters is large. The method is post-hoc, requires no modification to training, model architecture, or auxiliary data, and prunes the task vector rather than the pre-trained model itself. Across eight datasets covering natural language inference, General Q&A, Medical Q&A, Math Q&A, reading comprehension, and cloze tests, the reported result is that FAPM limits catastrophic forgetting to just $n$9 while maintaining 99.67% accuracy on downstream tasks; 90% sparsity in the task vector was identified as a strong trade-off point in the full fine-tuning setting (Huang et al., 10 Sep 2025).

Related settings supply forgetting signals even when they do not directly define a pruning metric. In federated learning, Flashback introduces a round-wise, per-class forgetting metric

$A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$0

which measures only negative class-wise accuracy changes between rounds. The paper argues that forgetting occurs in both client-local updates and server-side aggregation and proposes dynamic distillation with class-dependent weights based on label counts to reduce both forms. Although Flashback is not itself a pruning method, its formulation suggests a natural route to forgetting-aware pruning in federated settings: parameters important for classes with large recent $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$1 should be treated as stability-critical (Aljahdali et al., 2024).

6. Assessment, misconceptions, and recurring limitations

A recurring theme in the literature is that standard evaluation signals can obscure forgetting. In pre-training, perplexity and the memorization fraction $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$2 were found to potentially mask true forgetting because they are biased toward easy-to-remember elements. To better detect entity-memory retention, the pre-training study introduces $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$3, a token-level exact-match rate on the 32-token continuation following an entity,

$A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$4

and $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$5, a binary metric indicating whether the correct entity string appears in a generated continuation,

$A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$6

The paper reports that $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$7 and $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$8 show a more noticeable decline and harder rebound than traditional metrics, and that even under identical-distribution continued pre-training, entity-focused forgetting curves still decline significantly over time (Liao et al., 2024).

A second misconception concerns the universality of the metric. The reviewed work does not support a single invariant FAPM valid across domains. In transformers, the central issue is age fairness under causal masking. In multi-agent memory, it is semantic relevance, temporal decay, and consensus. In unlearning, it is comparative retain-versus-forget saliency. In sparse models, it is alignment of masks with retraining + repruning. This suggests that “forgetting awareness” names a constraint on what the pruning score must encode, not a unique functional form.

The trade-offs are also domain-specific. A2SF notes that higher $A^{l,h}_{n,k} = \sum_{q=k}^{n} S^{l,h}_{q,k},$9 preserves longer history and can help on MathQA, while too large a history generally reduces fairness across token ages. Co-Forgetting reports that PBFT adds communication overhead and that threshold choices $$A^h_{n,k} = \sum_{q=1}^{n} \alpha^{\,n-q} \, S^h_{q,k}, \quad 0 < \alpha < 1,$$0, $$A^h_{n,k} = \sum_{q=1}^{n} \alpha^{\,n-q} \, S^h_{q,k}, \quad 0 < \alpha < 1,$$1, and $$A^h_{n,k} = \sum_{q=1}^{n} \alpha^{\,n-q} \, S^h_{q,k}, \quad 0 < \alpha < 1,$$2 control aggressiveness of forgetting. FaLW observes that its balance factor improves tail classes at the cost of minor degradation in head forgetting accuracy under high skew. Un-pruning shows that topology recovery becomes harder at high sparsity and that ViT can exhibit much lower IoU than ResNet even when TA and UA remain close to retraining + repruning (Jo et al., 2024, Bach, 19 Jun 2025, Yu et al., 26 Jan 2026, Xiao et al., 24 Jul 2025).

Within that diversity, a stable encyclopedic characterization is possible. FAPM refers to pruning metrics that are explicitly calibrated to forgetting dynamics: they discount stale evidence, compare forget-specific and retain-specific importance, encode semantic or security obsolescence, penalize relative deviation from pre-trained knowledge, or measure whether sparse topology itself has forgotten deleted data. The field’s central methodological question is therefore not whether to use forgetting in pruning, but which notion of forgetting is operationally correct for the system under study.