---
title: Selective Forgetting-Aware Optimization (SFAO)
url: https://www.emergentmind.com/topics/selective-forgetting-aware-optimization-sfao
type: topic
---

# Selective Forgetting-Aware Optimization (SFAO)

Selective Forgetting-Aware Optimization (SFAO) denotes a family of optimization formulations in which forgetting is treated as an explicit design objective rather than an incidental side effect of adaptation. The term is used directly for a cosine-similarity–gated continual-learning optimizer that “regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability” [2603.26671]. Closely related formulations appear in machine unlearning, lifelong learning, agent memory management, federated unlearning, and nonlinear calibration, where the governing problem is to remove designated information while preserving retained utility, respecting memory or communication budgets, or satisfying safety and privacy constraints [2509.22483][2601.18642][2605.22589][2511.14980]. This suggests a broader umbrella concept: optimization procedures that are selectively aware of what must be forgotten, what must be retained, and which constraints dominate that trade-off.

## 1. Historical emergence and conceptual scope

Early selective-forgetting work in deep networks already had the essential SFAO structure. “Eternal Sunshine of the Spotless Net” formalized post-hoc scrubbing as an optimization over retained-task loss plus a KL term measuring how close the scrubbed model is to a model trained without the forget set, and argued that hiding effects from the output is insufficient because information may still be recoverable by probing the weights [1911.04933]. “Selective Forgetting of Deep Networks at a Finer Level than Samples” then moved beyond sample-level deletion by introducing a three-term optimization composed of forgetting, correction, and remembering, with the forgotten object represented by transformations such as backdoor patterns or leakage cues rather than by entire samples [2012.11849].

Recent work broadened the scope of selective forgetting from parameter updates to system design. “Synergy Aware Forgetting Ensemble” treats future forgetting cost as a design-time quantity and optimizes shard topology and adapter structure under an expected unlearning budget [2304.13169]. “Black-Box Forgetting” shows that selective forgetting can also be implemented without parameter access by optimizing prompts with a derivative-free objective that jointly encourages forgetting on target classes and retention on the rest [2411.00409]. In continual learning, the explicit SFAO designation appears in a geometry-aware optimizer that can accept, project, or discard updates based on cosine similarity to stored gradients, positioning forgetting as a tunable component of optimization rather than a failure mode to be eliminated absolutely [2603.26671].

Across these lines of work, SFAO is not a single algorithm. In agent-memory systems it can mean graded decay, promotion, demotion, conflict resolution, and fusion over runtime memory items [2601.18642]. In machine unlearning it can mean bi-level optimization with forgetting as an inner objective and retention as an outer objective [2509.22483]. In calibration it can mean algebraic downdating of Gauss–Newton sufficient statistics to remove selected quotes without full re-estimation [2511.14980].

## 2. Canonical objective families

A central SFAO pattern is asymmetric retain–forget optimization. OFMU formulates machine unlearning with a retain set $\mathcal{D}_r$ and a forget set $\mathcal{D}_f$, but rejects static scalarization as unstable when forget and retain gradients conflict. Its inner objective is
$$
\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),
$$
and the outer problem minimizes retain loss subject to $\theta$ being an optimizer of the forgetting objective [2509.22483]. The associated penalty reformulation,
$$
F(\theta)=\mathcal{L}_r(\theta)+\rho\left\|\nabla_\theta \Phi(\theta)\right\|^2,
$$
encodes the principle that forgetting is primary and retention is optimized conditionally.

A second family treats forgetting as a constrained utility-allocation problem over external memory. In the adaptive budgeted forgetting framework,
$$
I(m_i,t)=\alpha \cdot R(m_i,t)+\beta \cdot F(m_i)+\gamma \cdot S(m_i,q_t),
$$
with recency
$$
R(m_i,t)=\exp(-\lambda (t-t_i)),
$$
and memory is selected by
$$
\mathcal{M}_t^*=\arg\max_{\mathcal{M}'\subseteq \mathcal{M}_t}\sum_{m_i\in\mathcal{M}'} I(m_i,t)
\quad \text{s.t.}\quad |\mathcal{M}'|\le \mathcal{B}.
$$
The same work adds a global loss
$$
\mathcal{L}_{total}=\mathcal{L}_{task}+\eta\cdot \frac{|\mathcal{M}_t|}{\mathcal{B}},
$$
making the budget explicit in optimization rather than in post-hoc pruning [2604.02280].

A third family models forgetting as controlled decay and consolidation. FadeMem defines each memory as
$$
m_i(t)=(c_i,s_i,v_i(t),\tau_i,f_i),
$$
with importance
$$
I_i(t)=\alpha\cdot \mathrm{rel}(c_i,Q_t)+\beta\cdot \frac{f_i}{1+f_i}+\gamma\cdot \mathrm{recency}(\tau_i,t),
$$
and strength decay
$$
v_i(t)=v_i(0)\cdot \exp\!\left(-\lambda_i\cdot (t-\tau_i)^{\beta_i}\right).
$$
Here forgetting is continuous, importance-aware, and hierarchical, not binary [2601.18642].

A fourth family frames forgetting through exact or near-exact model equivalence. Scrubbing-based selective forgetting minimizes a “Forgetting Lagrangian”
$$
\mathcal{L}=\mathbb{E}_{S(w)}[L_{D_r}(w)] + \lambda\, KL\big(P(\mathrm{Scrub}(w)\mid D)\,\|\,P(\mathrm{Scrub}_0(w)\mid D_r)\big),
$$
while PALL defines exact task unlearning by requiring the post-unlearning model to match the predictive distribution of a model trained from scratch without the forgotten task [1911.04933][2505.10941]. SAFE analogously defines exact forgetting by distributional equality between post-processed and retrained shard-graph models [2304.13169].

## 3. Parameter- and representation-level mechanisms

One major SFAO axis is where in parameter or representation space forgetting is enforced. In lifelong learning with selective forgetting via contrastive strategy, preserved classes are made compact in feature space, whereas deleted classes are driven toward “dispersed and irregular” features, so the network has no regular response to samples from a deleted class; the abstract states that this joint task is dubbed Learning with Selective Forgetting and that experiments on four benchmark datasets achieve a new state of the art [2405.18663]. This is a representation-shaping view: forgetting is induced by disturbing class structure rather than directly zeroing parameters.

A more explicit three-term formulation appears in finer-than-sample selective forgetting. There the optimization combines a forgetting term, a correction term on cleaned versions of the forget set, and a remembering term based on a diagonal Fisher penalty. The paper uses random network distillation or random label distillation as the forgetting term and an EWC-style quadratic constraint to preserve the rest of the task, showing that selective forgetting can target particular transformed features such as backdoor patterns or leakage cues [2012.11849].

Machine-unlearning variants operationalize selectivity directly in gradient and parameter space. OFMU adds similarity-aware gradient decorrelation to its inner forgetting objective so that updates that increase forget loss are pushed away from retain-gradient directions [2509.22483]. SIMU localizes the forget-set influence by computing neuron-level attribution scores and then runs second-order GradDiff updates only on critical neurons and attention projections, freezing the rest of the MLP parameters; the update is therefore selective both in data space and in parameter space [2510.07822].

Generative-model work extends the same logic to multi-concept forgetting. In diffusion models, Dynamic Mask applies a time-varying sparsity mask to gradients, while the Concept-Aware Loss aligns forbidden concepts with their superclasses and uses distillation to preserve earlier forgetting during sequential unlearning. The total objective
$$
\mathcal{L}_{total}=\mathcal{L}_{unlearn}+\alpha \mathcal{L}_{align}+\beta \mathcal{L}_{reg}
$$
therefore makes SFAO simultaneously parameter-selective, semantics-aware, and history-aware [2504.09039].

A black-box variant replaces parameter access with prompt-space optimization. “Black-Box Forgetting” defines a memorization cross-entropy on retained classes, a forgetting loss based on negative entropy on forgotten classes, and optimizes a shared latent prompt with CMA-ES under black-box confidence access. Latent Context Sharing introduces common and unique low-dimensional components across prompt tokens, reducing the search dimension while preserving the joint forget–retain objective [2411.00409].

## 4. Architecture- and system-level realizations

In long-horizon agent memory, SFAO often appears as architecture-level control rather than as one optimizer step. FadeMem uses a dual-layer memory hierarchy with Long-term Memory Layer and Short-term Memory Layer, differential decay rates, reinforcement on access, threshold-based pruning, LLM-guided conflict resolution, and LLM-guided fusion. The complete update cycle,
$$
\mathcal{M}_{t+\Delta t}=\mathrm{Fusion}\big(\mathrm{Resolution}(\mathrm{Decay}(\mathcal{M}_t,\Delta t)\cup\{m_{new}\})\big),
$$
makes forgetting a first-class policy over memory utility under capacities of 1000 entries in LML and 500 in SML [2601.18642].

A closely related agent-memory formulation is the adaptive budgeted forgetting framework, which treats memory retention as bounded optimization over relevance-guided scores and hard capacity constraints [2604.02280]. FSFM extends the systems perspective by defining four forgetting families—passive decay-based, active deletion-based, safety-triggered, and adaptive reinforcement-based—and by scoring memory with
$$
\text{Importance Score}=\alpha \times \text{CQA}+\beta \times \text{BVE}+\gamma \times \text{TRS}+\delta \times \text{SRC},
$$
with experimentally chosen weights $(0.4,0.3,0.2,0.1)$ and TRS given by $e^{-\lambda t}\cdot \text{usage\_frequency}$ [2604.20300].

Federated and distributed variants move the selectivity to networked optimization. SCALE first computes layer sensitivity scores from parameter correlation and KL-based distributional impact, selects top-$M$ sensitive layers, and then applies Age-of-Information–aware adaptive sparsification at weight-subgroup level via PPO. Its reward combines forgetting effectiveness and information freshness, and its theory gives an $O(\sqrt{L/|\mathcal{L}_s|})$ convergence advantage for dual-level selection over uniform modification [2605.22589]. SAFE likewise treats future forgetting cost as an optimization variable by training on shard graphs rather than independent shards, using adapters and clique-structured connectivity to trade off synergy against retraining scope [2304.13169].

Task-incremental lifelong learning offers yet another systems realization. PALL optimizes task-specific sparse subnetworks with masks, freezes previously used parameters, reinitializes unused parameters to prevent leakage, and performs exact task unlearning by resetting task-owned parameters and replay-retraining only the affected shared subset using retained-task buffers [2505.10941]. Outside machine learning in the narrow sense, option calibration yields an operator-theoretic SFAO: after caching per-quote Gauss–Newton contributions $u_i=J_i^\top r_i$ and $\psi_i=J_i^\top J_i$, selective forgetting of a quote subset $F$ is implemented by
$$
H' = H - \sum_{i\in F}\psi_i,\qquad G' = G - \sum_{i\in F}u_i,
$$
followed by a tiny linear solve, which is locally exact under the fixed linearization [2511.14980].

## 5. Evaluation regimes and empirical trade-offs

Reported evaluation criteria vary sharply by domain, which is itself a defining feature of SFAO. Continual-learning SFAO measures average accuracy, backward transfer, forgetting, plasticity–stability trade-offs, and memory cost [2603.26671]. Machine-unlearning work uses retain/forget accuracy, Forget Quality, Model Utility, Forget Truth Ratio, or membership-inference metrics [2509.22483][2510.07822]. Agent-memory systems report retention, retrieval quality, reasoning fidelity, false-memory behavior, and storage reduction [2601.18642][2604.02280][2604.20300]. Feature-level selective forgetting evaluates separate accuracies on $D_f$, $D_r$, $D_{f,\mathrm{clean}}$, and $D_{r,\mathrm{extra}}$, precisely because the forgotten object is a transformation or cue rather than a sample [2012.11849].

| Setting | Representative reported outcome | Paper |
|---|---|---|
| Continual-learning SFAO | markedly lower memory cost, a 90% reduction | [2603.26671] |
| Agent memory | 45% storage reduction with Critical Facts 82.1%, Contextual Info 71.0%, Storage Used 55.0% on LTI-Bench | [2601.18642] |
| Agent memory | access efficiency (+8.49%), content quality (+29.2% signal-to-noise ratio), security performance (100% elimination of security risks) | [2604.20300] |
| Black-box forgetting | CIFAR-10: $Err_{\mathrm{for}}=96.10$, $Acc_{\mathrm{mem}}=94.06$, $H=95.07$ | [2411.00409] |
| Diffusion multi-concept unlearning | 10-class unlearning: Total Acc $0.084$, CLIP $\approx 26.43$ | [2504.09039] |
| Safety-sensitive unlearning | WMDP: 73.0% (Bio), 70.2% (Cyber), 74.8% (MMLU) | [2509.22483] |

These metrics also expose a recurrent distinction between approximate and exact forgetting. OFMU, SIMU, diffusion unlearning, and black-box prompt forgetting optimize a retain–forget trade-off but do not claim retraining-equivalent erasure [2509.22483][2510.07822][2504.09039][2411.00409]. By contrast, SAFE, PALL, and operator-theoretic Gauss–Newton forgetting each define exactness by equivalence to a model trained without the forgotten subset, although the mechanism differs—distributional equality in SAFE, predictive-distribution equality in PALL, and local exactness under fixed linearization in option calibration [2304.13169][2505.10941][2511.14980]. A related caution appears in scrubbing work: low output accuracy on the forget set is not enough, because information may remain recoverable from weights unless the post-scrub distribution is controlled [1911.04933].

## 6. Limitations, distinctions, and open directions

A first distinction is between forgetting in model parameters and forgetting in external memory. In OFMU, SIMU, diffusion unlearning, or scrubbing, the target is the model itself [2509.22483][2510.07822][2504.09039][1911.04933]. In FadeMem, FSFM, or adaptive budgeted forgetting, the target is a runtime memory store, and forgetting may be realized by continuous decay, promotion/demotion, or active deletion rather than by parameter change [2601.18642][2604.20300][2604.02280]. Treating these as identical would be misleading; the commonality is selective optimization, not substrate.

A second distinction is between exactness and utility-oriented approximation. SAFE and PALL are designed around exact unlearning definitions, while many strong empirical methods remain heuristic or local in their guarantees [2304.13169][2505.10941]. Option-calibration forgetting is exact only under a fixed Gauss–Newton linearization and inherits the usual conditioning limitations of nonlinear least squares [2511.14980]. Scrubbing methods depend on stability and local quadratic assumptions, and their deep-network implementations rely on Fisher or variational approximations rather than exact Hessian geometry [1911.04933].

Current limitations are also highly domain-specific. FadeMem notes hand-tuned hyperparameters, risk of forgetting rare but important information, a fixed dual-layer architecture, and the absence of an explicit global objective [2601.18642]. FSFM points to evaluation difficulties, absence of universally accepted benchmarks, and the need for interpretable and auditable policies [2604.20300]. Black-box forgetting requires access to continuous prompt embeddings and is computationally heavy because CMA-ES runs for hundreds of iterations with population size 20 [2411.00409]. Continual-learning SFAO depends on threshold tuning, task ordering, and a conservative Monte Carlo approximation that may underuse beneficial updates [2603.26671]. PALL assumes task-incremental supervision, task-level rather than data-point-level unlearning, and episodic memory for retained tasks [2505.10941].

The present literature therefore supports a precise but plural view of SFAO. Forgetting can be continuous or discrete, exact or approximate, parameter-level or memory-level, white-box or black-box, and local or architecture-wide. The unifying principle is the explicit optimization of selective erasure under competing constraints—retention, plasticity, safety, privacy, memory, compute, or calibration fidelity. This suggests that future work will likely focus less on a single universal optimizer than on sharper interfaces between objective design, attribution or sensitivity estimation, system architecture, and formal notions of what it means for information to be forgotten.

Source: https://www.emergentmind.com/topics/selective-forgetting-aware-optimization-sfao