Selective Forgetting-Aware Optimization (SFAO)

• Selective Forgetting-Aware Optimization (SFAO) is a framework that treats forgetting as a design objective, explicitly regulating gradient directions and memory updates.
• It encompasses diverse methodologies including cosine-similarity gating, constrained optimization, and continuous decay to achieve both exact and approximate unlearning.
• SFAO balances the trade-offs between erasing unwanted information and retaining useful knowledge, addressing challenges in privacy, efficiency, and safety.

Selective Forgetting-Aware Optimization (SFAO) denotes a family of optimization formulations in which forgetting is treated as an explicit design objective rather than an incidental side effect of adaptation. The term is used directly for a cosine-similarity–gated continual-learning optimizer that “regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability” (Singh et al., 8 Feb 2026). Closely related formulations appear in machine unlearning, lifelong learning, agent memory management, federated unlearning, and nonlinear calibration, where the governing problem is to remove designated information while preserving retained utility, respecting memory or communication budgets, or satisfying safety and privacy constraints (Asif et al., 26 Sep 2025, Wei et al., 26 Jan 2026, Ding et al., 21 May 2026, Özsoy, 18 Nov 2025). This suggests a broader umbrella concept: optimization procedures that are selectively aware of what must be forgotten, what must be retained, and which constraints dominate that trade-off.

1. Historical emergence and conceptual scope

Early selective-forgetting work in deep networks already had the essential SFAO structure. “Eternal Sunshine of the Spotless Net” formalized post-hoc scrubbing as an optimization over retained-task loss plus a KL term measuring how close the scrubbed model is to a model trained without the forget set, and argued that hiding effects from the output is insufficient because information may still be recoverable by probing the weights (Golatkar et al., 2019). “Selective Forgetting of Deep Networks at a Finer Level than Samples” then moved beyond sample-level deletion by introducing a three-term optimization composed of forgetting, correction, and remembering, with the forgotten object represented by transformations such as backdoor patterns or leakage cues rather than by entire samples (Hayase et al., 2020).

Recent work broadened the scope of selective forgetting from parameter updates to system design. “Synergy Aware Forgetting Ensemble” treats future forgetting cost as a design-time quantity and optimizes shard topology and adapter structure under an expected unlearning budget (Dukler et al., 2023). “Black-Box Forgetting” shows that selective forgetting can also be implemented without parameter access by optimizing prompts with a derivative-free objective that jointly encourages forgetting on target classes and retention on the rest (Kuwana et al., 2024). In continual learning, the explicit SFAO designation appears in a geometry-aware optimizer that can accept, project, or discard updates based on cosine similarity to stored gradients, positioning forgetting as a tunable component of optimization rather than a failure mode to be eliminated absolutely (Singh et al., 8 Feb 2026).

Across these lines of work, SFAO is not a single algorithm. In agent-memory systems it can mean graded decay, promotion, demotion, conflict resolution, and fusion over runtime memory items (Wei et al., 26 Jan 2026). In machine unlearning it can mean bi-level optimization with forgetting as an inner objective and retention as an outer objective (Asif et al., 26 Sep 2025). In calibration it can mean algebraic downdating of Gauss–Newton sufficient statistics to remove selected quotes without full re-estimation (Özsoy, 18 Nov 2025).

2. Canonical objective families

A central SFAO pattern is asymmetric retain–forget optimization. OFMU formulates machine unlearning with a retain set $\mathcal{D}_r$ and a forget set $\mathcal{D}_f$, but rejects static scalarization as unstable when forget and retain gradients conflict. Its inner objective is

$$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$

and the outer problem minimizes retain loss subject to $\theta$ being an optimizer of the forgetting objective (Asif et al., 26 Sep 2025). The associated penalty reformulation,

$$F(\theta)=\mathcal{L}_r(\theta)+\rho\left\|\nabla_\theta \Phi(\theta)\right\|^2,$$

encodes the principle that forgetting is primary and retention is optimized conditionally.

A second family treats forgetting as a constrained utility-allocation problem over external memory. In the adaptive budgeted forgetting framework,

$$I(m_i,t)=\alpha \cdot R(m_i,t)+\beta \cdot F(m_i)+\gamma \cdot S(m_i,q_t),$$

with recency

$R(m_i,t)=\exp(-\lambda (t-t_i)),$

and memory is selected by

$$\mathcal{M}_t^*=\arg\max_{\mathcal{M}'\subseteq \mathcal{M}_t}\sum_{m_i\in\mathcal{M}'} I(m_i,t) \quad \text{s.t.}\quad |\mathcal{M}'|\le \mathcal{B}.$$

The same work adds a global loss

$$\mathcal{L}_{total}=\mathcal{L}_{task}+\eta\cdot \frac{|\mathcal{M}_t|}{\mathcal{B}},$$

making the budget explicit in optimization rather than in post-hoc pruning (Fofadiya et al., 2 Apr 2026).

A third family models forgetting as controlled decay and consolidation. FadeMem defines each memory as

$m_i(t)=(c_i,s_i,v_i(t),\tau_i,f_i),$

with importance

$\mathcal{D}_f$0

and strength decay

$\mathcal{D}_f$1

Here forgetting is continuous, importance-aware, and hierarchical, not binary (Wei et al., 26 Jan 2026).

A fourth family frames forgetting through exact or near-exact model equivalence. Scrubbing-based selective forgetting minimizes a “Forgetting Lagrangian”

$\mathcal{D}_f$2

while PALL defines exact task unlearning by requiring the post-unlearning model to match the predictive distribution of a model trained from scratch without the forgotten task (Golatkar et al., 2019, 2505.10941). SAFE analogously defines exact forgetting by distributional equality between post-processed and retrained shard-graph models (Dukler et al., 2023).

3. Parameter- and representation-level mechanisms

One major SFAO axis is where in parameter or representation space forgetting is enforced. In lifelong learning with selective forgetting via contrastive strategy, preserved classes are made compact in feature space, whereas deleted classes are driven toward “dispersed and irregular” features, so the network has no regular response to samples from a deleted class; the abstract states that this joint task is dubbed Learning with Selective Forgetting and that experiments on four benchmark datasets achieve a new state of the art (Shan et al., 2024). This is a representation-shaping view: forgetting is induced by disturbing class structure rather than directly zeroing parameters.

A more explicit three-term formulation appears in finer-than-sample selective forgetting. There the optimization combines a forgetting term, a correction term on cleaned versions of the forget set, and a remembering term based on a diagonal Fisher penalty. The paper uses random network distillation or random label distillation as the forgetting term and an EWC-style quadratic constraint to preserve the rest of the task, showing that selective forgetting can target particular transformed features such as backdoor patterns or leakage cues (Hayase et al., 2020).

Machine-unlearning variants operationalize selectivity directly in gradient and parameter space. OFMU adds similarity-aware gradient decorrelation to its inner forgetting objective so that updates that increase forget loss are pushed away from retain-gradient directions (Asif et al., 26 Sep 2025). SIMU localizes the forget-set influence by computing neuron-level attribution scores and then runs second-order GradDiff updates only on critical neurons and attention projections, freezing the rest of the MLP parameters; the update is therefore selective both in data space and in parameter space (Agarwal et al., 9 Oct 2025).

Generative-model work extends the same logic to multi-concept forgetting. In diffusion models, Dynamic Mask applies a time-varying sparsity mask to gradients, while the Concept-Aware Loss aligns forbidden concepts with their superclasses and uses distillation to preserve earlier forgetting during sequential unlearning. The total objective

$\mathcal{D}_f$3

therefore makes SFAO simultaneously parameter-selective, semantics-aware, and history-aware (Li et al., 12 Apr 2025).

A black-box variant replaces parameter access with prompt-space optimization. “Black-Box Forgetting” defines a memorization cross-entropy on retained classes, a forgetting loss based on negative entropy on forgotten classes, and optimizes a shared latent prompt with CMA-ES under black-box confidence access. Latent Context Sharing introduces common and unique low-dimensional components across prompt tokens, reducing the search dimension while preserving the joint forget–retain objective (Kuwana et al., 2024).

4. Architecture- and system-level realizations

In long-horizon agent memory, SFAO often appears as architecture-level control rather than as one optimizer step. FadeMem uses a dual-layer memory hierarchy with Long-term Memory Layer and Short-term Memory Layer, differential decay rates, reinforcement on access, threshold-based pruning, LLM-guided conflict resolution, and LLM-guided fusion. The complete update cycle,

$\mathcal{D}_f$4

makes forgetting a first-class policy over memory utility under capacities of 1000 entries in LML and 500 in SML (Wei et al., 26 Jan 2026).

A closely related agent-memory formulation is the adaptive budgeted forgetting framework, which treats memory retention as bounded optimization over relevance-guided scores and hard capacity constraints (Fofadiya et al., 2 Apr 2026). FSFM extends the systems perspective by defining four forgetting families—passive decay-based, active deletion-based, safety-triggered, and adaptive reinforcement-based—and by scoring memory with

$\mathcal{D}_f$5

with experimentally chosen weights $\mathcal{D}_f$6 and TRS given by $\mathcal{D}_f$7 (Gu et al., 22 Apr 2026).

Federated and distributed variants move the selectivity to networked optimization. SCALE first computes layer sensitivity scores from parameter correlation and KL-based distributional impact, selects top-$\mathcal{D}_f$8 sensitive layers, and then applies Age-of-Information–aware adaptive sparsification at weight-subgroup level via PPO. Its reward combines forgetting effectiveness and information freshness, and its theory gives an $\mathcal{D}_f$9 convergence advantage for dual-level selection over uniform modification (Ding et al., 21 May 2026). SAFE likewise treats future forgetting cost as an optimization variable by training on shard graphs rather than independent shards, using adapters and clique-structured connectivity to trade off synergy against retraining scope (Dukler et al., 2023).

Task-incremental lifelong learning offers yet another systems realization. PALL optimizes task-specific sparse subnetworks with masks, freezes previously used parameters, reinitializes unused parameters to prevent leakage, and performs exact task unlearning by resetting task-owned parameters and replay-retraining only the affected shared subset using retained-task buffers (2505.10941). Outside machine learning in the narrow sense, option calibration yields an operator-theoretic SFAO: after caching per-quote Gauss–Newton contributions $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$0 and $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$1, selective forgetting of a quote subset $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$2 is implemented by

$$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$3

followed by a tiny linear solve, which is locally exact under the fixed linearization (Özsoy, 18 Nov 2025).

5. Evaluation regimes and empirical trade-offs

Reported evaluation criteria vary sharply by domain, which is itself a defining feature of SFAO. Continual-learning SFAO measures average accuracy, backward transfer, forgetting, plasticity–stability trade-offs, and memory cost (Singh et al., 8 Feb 2026). Machine-unlearning work uses retain/forget accuracy, Forget Quality, Model Utility, Forget Truth Ratio, or membership-inference metrics (Asif et al., 26 Sep 2025, Agarwal et al., 9 Oct 2025). Agent-memory systems report retention, retrieval quality, reasoning fidelity, false-memory behavior, and storage reduction (Wei et al., 26 Jan 2026, Fofadiya et al., 2 Apr 2026, Gu et al., 22 Apr 2026). Feature-level selective forgetting evaluates separate accuracies on $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$4, $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$5, $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$6, and $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$7, precisely because the forgotten object is a transformation or cue rather than a sample (Hayase et al., 2020).

Setting	Representative reported outcome	Paper
Continual-learning SFAO	markedly lower memory cost, a 90% reduction	(Singh et al., 8 Feb 2026)
Agent memory	45% storage reduction with Critical Facts 82.1%, Contextual Info 71.0%, Storage Used 55.0% on LTI-Bench	(Wei et al., 26 Jan 2026)
Agent memory	access efficiency (+8.49%), content quality (+29.2% signal-to-noise ratio), security performance (100% elimination of security risks)	(Gu et al., 22 Apr 2026)
Black-box forgetting	CIFAR-10: $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$8, $$\Phi(\theta)=\mathcal{L}_f(\theta)-\beta\cdot \mathrm{Sim}\!\left(\nabla_\theta \mathcal{L}_f(\theta),\nabla_\theta \mathcal{L}_r(\theta)\right),$$9, $\theta$0	(Kuwana et al., 2024)
Diffusion multi-concept unlearning	10-class unlearning: Total Acc $\theta$1, CLIP $\theta$2	(Li et al., 12 Apr 2025)
Safety-sensitive unlearning	WMDP: 73.0% (Bio), 70.2% (Cyber), 74.8% (MMLU)	(Asif et al., 26 Sep 2025)

These metrics also expose a recurrent distinction between approximate and exact forgetting. OFMU, SIMU, diffusion unlearning, and black-box prompt forgetting optimize a retain–forget trade-off but do not claim retraining-equivalent erasure (Asif et al., 26 Sep 2025, Agarwal et al., 9 Oct 2025, Li et al., 12 Apr 2025, Kuwana et al., 2024). By contrast, SAFE, PALL, and operator-theoretic Gauss–Newton forgetting each define exactness by equivalence to a model trained without the forgotten subset, although the mechanism differs—distributional equality in SAFE, predictive-distribution equality in PALL, and local exactness under fixed linearization in option calibration (Dukler et al., 2023, 2505.10941, Özsoy, 18 Nov 2025). A related caution appears in scrubbing work: low output accuracy on the forget set is not enough, because information may remain recoverable from weights unless the post-scrub distribution is controlled (Golatkar et al., 2019).

6. Limitations, distinctions, and open directions

A first distinction is between forgetting in model parameters and forgetting in external memory. In OFMU, SIMU, diffusion unlearning, or scrubbing, the target is the model itself (Asif et al., 26 Sep 2025, Agarwal et al., 9 Oct 2025, Li et al., 12 Apr 2025, Golatkar et al., 2019). In FadeMem, FSFM, or adaptive budgeted forgetting, the target is a runtime memory store, and forgetting may be realized by continuous decay, promotion/demotion, or active deletion rather than by parameter change (Wei et al., 26 Jan 2026, Gu et al., 22 Apr 2026, Fofadiya et al., 2 Apr 2026). Treating these as identical would be misleading; the commonality is selective optimization, not substrate.

A second distinction is between exactness and utility-oriented approximation. SAFE and PALL are designed around exact unlearning definitions, while many strong empirical methods remain heuristic or local in their guarantees (Dukler et al., 2023, 2505.10941). Option-calibration forgetting is exact only under a fixed Gauss–Newton linearization and inherits the usual conditioning limitations of nonlinear least squares (Özsoy, 18 Nov 2025). Scrubbing methods depend on stability and local quadratic assumptions, and their deep-network implementations rely on Fisher or variational approximations rather than exact Hessian geometry (Golatkar et al., 2019).

Current limitations are also highly domain-specific. FadeMem notes hand-tuned hyperparameters, risk of forgetting rare but important information, a fixed dual-layer architecture, and the absence of an explicit global objective (Wei et al., 26 Jan 2026). FSFM points to evaluation difficulties, absence of universally accepted benchmarks, and the need for interpretable and auditable policies (Gu et al., 22 Apr 2026). Black-box forgetting requires access to continuous prompt embeddings and is computationally heavy because CMA-ES runs for hundreds of iterations with population size 20 (Kuwana et al., 2024). Continual-learning SFAO depends on threshold tuning, task ordering, and a conservative Monte Carlo approximation that may underuse beneficial updates (Singh et al., 8 Feb 2026). PALL assumes task-incremental supervision, task-level rather than data-point-level unlearning, and episodic memory for retained tasks (2505.10941).

The present literature therefore supports a precise but plural view of SFAO. Forgetting can be continuous or discrete, exact or approximate, parameter-level or memory-level, white-box or black-box, and local or architecture-wide. The unifying principle is the explicit optimization of selective erasure under competing constraints—retention, plasticity, safety, privacy, memory, compute, or calibration fidelity. This suggests that future work will likely focus less on a single universal optimizer than on sharper interfaces between objective design, attribution or sensitivity estimation, system architecture, and formal notions of what it means for information to be forgotten.