Adaptive Budgeted Forgetting Architecture

Updated 2 May 2026

Adaptive Budgeted Forgetting is a framework that dynamically tunes forgetting rates to optimize resource-constrained models while preserving essential knowledge.
It leverages relevance scores, decay dynamics, and reinforcement learning to mitigate catastrophic forgetting and enforce strict memory and compute budgets.
The architecture delivers practical benefits such as significant storage reduction, improved throughput, and enhanced security in continual and online learning applications.

Adaptive Budgeted Forgetting Architecture refers to a class of memory and learning systems—spanning neural data structures, continual learning models, agent memory, and unlearning protocols—that explicitly regulate resource consumption and knowledge persistence via data- or task-driven, dynamically tunable forgetting processes. The core concept is to maintain system efficiency, scalability, and performance under fixed memory, compute, privacy, or regulatory budgets, while mitigating catastrophic forgetting and uncontrolled accumulation. Architectures typically deploy adaptive scores, reinforcement or meta-learning, decay dynamics, or structured pruning to balance relevance and resource constraints.

1. Foundations and Principles

Adaptive budgeted forgetting emerged from the need to bridge the retention stability–plasticity dilemma in resource-constrained settings: systems must forget selectively to make room for novel, valuable, and up-to-date content, without indiscriminately erasing useful prior knowledge. Conventional binary retention or monotonic weight-decay does not adapt to the dynamic importance, usage, or risk profile of memory items or model parameters.

Key distinguishing characteristics include:

Explicit, usually bounded, budgets for memory, compute, or update capacity.
Relevance- and importance-guided retention scores incorporating recency, frequency, semantic similarity, or utility.
Adaptive adjustment of forgetting rates, thresholds, or subspace allocations in response to workload, context, or performance signals.
Composite, often multi-objective, loss or reward formulations trading off accuracy, retention, resource constraints, and (when applicable) compliance requirements.

2. Core Architectural Strategies

Multiple instantiations of adaptive budgeted forgetting have been developed across memory-augmented agents, continual learning models, and machine unlearning:

Memory-Selective Agents

Architectures such as FadeMem (Wei et al., 26 Jan 2026) and FSFM (Gu et al., 22 Apr 2026) introduce multi-layered memory with active forgetting:

Dual-layer Memory: Short-term and long-term layers apply fast and slow adaptive decays, respectively, controlled by dynamically computed importance scores tied to semantic relevance, decayed frequency, and recency.
Budget Enforcement: Hard or soft storage limits are maintained; low-importance or stale items are pruned or down-weighted when budgets (capacity or utility) are exceeded.
Selective Mechanisms: Forgetting is orchestrated by passive decay (exponential, power-law, or composite decay), active deletion (triggered by policy or user request), safety/risk triggers for privacy/security, and, in advanced forms, adaptive reinforcement modules learning drop/retain policies via RL.

Continual and Online Learning

Adaptive weight decay mechanisms, such as FADE (Ramesh et al., 29 Apr 2026) and orthogonal adapters with dynamic budgets (Wan et al., 28 May 2025), operate directly on model parameters:

Per-Parameter Adaptive Forgetting: Meta-gradient or policy-based methods allocate higher decay to weights sensitive to nonstationary targets, and lower decay to stable, important knowledge, optimizing retention within a fixed or learned budget.
Budgeted Adapter Allocation: OA-Adapter tunes per-task/layer parameter budgets via a bidirectional soft-thresholding mechanism, learning both allocation and task objectives concurrently, with orthogonality constraints to prevent destructive interference between tasks.

Machine Unlearning and Security

Protocols such as SAFE (Dukler et al., 2023) and FiCABU (Cho et al., 6 Nov 2025) enforce budgeted and adaptive forgetting or erasure at the data or hardware level:

Shard-graph structured unlearning: By compartmentalizing training data into graph-connected shards, unlearning cost is minimized (expected number of retrained samples), and budget/accuracy trade-offs are tuned via graph topology, shard size, and adapter composition.
Resource/energy-conscious hardware unlearning: FiCABU orchestrates context-adaptive unlearning by halting layer-wise edits once the forget budget or statistical target is reached, applying depth-aware scaling to preserve retain accuracy while minimizing energy and compute expenditures.

3. Mathematical Formulations and Optimization

Adaptive budgeted forgetting mechanisms typically rely on a combination of formal score assignments, resource constraints, and dynamic control policies:

Relevance Scoring:

$I_i(t) = \alpha \cdot \mathrm{rel}(c_i, Q_t) + \beta \frac{\tilde{f}_i}{1+\tilde{f}_i} + \gamma \exp(-\delta (t-\tau_i))$

Decay Scheduling:

$v_i(t) = v_i(0)\,\exp(-\lambda_i\,(t-\tau_i)^{\beta_i}), \quad \lambda_i = \lambda_\mathrm{base}\,\exp(-\mu I_i(t))$

Budgeted Retention:

$\mathcal{M}_t^* = \arg\max_{S \subseteq \mathcal{M}_t} \sum_{m \in S} R(m, t) \quad \text{s.t.}\quad |S| \leq B$

Reinforcement-based Forgetting Objective:

$\max_\pi \mathbb{E}_\pi \left[\sum_{t=0}^T \gamma^t R(s_t, a_t)\right],\quad \text{subject to}~ \sum_{m_i \in M_t} \mathrm{size}(m_i) \leq B\ \forall t$

Adapter Budget Adaptation (OA-Adapter):
- Soft-threshold mask: $\gamma_i = \mathrm{sign}(g_i) \cdot \max(|g_i| - \tau, 0)$
- Orthogonality penalty: $L_{\mathrm{orth}}^{(s,t)} = \sum_{i,j} \langle W_2^{(t)}[:, i], \tilde{W}_2^{(s)}[:, j]\rangle^2$

System pseudocode and update routines mirror these dynamics, enforcing hard/soft limits, gated retention, and runtime adaptation during insertion, consolidation, pruning, and retraining steps.

4. Evaluation and Comparative Results

Empirical studies across diverse settings highlight the impact of adaptive budgeted forgetting:

Agent Memory and Reasoning:
- FadeMem (Wei et al., 26 Jan 2026) achieves 45% storage reduction, maintains or improves reasoning (e.g., Multi-hop F1: 29.43 vs. 28.37, MSC RP@10: 77.2%), and sharply attenuates false memory propagation.
- FSFM (Gu et al., 22 Apr 2026) eliminates 100% of dangerous content under the safety-triggered budget, with storage utilization down 30% and +30% query throughput.
Continual Learning:
- OA-Adapter (Wan et al., 28 May 2025) outperforms fixed-budget LoRA by up to 0.7% accuracy, achieving comparable or better performance with 46–58% fewer parameters, capping forgetting at ~14% vs. 100% for naive baselines.
- SFAO (Singh et al., 8 Feb 2026) reduces memory cost by 90% while maintaining competitive task performance.
Unlearning and Edge AI:
- SAFE (Dukler et al., 2023) enables explicit trade-offs in the accuracy–forgetting cost space, uniformly dominating SISA by 5–15% at fixed cost.
- FiCABU (Cho et al., 6 Nov 2025) achieves random-guess forget accuracy, up to 87.5% compute reduction, and 0.13–6.5% energy usage vs. baseline.

Table: Summary of Representative Architectures

Method	Core Mechanism	Budget Type
FadeMem (Wei et al., 26 Jan 2026)	Dual-layer adaptive decay & importance scoring	Memory, semantic relevance
OA-Adapter (Wan et al., 28 May 2025)	Soft-thresholded, bidirectional adapter allocation & orthogonality	Parameter count, task complexity
FSFM (Gu et al., 22 Apr 2026)	Multi-faceted (passive, active, RL-based) forgetting	Memory, safety/security
FADE (Ramesh et al., 29 Apr 2026)	Meta-gradient per-parameter weight decay	Parameter capacity
SAFE (Dukler et al., 2023)	Shard-graph, graph-structured adapters, perfect unlearning	Retraining cost, accuracy
FiCABU (Cho et al., 6 Nov 2025)	Context-adaptive hardware-level unlearning	Compute, energy, forget accuracy

5. Practical Implementation and Hyperparameterization

Robust deployment of adaptive budgeted forgetting architectures requires careful configuration of decay rates, importance weights, gating thresholds, and resource monitoring pipelines:

Decay base and shape factors, e.g., $\lambda_\mathrm{base}=0.1$ , $\beta_\mathrm{LML}=0.8$ , $\beta_\mathrm{SML}=1.2$ (Wei et al., 26 Jan 2026).
Budget caps suited to task demands (e.g., $B=100$ – $v_i(t) = v_i(0)\,\exp(-\lambda_i\,(t-\tau_i)^{\beta_i}), \quad \lambda_i = \lambda_\mathrm{base}\,\exp(-\mu I_i(t))$ 0 units in conversational agents (Fofadiya et al., 2 Apr 2026)), RL learning rates (e.g., $v_i(t) = v_i(0)\,\exp(-\lambda_i\,(t-\tau_i)^{\beta_i}), \quad \lambda_i = \lambda_\mathrm{base}\,\exp(-\mu I_i(t))$ 1 (Gu et al., 22 Apr 2026)), and storage structures (heap or ANN index for fast retrieval/pruning).
Adapter dimension and threshold schedules for OA-Adapter, balancing sparsity vs. expressivity on a per-layer, per-task basis (Wan et al., 28 May 2025).
Compliance and safety (GDPR, audit logging, safety-triggered deletion policies) are handled by active forget rules and audit logs (Gu et al., 22 Apr 2026, Dukler et al., 2023).

Integration typically involves periodic or event-driven trigger routines for updating memory, retraining modules, or performing unlearning operations. Hardware-aware methods (FiCABU) enforce real-time monitoring of compute and energy budgets, dynamically terminating unlearning passes as soon as statistical or hard resource limits are satisfied (Cho et al., 6 Nov 2025).

6. Theoretical and Practical Impact

Adaptive budgeted forgetting architectures have catalyzed advances across several domains:

Mitigating catastrophic forgetting in continual and online learning, especially under realistic, resource-constrained conditions (Wan et al., 28 May 2025, Ramesh et al., 29 Apr 2026).
Securing and controlling information propagation and retention in LLM-based agent systems, aligning with regulatory and privacy requirements (Gu et al., 22 Apr 2026, Dukler et al., 2023).
Machine unlearning on edge devices, achieving strong computational savings and strict adherence to privacy targets with minimal performance loss (Cho et al., 6 Nov 2025).
Reducing false memory rates and preventing context pollution in long-horizon dialogue agents (Fofadiya et al., 2 Apr 2026, Wei et al., 26 Jan 2026).
Maintaining or improving efficiency and throughput via structured relevance scoring, adaptive pruning, and compute-aware protocols.

These architectures demonstrate that biologically-inspired, dynamically regulated forgetting can be a robust enabler of scalable, responsible, and high-performing AI systems in diverse environments.

7. Open Challenges and Future Directions

Despite significant progress, pressing challenges remain:

Jointly optimizing for long-term stability, rapid adaptation, resource efficiency, and privacy in highly nonstationary, multi-agent, or federated settings.
Extending adaptive budgeted forgetting to multimodal, hierarchical, or cross-domain memory and learning architectures.
Developing general theoretical guarantees and analytic frameworks for safety-triggered or regulatory-compliant forgetting mechanisms in large-scale deployments.
Balancing competing objectives—task performance, computational budget, and knowledge retention—beyond linear or additive trade-off formulations.
Fine-grained, real-time monitoring and auditing of forgetting processes without incurring prohibitive overhead in live systems.

Adaptive budgeted forgetting architectures continue to unify ideas from cognitive science, RL, online optimization, and privacy-preserving machine learning, providing rigorous and scalable solutions for the evolving demands of modern AI.