Bounded Parameter-Efficient Unlearning

• Bounded parameter-efficient unlearning is a set of methods that restricts parameter changes to erase specific data influence while maintaining model utility.
• Techniques like per-parameter task arithmetic, low-rank updates, and module-aware masking demonstrate controlled interventions for effective data deletion.
• These methods balance the trade-off between forgetting and retention, achieving efficiency and formal guarantees compared to full retraining.

Bounded parameter-efficient unlearning, as the term is used across recent machine unlearning research, can be understood as a family of methods that remove the influence of a forget set while preserving the retain set or overall model utility under explicit restrictions on the intervention itself. Those restrictions may take the form of updating only a small subset of parameters, constraining update magnitudes, bounding parameter or functional drift, bounding privacy leakage, or bounding retraining workload. In current work, this design space includes per-parameter task arithmetic for LLMs, module-aware masking for Transformers, bounded LoRA parameterizations, anchored post-hoc optimization, sparse embedding edits, exact layer deactivation, activation-space protection, representation-space information bottlenecks, and even architectures in which unlearning is performed by deleting exemplars at test time rather than editing weights (Cai et al., 29 Jan 2026, Bao et al., 24 Aug 2025, Garg et al., 29 Sep 2025, Sanga et al., 17 Jun 2025, Miksa et al., 15 May 2026, Chowdhury et al., 2024, Zheng et al., 24 Mar 2026).

1. Problem setting and meanings of boundedness

Most formulations begin with a trained model and a partition of data into a forget set and a retain set. In the notation used for LLM unlearning, $\theta_0$ denotes pretrained parameters, $\mathcal{D}_{\mathrm{f}}$ the forget set, and $\mathcal{D}_{\mathrm{r}}$ the retain set. The central objective is to erase knowledge about $\mathcal{D}_{\mathrm{f}}$ without disrupting unrelated or retained behavior, but the literature differs on what is meant by a “bounded” intervention. Some papers bound parameter changes directly, some bound the effect of changes on outputs or hidden activations, some bound privacy leakage, and some bound the amount of retraining work permitted after a deletion request (Cai et al., 29 Jan 2026, Sanga et al., 17 Jun 2025, Miksa et al., 15 May 2026, Balordi et al., 26 Aug 2025, Li et al., 2024).

Mechanism	What is bounded or localized	Representative methods
Per-parameter or module selection	task vector rescaling, masks, PEFT subsets	PerTA, MAPE-Unlearn, Fast-NTK
Adapter or post-hoc parameter control	bounded LoRA updates, anchored deviation, selective reset	Stable Forgetting, FAMR, federated TIS reset
Activation or representation intervention	functional drift outside forget region, information bottleneck	BARRIER, Representation Unlearning
Architectural deletion	affected layers or explicit sample memory	S3T, deep semi-parametric models

A second axis is exactness. Approximate methods directly modify the current model and attempt to match retraining behavior efficiently. Exact methods isolate influence during training so that deletion can be implemented by deactivating the affected components rather than re-optimizing the whole model. Certified methods establish a formal closeness guarantee between the unlearned model and a retrained reference under assumptions such as convexity or strong convexity (Chowdhury et al., 2024, Chien et al., 2024).

This suggests that bounded parameter-efficient unlearning is not a single algorithmic recipe but a common design principle: restrict the locus of change and make the forget–retain trade-off explicit.

2. Per-parameter and low-rank unlearning in LLMs

A central LLM line of work starts from task arithmetic. Standard task arithmetic subtracts a task vector $V=\theta_{\rm fgt}-\theta_0$, where $\theta_{\rm fgt}$ is fine-tuned only on the forget set. Per-parameter Task Arithmetic (PerTA) replaces global rescaling by a per-parameter weight vector $W$:

$$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$

The weights quantify relative importance for forgetting versus retention and are estimated either from gradients (PerTA-grad) or from the diagonal Fisher information approximation (PerTA-fisher). The paper also defines a retain-forget ratio and a more general family $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$, with $\tau=1$ corresponding to PerTA-grad and $\mathcal{D}_{\mathrm{f}}$0 to PerTA-fisher. On TOFU and MUSE, PerTA-grad and PerTA-fisher outperform vanilla task vectors and, in many cases, surpass training-based baselines such as GA, GD, NPO, and NPO+, while retaining the efficiency of task arithmetic and reducing over-forgetting (Cai et al., 29 Jan 2026).

A related diagnosis concerns instability in gradient-difference unlearning for LLMs. In that setting, gradient descent on retained data is combined with gradient ascent on forget data,

$\mathcal{D}_{\mathrm{f}}$1

but ascent on cross-entropy can cause unbounded growth of weights and gradients in transformer MLP layers. “Stable Forgetting” addresses this by applying bounded functions such as $\mathcal{D}_{\mathrm{f}}$2 or $\mathcal{D}_{\mathrm{f}}$3 elementwise to LoRA low-rank updates in feedforward layers. The method is explicitly parameter-efficient, updating just $\mathcal{D}_{\mathrm{f}}$4–$\mathcal{D}_{\mathrm{f}}$5 of model weights, and is evaluated on TOFU, TDEC, and MUSE across GPT-Neo, Phi, and LLaMA scales from $\mathcal{D}_{\mathrm{f}}$6M to $\mathcal{D}_{\mathrm{f}}$7B (Garg et al., 29 Sep 2025).

Other LLM PEFT approaches bound interventions operationally rather than analytically. The SemEval-2025 system based on LoRA and layer-focused fine-tuning splits the forget set into disjoint chunks, merges each chunk with cyclically sampled retain samples, and optimizes a gradient-difference loss $\mathcal{D}_{\mathrm{f}}$8. The method reports that chunking and retain-dominant batching stabilize training, and its LoRA-based configuration ranked first on the leaderboard with a final score of $\mathcal{D}_{\mathrm{f}}$9 versus $\mathcal{D}_{\mathrm{r}}$0 for the second-best system (Premptis et al., 4 Mar 2025).

LLMEraser gives a different parameter-efficient route by editing only PEFT parameters through influence functions. It formalizes instance removal, query modification, and response correction as perturbations of the empirical risk objective, then reformulates the inverse-Hessian-vector computation as a convex quadratic optimization problem

$\mathcal{D}_{\mathrm{r}}$1

The paper reports linear time and space complexity in the PEFT parameter size, bounded estimation error from the Taylor approximation and optimization procedure, memory reduction of about $\mathcal{D}_{\mathrm{r}}$2–$\mathcal{D}_{\mathrm{r}}$3, and a typical speedup greater than $\mathcal{D}_{\mathrm{r}}$4 versus full retraining (Ding et al., 2024).

3. Structured sparsity, module locality, and selective reset

A broader parameter-efficient strategy is to identify influence-critical subsets rather than reweight every parameter. MAPE-Unlearn is explicitly module-aware for Transformers: it uses a learnable pair of binary masks over attention heads and feed-forward filters, with the mask objective derived from forget-set gradients and retain-set diagonal Fisher information. The combinatorial selection problem is solved by greedy search with a warm start and layer-wise refinement. The method reports that very high sparsity is viable; for BERT-base, the paper contrasts about $\mathcal{D}_{\mathrm{r}}$5K modules with $\mathcal{D}_{\mathrm{r}}$6M parameters, and a $\mathcal{D}_{\mathrm{r}}$7 sparse MAPE-SO configuration on SQuAD v2.0 attains Unlearn F1 $\mathcal{D}_{\mathrm{r}}$8, Retain F1 $\mathcal{D}_{\mathrm{r}}$9, and Test F1 $\mathcal{D}_{\mathrm{f}}$0, close to the full second-order baseline while updating far fewer parameters (Bao et al., 24 Aug 2025).

Sparse Token Embedding Unlearning (STEU) localizes unlearning even more aggressively. It selects forget-class tokens by a frequency-weighted PMI score,

$\mathcal{D}_{\mathrm{f}}$1

and then updates only the chosen token-embedding rows and a small classifier head while freezing the full encoder. In the primary BioClinicalBERT/MIMIC-IV setting, STEU modifies $\mathcal{D}_{\mathrm{f}}$2 parameters, or about $\mathcal{D}_{\mathrm{f}}$3 of the model, achieves forget F1 $\mathcal{D}_{\mathrm{f}}$4, and retains average F1 $\mathcal{D}_{\mathrm{f}}$5. The same study reports consistent behavior across MIMIC-IV, MIMIC-III, and eICU and across BioClinicalBERT, BERT-base, and DistilBERT (Hou et al., 11 Mar 2026).

Fast-NTK constrains an NTK-based closed-form unlearning update to a PEFT subspace. For CNNs, only batch-normalization parameters are updated; for ViTs, only visual prompts are updated. The resulting Jacobians and kernel matrices are restricted to that subset, making the method scalable to models with $\mathcal{D}_{\mathrm{f}}$6M parameters and datasets with $\mathcal{D}_{\mathrm{f}}$7k images. Reported PEFT ratios range from about $\mathcal{D}_{\mathrm{f}}$8 to $\mathcal{D}_{\mathrm{f}}$9 for BN layers in CNNs and about $V=\theta_{\rm fgt}-\theta_0$0 for prompts in ViT-Base, while retaining performance comparable to retraining on the retain set alone (Li et al., 2023).

In federated learning, an analogous principle appears as selective reset guided by second-order information. The method models leakage as a parameter-estimation problem, computes a Target Information Score from diagonal Hessians, resets the top $V=\theta_{\rm fgt}-\theta_0$1 sensitive parameters, and minimally retrains only the reset subset. The paper reports Normalized Accuracy against retrained benchmarks of approximately $V=\theta_{\rm fgt}-\theta_0$2, MIA accuracy reduced to about random guess, and backdoor attack success rate reduced from $V=\theta_{\rm fgt}-\theta_0$3 to below $V=\theta_{\rm fgt}-\theta_0$4 after unlearning (Balordi et al., 26 Aug 2025).

4. Beyond weight space: representations, activations, and semi-parametric deletion

Several recent systems argue that the most effective bound may be to move away from global weight editing altogether. Representation Unlearning learns a transformation $V=\theta_{\rm fgt}-\theta_0$5 over internal representations rather than modifying backbone parameters. The framework imposes an information bottleneck by preserving information about retained data and suppressing information about forget data, with an overall objective

$V=\theta_{\rm fgt}-\theta_0$6

It supports both a standard regime with retain and forget data and a zero-shot regime where only forget data are available. The paper reports speedups of $V=\theta_{\rm fgt}-\theta_0$7 in the standard setting and up to $V=\theta_{\rm fgt}-\theta_0$8 in the zero-shot setting versus retraining, with peak GPU memory around $V=\theta_{\rm fgt}-\theta_0$9MB–$\theta_{\rm fgt}$0GB rather than $\theta_{\rm fgt}$1–$\theta_{\rm fgt}$2GB or more for parameter-centric baselines (Almudévar et al., 29 Jan 2026).

BARRIER relocates the intervention to hidden-layer activation geometry. It uses Interval Arithmetic on SVD-based projections of forget-set activations to construct a forget interval inside an activation hypercube, then minimizes a protection loss that penalizes mean shift, residual-subspace drift, and worst-case interval drift outside that region. Its main theorem gives a probabilistic tail bound on functional drift:

$\theta_{\rm fgt}$3

Empirically, BARRIER intervenes in only $\theta_{\rm fgt}$4–$\theta_{\rm fgt}$5 of parameters, matches state-of-the-art trade-offs across classifiers and diffusion models, and is described as ensuring rigorous protection of the retain distribution when retain activations remain outside the forget hypercube (Miksa et al., 15 May 2026).

Deep semi-parametric models (SPMs) take a still more radical step. They fuse a parametric branch with a non-parametric branch over an explicit sample set $\theta_{\rm fgt}$6 and compute predictions as $\theta_{\rm fgt}$7. Unlearning is performed by test-time deletion:

$\theta_{\rm fgt}$8

with no parameter update at all. On ImageNet classification, the paper reports that SPMs reduce the prediction gap relative to a retrained oracle baseline by $\theta_{\rm fgt}$9 and achieve over $W$0 faster unlearning than existing approaches on parametric models (Zheng et al., 24 Mar 2026).

This suggests that bounded parameter-efficient unlearning increasingly includes methods where the primary object being bounded is not the parameter vector itself but the information path through which the forget set influences predictions.

5. Exactness, certification, and formal guarantees

The exact end of the spectrum is represented by Sequence-aware Sharded Sliced Training (S3T). S3T partitions data into shards and slices and sequentially trains LoRA-style PEFT layers so that each layer’s update depends only on the corresponding slices. If data from slice $W$1 must be deleted, the method simply zeroes out the PEFT parameters from slice $W$2 onward. The result is exact unlearning by layer deactivation, with no retraining or service downtime. The paper gives deletion-rate scaling $W$3 compared with $W$4, and training cost $W$5 rather than $W$6 for full training (Chowdhury et al., 2024).

A different formalization is Forget-Aligned Model Reconstruction (FAMR), which treats post-hoc forgetting as anchored optimization:

$W$7

The anchor term explicitly limits parameter drift from the original model, and the paper provides bounds on both parameter deviation and output deviation from a retrained reference. On class forgetting for ViT-L, reported results include Ret-Acc $W$8 and For-Acc $W$9 on CIFAR-100 and Ret-Acc $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$0 and For-Acc $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$1 on ImageNet-100 (Sanga et al., 17 Jun 2025).

Certified approximate unlearning under convexity is developed through projected noisy stochastic gradient descent. The method establishes an approximate unlearning guarantee in Rényi divergence between the unlearned distribution and the retrained distribution and supports both sequential and batch unlearning. Under the same privacy constraint, the paper reports using $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$2 and $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$3 of the gradient computations compared with state-of-the-art gradient-based approximate unlearning methods for mini-batch and full-batch settings, respectively (Chien et al., 2024).

At a statistical level, recent theory nearly characterizes the minimax cost of pure $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$4-unlearning for smooth strongly convex losses. The excess population risk upper and lower bounds match up to a condition-number factor and take the form

$$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$5

The paper states that when $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$6, the unlearning penalty becomes exponentially smaller and yields an exponential accuracy improvement over retraining from scratch and differentially private baselines, whereas when $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$7, retraining from scratch is optimal (Regehr et al., 1 Jun 2026).

6. Empirical trade-offs, failure modes, and recurrent debates

Across the literature, evaluation remains heterogeneous. LLM work often measures Forget Quality, Model Utility, Extraction Strength, Gibberish, and ROUGE-L on TOFU and MUSE; federated work uses Normalized Test Accuracy, Normalized Forgetting Score, Membership Inference Attack accuracy, Recovery Time Ratio, and Break Even Epochs; exact and semi-parametric methods compare against retrained or oracle references using deletion rate or prediction-gap metrics. This diversity reflects a genuine methodological split: some methods target indistinguishability from retraining, some target privacy leakage, some target targeted behavioral suppression, and some target utility preservation under bounded intervention (Cai et al., 29 Jan 2026, Balordi et al., 26 Aug 2025, Zheng et al., 24 Mar 2026).

Two recurring failure modes organize much of the design space. The first is over-forgetting: vanilla task-vector subtraction can strongly improve forgetting but sharply reduce model utility, and unrestricted gradient-difference unlearning can drive cross-entropy ascent into unstable weight and gradient growth. Per-parameter reweighting in PerTA and bounded adapter parameterizations in Stable Forgetting are both responses to this problem, though they act at different levels of granularity (Cai et al., 29 Jan 2026, Garg et al., 29 Sep 2025).

The second is scale under repeated or large deletion requests. Mini-Unlearning addresses high unlearning ratios by exploiting a contraction mapping between retrained and unlearned parameters, storing only the last $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$8 epochs of gradients and Hessian-vector products, with approximation error $$\theta_{\rm final}=\theta_{\rm full}+W\odot[-(\theta_{\rm fgt}-\theta_0)].$$9 and strong empirical performance at $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$0, $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$1, and $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$2 deletion ratios. Influence Approximation Unlearning (IAU) takes a first-order route: it replaces influence-function Hessian inversion with a single corrective update

$f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$3

and adds a Gradient Restriction loss during training. The paper reports utility loss within $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$4 of retraining and, on LeNet5/CIFAR10, unlearning in $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$5 seconds versus $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$6s for retraining and $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$7s for a Fisher method (Huang et al., 2024, Liu et al., 31 Jul 2025).

A common misconception is that parameter efficiency by itself implies strong unlearning guarantees. The literature does not support that simplification. Some PEFT methods are approximate and utility-oriented, some are exact by construction, and some are certified only under structural assumptions. The hybrid framework that switches between partial retraining and direct parameter update based on an acceptable workload threshold $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$8 makes this explicit: efficiency can be improved by $f_\text{oprt}(g_{\mathrm{f}},g_{\mathrm{r}})$9 to $\tau=1$0, but the framework distinguishes the high-accuracy retraining regime from the cheaper approximate regime rather than collapsing them into a single guarantee (Li et al., 2024).

Taken together, the field presents bounded parameter-efficient unlearning not as a settled recipe but as a set of increasingly precise answers to one question: how much of a model must be touched, and with what kind of formal control, to forget specific data while preserving everything else that matters.