Ssiuu: Robust Machine Unlearning in LLMs

• Ssiuu is an attribution-guided machine unlearning method that redefines unlearning by effectively erasing targeted knowledge in LLMs.
• It employs a regularization term to constrain negative attributions and mitigate spurious unlearning neurons that mask rather than remove information.
• Experimental results on Llama-3.2 and Qwen-2.5 demonstrate that Ssiuu maintains performance on non-target data and resists knowledge recovery under retraining.

Searching arXiv for the specified paper to ground the article and citation. Ssiuu, short for Suppressing Spurious Unlearning Neurons for Robust Unlearning, is a machine unlearning method for LLMs introduced to address the failure mode in which target knowledge is not actually erased, but instead hidden behind newly induced suppressive behavior (Yang et al., 26 Sep 2025). The method is motivated by the observation that many existing unlearning approaches achieve low forget-set accuracy under standard evaluation while remaining vulnerable to later retraining, at which point a substantial portion of the supposedly forgotten knowledge can resurface. Within this framework, Ssiuu is presented as an attribution-guided unlearning method that seeks faithful erasure of a forget set $\mathcal{C}_f$ while preserving retention and utility on non-target data, with particular emphasis on robustness under both adversarial and benign retraining scenarios (Yang et al., 26 Sep 2025).

1. Problem setting and motivation

Machine unlearning for LLMs concerns the removal of a forget set $\mathcal{C}_f$ from a model while preserving retention and utility on other data (Yang et al., 26 Sep 2025). The problem is especially salient because pretrained LLMs may memorize sensitive or private information, creating privacy and safety risks in deployment settings that include open-source models, fine-tuning APIs, and user-customized downstream training (Yang et al., 26 Sep 2025).

A central claim associated with Ssiuu is that standard forget-set evaluation can be misleading. Existing methods may appear successful because the model no longer reveals the target knowledge under the original prompting setup, yet the underlying knowledge can remain latent and be recovered after subsequent training (Yang et al., 26 Sep 2025). This establishes robust unlearning as a stricter requirement than benchmark-level suppression alone. The paper frames this as a deployment issue rather than merely an evaluation artifact: if forgotten knowledge can be recovered by later tuning, then the unlearning procedure has not delivered reliable removal in practical environments (Yang et al., 26 Sep 2025).

The paper further argues that this vulnerability matters across multiple application categories, including privacy-sensitive user data, copyright removal, harmful knowledge mitigation, open-source model release, and APIs that allow user adaptation (Yang et al., 26 Sep 2025). This suggests that the technical objective of unlearning cannot be separated from post-deployment adaptation dynamics.

2. Shallow alignment and spurious unlearning neurons

The conceptual core of Ssiuu is the distinction between erasing knowledge and hiding knowledge. The paper characterizes many existing unlearning methods as producing shallow alignment, a condition in which the target knowledge is not removed from the model’s weights or representations, but is instead masked by newly learned internal behavior (Yang et al., 26 Sep 2025). Under this interpretation, apparent unlearning reflects a change in expression rather than removal.

The proposed mechanism behind shallow alignment is the emergence of spurious unlearning neurons (Yang et al., 26 Sep 2025). These are described as new neurons or attribution patterns that do not erase the original knowledge, but instead generate negative influence that suppresses the output associated with that knowledge. The model therefore retains the original knowledge-bearing neurons while adding counteracting components. If later retraining weakens those suppressive components, the original knowledge can reappear (Yang et al., 26 Sep 2025).

This diagnosis changes the interpretation of unlearning failure. Rather than viewing relearning solely as reacquisition from new data, the paper treats some post-unlearning recovery as re-exposure of latent knowledge that was never removed in the first place (Yang et al., 26 Sep 2025). A plausible implication is that robustness to retraining functions as an operational test for whether unlearning has altered the underlying representation, rather than merely the model’s immediate output behavior.

Figure-based evidence in the paper reinforces this interpretation. Figure 1 illustrates shallow unlearning alignment with red neurons representing spurious unlearning neurons that hide knowledge rather than erase it, thereby enabling later resurfacing (Yang et al., 26 Sep 2025). Figure 2 shows that unlearned models can recover forgotten knowledge after benign or harmful retraining, linking the internal diagnosis to empirical vulnerability (Yang et al., 26 Sep 2025).

3. Attribution-based characterization

Ssiuu relies on an attribution-based analysis to determine whether target knowledge has been removed or merely suppressed (Yang et al., 26 Sep 2025). For a neuron $i$ and token position $k$, the attribution score is defined as

$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$

where $h_{\theta_i,k}$ is the $k$-th token representation of neuron $i$, and $$\frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$ measures how that neuron affects the output probability (Yang et al., 26 Sep 2025). Positive attribution, $A^{(x,y)}_{\theta_i,k} > 0$, denotes positive influence on the target output, whereas negative attribution, $\mathcal{C}_f$0, denotes negative influence (Yang et al., 26 Sep 2025).

To compare the model before unlearning, $\mathcal{C}_f$1, and after unlearning, $\mathcal{C}_f$2, the paper defines positive influence variation as

$\mathcal{C}_f$3

and defines negative influence variation $\mathcal{C}_f$4 analogously using $\mathcal{C}_f$5 instead of $\mathcal{C}_f$6 (Yang et al., 26 Sep 2025). It further introduces

$\mathcal{C}_f$7

to ignore contradictory signs and focus on meaningful changes (Yang et al., 26 Sep 2025).

The key empirical finding from this analysis is that, for many unlearning methods, negative influence increases more than positive influence decreases (Yang et al., 26 Sep 2025). The paper treats this asymmetry as the hallmark of shallow alignment: original positive knowledge pathways remain, while new negative pathways are learned to suppress them. Figure 3 visualizes this pattern by showing that negative influence variation is often larger than positive influence variation after unlearning (Yang et al., 26 Sep 2025).

The attribution analysis is also used diagnostically at the module and layer level. The paper reports that in GD, positive influence reduction is concentrated mainly in later layers and negative influence increases strongly, especially in attention $\mathcal{C}_f$8 and $\mathcal{C}_f$9 modules, whereas in Ssiuu, positive influence reduction is distributed across layers and modules while negative influence growth is suppressed (Yang et al., 26 Sep 2025). This supports the claim that Ssiuu specifically targets the formation of spurious unlearning neurons.

4. Objective and implementation

Ssiuu adds an attribution-guided regularization term to the unlearning objective in order to prevent the growth of spurious negative influence (Yang et al., 26 Sep 2025). Its core intuition is to constrain the negative attribution pattern to remain close to its pre-unlearning state. The intended effect is to reduce the positive influence tied to the forget set without inflating negative influence as a shortcut (Yang et al., 26 Sep 2025).

The optimization objective is written as

$i$0

where $i$1 is the base unlearning loss, $i$2 is the regularization weight, $i$3 and $i$4 are consecutive optimization steps, and $i$5 is the set of neuron indices with negative attribution scores (Yang et al., 26 Sep 2025). The regularizer penalizes changes in negative attributions, especially growth in negative influence. The paper explicitly states that the goal is not to destroy all negative attribution, because some negative influence may be part of normal language understanding; rather, the objective is to avoid adding extra negative influence that acts as a hiding mechanism (Yang et al., 26 Sep 2025).

For efficiency, attribution is approximated by multiplying parameters with gradients, and gradients are stopped through the previous-step attribution term $i$6 (Yang et al., 26 Sep 2025). Ssiuu is built on GD (Gradient Difference) as the backbone unlearning loss, although the method is presented as generally applicable (Yang et al., 26 Sep 2025).

The paper also reports an appendix analysis on the regularization weight $i$7. The described pattern is that too small a $i$8 yields weak robustness, too large a $i$9 hurts retention and convergence, and intermediate values work best (Yang et al., 26 Sep 2025). This suggests that Ssiuu operates by balancing suppression of spurious negative influence against preservation of optimization flexibility.

5. Experimental design and retraining protocols

The experimental setup spans two instruction-tuned model families, Llama-3.2 (3B) and Qwen-2.5 (3B), selected to reflect realistic deployment settings (Yang et al., 26 Sep 2025). Evaluation is conducted on two datasets. FaithUn contains real-world celebrity knowledge with 664 instances and serves as the main benchmark; its partitioning is forget set $k$0, retain set $k$1, and test set $k$2 (Yang et al., 26 Sep 2025). TOFU contains synthetic author-profile knowledge, is used to show generalization, requires pre-fine-tuning before unlearning, and uses forget set $k$3 (Yang et al., 26 Sep 2025).

The baseline methods are GA (Gradient Ascent), GD (Gradient Difference), DPO (Direct Preference Optimization), NPO (Negative Preference Optimization), RMU (Representation Misdirection/Randomization-based unlearning), and KLUE (Knowledge-Localized Unlearning), with Ssiuu instantiated on GD (Yang et al., 26 Sep 2025).

A major contribution of the work is its use of two retraining scenarios to test whether forgotten knowledge can reappear (Yang et al., 26 Sep 2025). The first is a harmful attack, or adversarial injection of private data, in which the unlearned model is fine-tuned on a small fraction of the forget set, with $k$4, and then evaluated on the remaining forgotten samples disjoint from the attack data (Yang et al., 26 Sep 2025). For FaithUn, only falsely answered instances are used for the attack pool because the benchmark treats chance-level answers as the stopping point (Yang et al., 26 Sep 2025).

The second is a benign attack using Alpaca, consisting of 1,000 instruction-following examples with learning rates $k$5 (Yang et al., 26 Sep 2025). This simulates ordinary downstream fine-tuning rather than explicit adversarial recovery. In the paper’s framing, resurfacing under this condition demonstrates fragility even in non-malicious adaptation settings (Yang et al., 26 Sep 2025).

6. Empirical results and diagnostic evidence

On FaithUn, the paper reports that forget-set accuracy is driven to $k$6 for all methods, but emphasizes that this alone is misleading because retraining reveals substantial recovery for weaker methods (Yang et al., 26 Sep 2025). For Llama-3.2, the reported harmful-attack and benign-attack accuracies are as follows:

Method	Harmful attack	Benign attack
GA	$k$7 / $k$8	$k$9
GD	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$0 / $$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$1	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$2
DPO	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$3 / $$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$4	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$5
NPO	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$6 / $$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$7	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$8
RMU	$$A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$$9 / $h_{\theta_i,k}$0	$h_{\theta_i,k}$1
KLUE	$h_{\theta_i,k}$2 / $h_{\theta_i,k}$3	$h_{\theta_i,k}$4
SSiuu	$h_{\theta_i,k}$5 / $h_{\theta_i,k}$6	$h_{\theta_i,k}$7

For Qwen-2.5, the paper states that Ssiuu again performs best on robustness, with harmful attack $h_{\theta_i,k}$8 / $h_{\theta_i,k}$9 and benign attack $k$0 (Yang et al., 26 Sep 2025).

On TOFU, Ssiuu is reported to achieve FS $k$1, RS $k$2, US $k$3, harmful attack $k$4 / $k$5, and benign attack $k$6 (Yang et al., 26 Sep 2025). The stated comparative interpretation is that these results indicate good retention and utility together with improved resistance to relearning (Yang et al., 26 Sep 2025).

The paper’s broader quantitative conclusion is that standard unlearning metrics can look good while robustness remains poor, and that Ssiuu is superior because it reduces recovery after retraining rather than only lowering forget accuracy in the initial evaluation (Yang et al., 26 Sep 2025). This conclusion is consistent with the broader conceptual claim that faithful unlearning must be judged by retraining resistance.

Several diagnostic analyses supplement the benchmark results. Using the logit lens, the authors report that GD often drives accuracy below chance in certain layers, indicating excessive unlearning and unstable internal representations, whereas Ssiuu brings the model to approximately chance-level behavior more cleanly, which is interpreted as faithful forgetting without over-destruction (Yang et al., 26 Sep 2025). After harmful retraining, attribution distributions under GD are reported as unstable and weakly correlated with the pre-attack state, while Ssiuu remains much more consistent: GD has correlation around $k$7, NPO around $k$8, and Ssiuu about $k$9 (Yang et al., 26 Sep 2025). The paper presents this as the strongest attribution stability among the compared methods.

7. Significance, evaluation criteria, and deployment implications

The principal significance of Ssiuu lies in its reformulation of unlearning quality. The paper argues that existing unlearning methods often do not truly erase knowledge, that this makes them vulnerable to relearning, and that robust unlearning must therefore be judged by retraining resistance rather than forget-set performance alone (Yang et al., 26 Sep 2025). Within that argument, Ssiuu is not only a new optimization term but also an evaluation stance: unlearning should be tested against downstream perturbations that can expose hidden knowledge.

This position also clarifies a common misconception. A model that answers forget-set prompts incorrectly after unlearning is not necessarily one that has forgotten the target knowledge in a robust sense (Yang et al., 26 Sep 2025). If subsequent retraining can recover that knowledge, the original unlearning may have been shallow alignment rather than erasure. The distinction is technically important because it shifts attention from output-level suppression to representation-level change.

The deployment implications given in the paper are explicitly safety-oriented. Safe deployment of LLMs requires unlearning that actually removes information from internal representations, not just suppresses it temporarily, and that remains robust under downstream fine-tuning (Yang et al., 26 Sep 2025). The domains named in this context are privacy-sensitive user data, copyright removal, harmful knowledge mitigation, open-source model release, and APIs that permit user adaptation (Yang et al., 26 Sep 2025).

Taken together, these claims position Ssiuu as an attribution-guided approach to faithful erasure that explicitly targets the failure mode of spurious unlearning neurons (Yang et al., 26 Sep 2025). The paper’s central contribution is therefore twofold: it identifies shallow alignment as a structural explanation for relearning vulnerability, and it proposes a regularized objective designed to suppress the negative-attribution growth that enables hidden knowledge to re-emerge.