---
title: 'Ssiuu: Robust Machine Unlearning in LLMs'
url: https://www.emergentmind.com/topics/ssiuu
type: topic
---

# Ssiuu: Robust Machine Unlearning in LLMs

Searching arXiv for the specified paper to ground the article and citation.
Ssiuu, short for **Suppressing Spurious Unlearning Neurons for Robust Unlearning**, is a machine unlearning method for large language models (LLMs) introduced to address the failure mode in which target knowledge is not actually erased, but instead hidden behind newly induced suppressive behavior [2509.22263]. The method is motivated by the observation that many existing unlearning approaches achieve low forget-set accuracy under standard evaluation while remaining vulnerable to later retraining, at which point a substantial portion of the supposedly forgotten knowledge can resurface. Within this framework, Ssiuu is presented as an attribution-guided unlearning method that seeks faithful erasure of a forget set $\mathcal{C}_f$ while preserving retention and utility on non-target data, with particular emphasis on robustness under both adversarial and benign retraining scenarios [2509.22263].

## 1. Problem setting and motivation

Machine unlearning for LLMs concerns the removal of a forget set $\mathcal{C}_f$ from a model while preserving retention and utility on other data [2509.22263]. The problem is especially salient because pretrained LLMs may memorize sensitive or private information, creating privacy and safety risks in deployment settings that include open-source models, fine-tuning APIs, and user-customized downstream training [2509.22263].

A central claim associated with Ssiuu is that standard forget-set evaluation can be misleading. Existing methods may appear successful because the model no longer reveals the target knowledge under the original prompting setup, yet the underlying knowledge can remain latent and be recovered after subsequent training [2509.22263]. This establishes robust unlearning as a stricter requirement than benchmark-level suppression alone. The paper frames this as a deployment issue rather than merely an evaluation artifact: if forgotten knowledge can be recovered by later tuning, then the unlearning procedure has not delivered reliable removal in practical environments [2509.22263].

The paper further argues that this vulnerability matters across multiple application categories, including privacy-sensitive user data, copyright removal, harmful knowledge mitigation, open-source model release, and APIs that allow user adaptation [2509.22263]. This suggests that the technical objective of unlearning cannot be separated from post-deployment adaptation dynamics.

## 2. Shallow alignment and spurious unlearning neurons

The conceptual core of Ssiuu is the distinction between erasing knowledge and hiding knowledge. The paper characterizes many existing unlearning methods as producing **shallow alignment**, a condition in which the target knowledge is not removed from the model’s weights or representations, but is instead masked by newly learned internal behavior [2509.22263]. Under this interpretation, apparent unlearning reflects a change in expression rather than removal.

The proposed mechanism behind shallow alignment is the emergence of **spurious unlearning neurons** [2509.22263]. These are described as new neurons or attribution patterns that do not erase the original knowledge, but instead generate negative influence that suppresses the output associated with that knowledge. The model therefore retains the original knowledge-bearing neurons while adding counteracting components. If later retraining weakens those suppressive components, the original knowledge can reappear [2509.22263].

This diagnosis changes the interpretation of unlearning failure. Rather than viewing relearning solely as reacquisition from new data, the paper treats some post-unlearning recovery as re-exposure of latent knowledge that was never removed in the first place [2509.22263]. A plausible implication is that robustness to retraining functions as an operational test for whether unlearning has altered the underlying representation, rather than merely the model’s immediate output behavior.

Figure-based evidence in the paper reinforces this interpretation. Figure 1 illustrates shallow unlearning alignment with red neurons representing spurious unlearning neurons that hide knowledge rather than erase it, thereby enabling later resurfacing [2509.22263]. Figure 4 shows that unlearned models can recover forgotten knowledge after benign or harmful retraining, linking the internal diagnosis to empirical vulnerability [2509.22263].

## 3. Attribution-based characterization

Ssiuu relies on an attribution-based analysis to determine whether target knowledge has been removed or merely suppressed [2509.22263]. For a neuron $i$ and token position $k$, the attribution score is defined as

\[
A^{(x,y)}_{\theta_i,k} = h_{\theta_i,k}\times \frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}
\]

where $h_{\theta_i,k}$ is the $k$-th token representation of neuron $i$, and $\frac{\partial P_{\theta}(y|x)}{\partial h_{\theta_i,k}}$ measures how that neuron affects the output probability [2509.22263]. Positive attribution, $A^{(x,y)}_{\theta_i,k} > 0$, denotes positive influence on the target output, whereas negative attribution, $A^{(x,y)}_{\theta_i,k} < 0$, denotes negative influence [2509.22263].

To compare the model before unlearning, $\theta$, and after unlearning, $\theta'$, the paper defines positive influence variation as

\[
D^{+}_{i} = \frac{1}{n} \sum_{(x,y)\in \mathcal{C}_f} \left( \max_k A^{(x,y)}_{\theta_i,k} - \max_k A^{(x,y)}_{\theta'_i,k} \right)
\]

and defines negative influence variation $D^-_i$ analogously using $\min_k$ instead of $\max_k$ [2509.22263]. It further introduces

\[
\widetilde{D}^{+}_{i}=\max(D^{+}_{i},0), \qquad \widetilde{D}^{-}_{i}=\max(D^{-}_{i},0)
\]

to ignore contradictory signs and focus on meaningful changes [2509.22263].

The key empirical finding from this analysis is that, for many unlearning methods, negative influence increases more than positive influence decreases [2509.22263]. The paper treats this asymmetry as the hallmark of shallow alignment: original positive knowledge pathways remain, while new negative pathways are learned to suppress them. Figure 3 visualizes this pattern by showing that negative influence variation is often larger than positive influence variation after unlearning [2509.22263].

The attribution analysis is also used diagnostically at the module and layer level. The paper reports that in GD, positive influence reduction is concentrated mainly in later layers and negative influence increases strongly, especially in attention $Q$ and $K$ modules, whereas in Ssiuu, positive influence reduction is distributed across layers and modules while negative influence growth is suppressed [2509.22263]. This supports the claim that Ssiuu specifically targets the formation of spurious unlearning neurons.

## 4. Objective and implementation

Ssiuu adds an **attribution-guided regularization** term to the unlearning objective in order to prevent the growth of spurious negative influence [2509.22263]. Its core intuition is to constrain the negative attribution pattern to remain close to its pre-unlearning state. The intended effect is to reduce the positive influence tied to the forget set without inflating negative influence as a shortcut [2509.22263].

The optimization objective is written as

\[
\arg\min_{\theta^t} \mathcal{L}_{\theta^t} + \lambda \sum_{(x,y)\in \mathcal{C}_f} \sum_{i \in \mathcal{I}^{-} \left\| A^{(x,y)}_{\theta^{t-1}_i} - A^{(x,y)}_{\theta^{t}_i} \right\|_{2}
\]

where $\mathcal{L}_{\theta^t}$ is the base unlearning loss, $\lambda$ is the regularization weight, $\theta^{t-1}$ and $\theta^t$ are consecutive optimization steps, and $\mathcal{I}^{-}$ is the set of neuron indices with negative attribution scores [2509.22263]. The regularizer penalizes changes in negative attributions, especially growth in negative influence. The paper explicitly states that the goal is not to destroy all negative attribution, because some negative influence may be part of normal language understanding; rather, the objective is to avoid adding extra negative influence that acts as a hiding mechanism [2509.22263].

For efficiency, attribution is approximated by multiplying parameters with gradients, and gradients are stopped through the previous-step attribution term $A^{(x,y)}_{\theta^{t-1}_i}$ [2509.22263]. Ssiuu is built on **GD (Gradient Difference)** as the backbone unlearning loss, although the method is presented as generally applicable [2509.22263].

The paper also reports an appendix analysis on the regularization weight $\lambda$. The described pattern is that too small a $\lambda$ yields weak robustness, too large a $\lambda$ hurts retention and convergence, and intermediate values work best [2509.22263]. This suggests that Ssiuu operates by balancing suppression of spurious negative influence against preservation of optimization flexibility.

## 5. Experimental design and retraining protocols

The experimental setup spans two instruction-tuned model families, **Llama-3.2 (3B)** and **Qwen-2.5 (3B)**, selected to reflect realistic deployment settings [2509.22263]. Evaluation is conducted on two datasets. **FaithUn** contains real-world celebrity knowledge with 664 instances and serves as the main benchmark; its partitioning is forget set $= 5\%$, retain set $= 10\%$, and test set $= 70\%$ [2509.22263]. **TOFU** contains synthetic author-profile knowledge, is used to show generalization, requires pre-fine-tuning before unlearning, and uses forget set $= 1\%$ [2509.22263].

The baseline methods are **GA** (Gradient Ascent), **GD** (Gradient Difference), **DPO** (Direct Preference Optimization), **NPO** (Negative Preference Optimization), **RMU** (Representation Misdirection/Randomization-based unlearning), and **KLUE** (Knowledge-Localized Unlearning), with Ssiuu instantiated on GD [2509.22263].

A major contribution of the work is its use of two retraining scenarios to test whether forgotten knowledge can reappear [2509.22263]. The first is a harmful attack, or adversarial injection of private data, in which the unlearned model is fine-tuned on a small fraction of the forget set, with $p \in \{0.1, 0.3\}$, and then evaluated on the remaining forgotten samples disjoint from the attack data [2509.22263]. For FaithUn, only falsely answered instances are used for the attack pool because the benchmark treats chance-level answers as the stopping point [2509.22263].

The second is a benign attack using **Alpaca**, consisting of 1,000 instruction-following examples with learning rates $10^{-5}, 10^{-6}, 10^{-7}$ [2509.22263]. This simulates ordinary downstream fine-tuning rather than explicit adversarial recovery. In the paper’s framing, resurfacing under this condition demonstrates fragility even in non-malicious adaptation settings [2509.22263].

## 6. Empirical results and diagnostic evidence

On **FaithUn**, the paper reports that forget-set accuracy is driven to $0.0$ for all methods, but emphasizes that this alone is misleading because retraining reveals substantial recovery for weaker methods [2509.22263]. For **Llama-3.2**, the reported harmful-attack and benign-attack accuracies are as follows:

| Method | Harmful attack | Benign attack |
|---|---:|---:|
| GA | $68.42$ / $73.33$ | $16.71$ |
| GD | $48.13$ / $54.76$ | $33.33$ |
| DPO | $31.58$ / $46.67$ | $15.34$ |
| NPO | $18.33$ / $18.75$ | $18.62$ |
| RMU | $52.63$ / $75.53$ | $14.29$ |
| KLUE | $57.14$ / $62.96$ | $28.33$ |
| SSiuu | $14.81$ / $14.29$ | $13.33$ |

For **Qwen-2.5**, the paper states that Ssiuu again performs best on robustness, with harmful attack $4.76$ / $29.41$ and benign attack $13.04$ [2509.22263].

On **TOFU**, Ssiuu is reported to achieve FS $17.75$, RS $92.74$, US $91.67$, harmful attack $31.82$ / $37.53$, and benign attack $21.08$ [2509.22263]. The stated comparative interpretation is that these results indicate good retention and utility together with improved resistance to relearning [2509.22263].

The paper’s broader quantitative conclusion is that standard unlearning metrics can look good while robustness remains poor, and that Ssiuu is superior because it reduces recovery after retraining rather than only lowering forget accuracy in the initial evaluation [2509.22263]. This conclusion is consistent with the broader conceptual claim that faithful unlearning must be judged by retraining resistance.

Several diagnostic analyses supplement the benchmark results. Using the **logit lens**, the authors report that **GD** often drives accuracy below chance in certain layers, indicating excessive unlearning and unstable internal representations, whereas **Ssiuu** brings the model to approximately chance-level behavior more cleanly, which is interpreted as faithful forgetting without over-destruction [2509.22263]. After harmful retraining, attribution distributions under GD are reported as unstable and weakly correlated with the pre-attack state, while Ssiuu remains much more consistent: GD has correlation around $\rho = 0.73$, NPO around $\rho = 0.87$, and Ssiuu about $\rho = 0.99$ [2509.22263]. The paper presents this as the strongest attribution stability among the compared methods.

## 7. Significance, evaluation criteria, and deployment implications

The principal significance of Ssiuu lies in its reformulation of unlearning quality. The paper argues that existing unlearning methods often do not truly erase knowledge, that this makes them vulnerable to relearning, and that robust unlearning must therefore be judged by retraining resistance rather than forget-set performance alone [2509.22263]. Within that argument, Ssiuu is not only a new optimization term but also an evaluation stance: unlearning should be tested against downstream perturbations that can expose hidden knowledge.

This position also clarifies a common misconception. A model that answers forget-set prompts incorrectly after unlearning is not necessarily one that has forgotten the target knowledge in a robust sense [2509.22263]. If subsequent retraining can recover that knowledge, the original unlearning may have been shallow alignment rather than erasure. The distinction is technically important because it shifts attention from output-level suppression to representation-level change.

The deployment implications given in the paper are explicitly safety-oriented. Safe deployment of LLMs requires unlearning that actually removes information from internal representations, not just suppresses it temporarily, and that remains robust under downstream fine-tuning [2509.22263]. The domains named in this context are privacy-sensitive user data, copyright removal, harmful knowledge mitigation, open-source model release, and APIs that permit user adaptation [2509.22263].

Taken together, these claims position Ssiuu as an attribution-guided approach to faithful erasure that explicitly targets the failure mode of spurious unlearning neurons [2509.22263]. The paper’s central contribution is therefore twofold: it identifies shallow alignment as a structural explanation for relearning vulnerability, and it proposes a regularized objective designed to suppress the negative-attribution growth that enables hidden knowledge to re-emerge.

Source: https://www.emergentmind.com/topics/ssiuu