Selective Representation Misdirection for Unlearning (SRMU)

• SRMU is a principled framework for machine unlearning that selectively perturbs latent features to erase specified data influence while preserving core model functionality.
• It employs dynamic importance mapping to identify forget-relevant dimensions and applies directional misdirection to minimize collateral loss on retained knowledge.
• Empirical evaluations demonstrate that SRMU achieves superior forgetting of sensitive data with minimal utility degradation, outperforming conventional unlearning methods.

Selective Representation Misdirection for Unlearning (SRMU) is a principled framework for machine unlearning that operates via structured, feature-aware perturbations of neural activations. SRMU targets scenarios where the distributions of data to be "forgotten" and "retained" are highly entangled in feature space, a setting where conventional perturbation-based unlearning techniques experience substantial performance degradation. The approach leverages a dynamic importance map to localize unlearning efforts to latent dimensions most associated with the forget set and applies a directionally consistent misdirection to those features. SRMU is designed for computational efficiency, scalability, and robust retention of benign knowledge while mitigating the influence of sensitive or hazardous content in pretrained models, particularly LLMs (Chen et al., 18 Dec 2025).

1. Problem Setup and Motivation

In practical model governance and sensitive-knowledge removal, the challenge is to erase the influence of a specified forget set $\mathcal{D}_f$ from a deployed model $M$ with parameters $\theta_0$ while preserving the model's utility on a retain set $\mathcal{D}_r$. Unlike protocols that assume disjoint, non-overlapping support between $\mathcal{D}_f$ and $\mathcal{D}_r$, SRMU is intended for high-entropy, operational data regimes exhibiting substantial entanglement (e.g., 20–30% unigram feature overlap observed on WMDP Bio/Cyber benchmarks). The forget and retain sets are thus entangled in the activation space, making selective, minimally destructive unlearning feasible only through targeted intervention.

SRMU addresses this scenario by editing a chosen intermediate (typically MLP) layer $\ell$, where the hidden activation $H_\theta(x) \in \mathbb{R}^d$ provides a structured locus for representation perturbation. The aim is to selectively suppress those features most implicated in the forget set, while maintaining the integrity of features crucial to the retain set (Chen et al., 18 Dec 2025).

2. Methodological Framework

2.1 Dynamic Importance Mapping

SRMU computes two key statistics at layer $\ell$:

• $$v_f = \mathbb{E}_{x_f \sim \mathcal{D}_f} \left[ H_{\theta_0}(x_f) \right]$$, the mean activation for the forget set.
• $$v_r = \mathbb{E}_{x_r \sim \mathcal{D}_r} \left[ H_{\theta_0}(x_r) \right]$$, the mean activation for the retain set.

The raw importance vector $A \in \mathbb{R}^d$ is fusioned via rules such as:

• Ratio: $$A_i = \log\left(1 + \frac{v_{f,i}}{v_{r,i} + \epsilon}\right)$$,
• Difference: $A_i = \mathrm{ReLU}(v_{f,i} - \lambda v_{r,i})$,
• Product: $$A_i = \frac{v_{f,i} v_{r,i}}{\mathrm{mean}(v_f)\mathrm{mean}(v_r) + \epsilon}$$.

This is normalized to $A_\mathrm{norm} \in [0,1]^d$ by dividing by the maximum entry, yielding a channel-wise importance mask for feature selectivity.

2.2 Directional Misdirection

A misdirection vector $v \in \{-1, +1\}^d$ is sampled and fixed for all optimization steps, providing each feature directionality—consistent push or pull—rather than stochastic Gaussian noise.

A misdirection target for forget samples is defined as:

$$T_\mathrm{misdir}(x_f) = c_\mathrm{map} \cdot (v \odot A_\mathrm{norm}),$$

where $c_\mathrm{map} > 0$ controls perturbation magnitude.

2.3 Loss Function

SRMU updates only the weights of layer $\ell$, optimizing the following objective:

$$\mathcal{L}_\mathrm{SRMU}(\theta) = \mathbb{E}_{x_f \sim \mathcal{D}_f} \left[ \| H_\theta(x_f) - T_\mathrm{misdir}(x_f) \|_2^2 \right] + \alpha \cdot \mathbb{E}_{x_r \sim \mathcal{D}_r} \left[ \| H_\theta(x_r) - H_{\theta_0}(x_r) \|_2^2 \right]$$

with $\alpha \gtrsim 10^3$, preserving retain-set activations as a strong regularization anchor.

3. Algorithmic Mechanism and Key Distinctions

SRMU proceeds in four steps:

1. Compute $v_f$, $v_r$, $A_\mathrm{norm}$ from reference activations.
2. Sample and fix $v \in \{-1,+1\}^d$.
3. For $T$ unlearning steps, use mini-batches from $\mathcal{D}_f$ and $\mathcal{D}_r$ to:
   • Calculate activations and losses as above.
   • Backpropagate $\nabla_{\theta_\ell}(\mathcal{L}_\mathrm{SRMU})$ and update only $\theta_\ell$ parameters.
4. Return updated model.

Essential innovations:

• Feature selectivity by way of $A_\mathrm{norm}$, localizing model edits to forget-relevant dimensions.
• Directional, structured activation perturbation, contrasting with unstructured approaches (e.g., random misdirection).
• Explicit utility regularization on $\mathcal{D}_r$ examples prevents utility collapse.

Ablation studies demonstrate almost no forgetting occurs if both $v$ and $A_\mathrm{norm}$ are omitted, while purely non-selective (uniform) perturbation results in drastic utility loss (Chen et al., 18 Dec 2025).

4. Theoretical Properties

SRMU's theoretical rationale is rooted in convex optimization:

• The combined least-squares plus strong retain anchor ensures that the algorithm finds an optimal balance between effective misdirection of forget activations and minimal collateral change to retain activations.
• The selective importance mask ensures that updates remain confined to the principal forget subspace, controlling the risk of unintentional erasure of benign knowledge.
• The approach provably achieves $\|\Delta H(x_r)\|$ lower than unstructured methods at a fixed $\|\Delta H(x_f)\|$, quantifying collateral damage and affirming safe unlearning in linear regime (Chen et al., 18 Dec 2025).

5. Empirical Evaluation

5.1 Datasets and Regimes

SRMU is validated on the WMDP Bio and Cyber benchmarks (hazardous knowledge, high feature entanglement). Low- and high-entanglement regimes are defined by unigram/bigram overlaps—from $\leq$5% up to 27.5% (Cyber, high).

5.2 Forgetting vs. Utility Trade-off

The trade-off is assessed using:

• WMDP Accuracy (forgetting metric, lower is better)
• MMLU Accuracy (utility metric, higher is better)

Method	MMLU ↑	WMDP-Bio ↓	WMDP-Cyber ↓	WMDP Avg ↓ (Low Ent.)	MMLU ↑	WMDP-Bio ↓	WMDP-Cyber ↓	WMDP Avg ↓ (High Ent.)
Original	58.5	64.7	44.8	54.7	58.5	64.7	44.8	54.7
RMU	56.9	28.8	28.0	28.4	51.9	48.5	41.1	44.8
Adaptive RMU	55.0	25.3	26.7	26.0	51.2	49.3	37.7	43.5
SRMU (Ours)	57.1	28.5	25.8	27.2	52.5	38.3	37.1	37.7

SRMU delivers state-of-the-art forgetting with minimal utility loss. Under high entanglement (20–30% overlap), existing baselines collapse (WMDP Avg >43%), whereas SRMU preserves both utility and superior unlearning (WMDP Avg 37.7%, MMLU 52.5%) (Chen et al., 18 Dec 2025).

5.3 Ablation and Variant Analysis

Ablation studies confirm that ratio-fusion in $A_\mathrm{norm}$ offers the best Pareto trade-off. Removing selectivity or directionality results in loss of unlearning or utility collapse.

6. Extensions and Related Work

SRMU instantiates a general principle of selective representation misdirection, sharing conceptual kinship with recent approaches employing linear subspace redirection (LUNAR (Shen et al., 11 Feb 2025)), mask-based subspace redirection (REM (Schoepf et al., 23 May 2025)), federated variants (REMISVFU (Wu et al., 11 Dec 2025)), and sparse autoencoder-based projection (SSPU (Wang et al., 30 May 2025)). Across these, localizing model updates to feature dimensions strongly associated with the forget set recurrently emerges as a key for robust, efficacious, and minimally damaging unlearning, particularly in challenging, overlap-rich data regimes.

7. Significance and Practical Implications

SRMU furnishes a practical, computationally efficient foundation for safety-driven model governance and privacy compliance by allowing explicit, controllable removal of sensitive or high-risk knowledge. It is robust to nonideal, real-world data scenarios, scales to large models via single-layer updates, and avoids the catastrophic utility drop-offs typical of unstructured perturbation baselines. SRMU’s explicit retain-anchor and selective intervention yield reliable compliance even under adversarial or entangled conditions (Chen et al., 18 Dec 2025).