SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

Abstract: Machine unlearning for LLMs aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.

• The paper introduces SHRED, a retain-set-free unlearning framework that uses token-local logit demotion via self-distillation to effectively remove memorized sensitive data.
• It employs autoregressive token probability for high-information token selection and demonstrates superior forget-utility tradeoffs on four LLM benchmarks.
• The paper validates SHRED's robustness against membership inference and relearning attacks while preserving model utility with efficient training.

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

Motivation and Problem Statement

The proliferation of LLMs has exacerbated challenges around selective knowledge removal, notably for privacy-sensitive, copyrighted, or hazardous content. Conventional unlearning protocols employ a retain set—a carefully curated auxiliary dataset to stabilize utility during content removal. Such reliance hampers scalability and introduces domain bias. "SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion" (2605.07482) eliminates the retain-set dependency, proposing a method that operates solely on the targeted forget set. The central technical insight is that memorized instance knowledge is token-local: high-information tokens encode memorized content, while remaining tokens anchor generic linguistic or commonsense structures.

Methodology: Logit Demotion with Self-Distillation

SHRED adopts a two-stage pipeline: (1) Information-dense token selection and (2) selective logit demotion via self-distillation.

Token Selection

SHRED leverages autoregressive token probabilities from the memorized model. Tokens with relatively low conditional probability (high Shannon information) are empirically shown to comprise the critical memorized fragments (names, specific facts, uncommon relations) for the forget set (Figure 1).

Figure 1: Autoregressive token probability across six unlearning scenarios; information-dense tokens (outlined) are precisely those carrying forget-set knowledge.

Self-Distillation Loss Construction

For each forget instance, SHRED computes a teacher distribution using the model's own output logits. A demotion mask is applied selectively to high-surprisal positions, setting token logits (and optionally, those of paraphrastic alternatives) to $-\infty$, zeroing their softmax probability. At benign (low-information) positions, teacher logits are kept unchanged, forming implicit anchors for language structure and general-purpose knowledge. The student model is trained via a KL-divergence objective restricted to the teacher's top-$K$ vocabulary indices (Figure 2).

Figure 2: SHRED training objective; selective logit demotion at forget positions and standard self-distillation at retain-anchor positions.

Variants include demoting only the memorized token versus nucleus demotion (top-$p$ probability mass), enabling control over the breadth of knowledge removed.

Empirical Evaluation

SHRED is validated on four prominent LLM unlearning benchmarks: TOFU [maini2024tofu], MUSE [shi2024muse], RWKU [jin2024rwku], and Hubble [wei2025hubble], using established metrics for both content removal and utility preservation (e.g., probability assigned to forget-set answers, ROUGE-L overlap, MMLU accuracy, and membership inference leakage).

Pareto Frontier Advancement

Empirical results demonstrate that SHRED achieves a strictly superior Pareto frontier in the forget-utility space—higher utility at matched or stronger forgetting compared to retain-set-dependent baselines (Figure 3).

Figure 3: Pareto frontiers for forget-set memorization versus model utility; SHRED shifts the frontier outward across four benchmarks.

SHRED matches retrained oracle models in knowledge removal without inducing over-forgetting (utility collapse), and utility remains robust across a wide $P$ sweep of demoted token fractions.

Robustness and Stability

SHRED exhibits resilience against membership inference attacks, maintaining privacy leakage scores close to ideal retrain targets. It further resists rapid relearning attacks and tolerates sequential, continual unlearning with only gradual utility degradation. Overfitting and catastrophic forgetting are mitigated by the self-distillation anchoring at non-demoted positions, resulting in stable utility across long overtraining horizons.

Efficiency and Practicality

SHRED’s effectiveness persists under full-fine-tuning and 8-bit optimizer quantization. However, LoRA-based adapter fine-tuning yields notable utility degradation, indicating the necessity for full-rank parameter updates in this self-distillation context.

WMDP-Cyber Domain Analysis

On domain-wide benchmarks such as WMDP-Cyber [li2024wmdp], SHRED provides milder tradeoff curves: accuracy reductions are smaller for corpus-targeted removal compared to broad retain-set-free methods (Figure 4). The corpus-question alignment analysis clarifies that most MCQA questions probe domain expertise rather than corpus-specific memorization.

Figure 4: WMDP-Cyber accuracy vs. MMLU; SHRED achieves closer proximity to full-model performance at matched cyber accuracy, highlighting the corpus-question mismatch.

Hyperparameter Sensitivity

The demote percentage $P$ is identified as the critical tradeoff knob: larger $P$ yields deeper forgetting but at higher utility cost. Small batch sizes are empirically optimal, preserving utility and facilitating sharp, per-instance gradient updates.

Practical and Theoretical Implications

SHRED eliminates retain-set-related operational complexities and deployment bias, streamlining compliance workflows for content removal (copyright takedowns, privacy requests, hazardous knowledge). Theoretically, the approach demonstrates that preservation of model utility does not require explicit retain sets, provided judicious self-distillation anchors exist.

Model-editing and continual learning stand to benefit from the SHRED paradigm, as it further evidences the efficacy of self-distillation in controlling catastrophic forgetting and modular knowledge modification.

Conclusion

SHRED offers a retain-set-free, self-distillation-based framework for LLM unlearning, achieving Pareto-optimal tradeoffs across several benchmarks with demonstrated robustness, efficiency, and practical utility. Its methodology—token-local logit demotion anchored by implicit self-distillation retain signals—positions it as a benchmark technique for scalable, controlled knowledge removal, and sets the stage for the next generation of unlearning protocols in AI systems.