Exclusive Unlearning

Abstract: When introducing LLMs into industrial applications, such as healthcare and education, the risk of generating harmful content becomes a significant challenge. While existing machine unlearning methods can erase specific harmful knowledge and expressions, diverse harmful content makes comprehensive removal difficult. In this study, instead of individually listing targets for forgetting, we propose Exclusive Unlearning (EU), which aims for broad harm removal by extensively forgetting everything except for the knowledge and expressions we wish to retain. We demonstrate that through Exclusive Unlearning, it is possible to obtain a model that ensures safety against a wide range of inputs, including jailbreaks, while maintaining the ability to respond to diverse instructions related to specific domains such as medicine and mathematics.

• The paper presents a retention-centric unlearning framework that defines what to retain, systematically erasing harmful or adversarial content from language models.
• The methodology employs an entropy-maximizing loss on self-generated outputs alongside standard retention loss, achieving near-zero attack success rates on sophisticated jailbreak prompts.
• The framework demonstrates robust, domain-specific performance, outperforming traditional unlearning methods and offering safer deployment in high-stakes applications.

Exclusive Unlearning: A Paradigm Shift in Machine Unlearning for LLMs

Introduction

The deployment of LLMs in industrial applications such as healthcare and education is hindered by the risk of harmful, private, or copyright-infringing outputs. Traditional approaches to machine unlearning in LLMs attempt to enumerate and erase specific harmful items from models, but the intrinsic diversity and unpredictability of adversarial, especially jailbreak, attacks renders exhaustive harmful data curation infeasible. "Exclusive Unlearning" (EU) (2604.06154) reframes this problem: instead of specifying "what to forget," it introduces a retention-centric perspective, specifying only "what to retain" and systematically forgetting all other content—harmful or otherwise.

Figure 1: Exclusive Unlearning circumvents the enumeration bottleneck of traditional unlearning by exclusively defining what knowledge to keep and forgetting all remaining model capabilities.

Methodology

The EU framework operationalizes a simple, entropy-maximizing unlearning objective. Rather than curating a diverse forget set, EU exclusively fine-tunes retention data for tasks to preserve (e.g., medical or mathematical instruction datasets) while maximizing the entropy of model outputs on its own generative distribution over all other text. Concretely, the model samples its own outputs, and the distribution of predicted tokens is pushed toward uniformity on non-retained content, effectively ensuring that all non-target knowledge is unlearned.

This is formalized as a joint loss:

$$\mathcal{L}(\theta) = \lambda \cdot \mathcal{L}_\text{forget}(\theta) + (1-\lambda) \cdot \mathcal{L}_\text{retain}(\theta)$$

where $\mathcal{L}_\text{forget}$ is the entropy-maximizing (uniformity) loss on self-generated outputs and $\mathcal{L}_\text{retain}$ is the standard negative log-likelihood on retention data. The balancing parameter $\lambda$ controls the compromise between forgetting and retaining, and extensive ablations demonstrate robustness to its setting.

Experimental Evaluation

Retention and Safety on Domain Tasks

EU is evaluated using Llama-3 and OLMo-2 models in medical and mathematical domains. Retention sets are comprised of domain-specific instruction-tuning datasets (e.g., MedInstruct-52k for medicine, MetaMathQA for mathematics). For each domain, both benchmark (multiple-choice, summarization) and LLM-as-a-judge evaluations are conducted. As shown in summary results:

• EU retains domain-specific ability to a degree comparable to or surpassing standard domain fine-tuning (RetainOnlyFT)
• On tasks involving diverse, curated harmful and sophisticated jailbreak prompts (from datasets such as GPTFuzzer, WildAttack, and others), EU reduces Attack Success Rate (ASR) to nearly zero, outperforming standard unlearning, Direct Preference Optimization (DPO), and advanced enumerate-and-generalize baselines (Eraser, SKU).

Judge-based evaluation using GPT-5 over free-form medical queries shows that EU matches or slightly trails RetainOnlyFT in terms of accuracy, utility, comprehensibility, and succinctness—a demonstration that retention is preserved despite aggressive general forgetting.

Robustness Against Jailbreaks and Unseen Harm

Baselines relying on harmful data enumeration (standard unlearning, DPO, Eraser, SKU) achieve low ASR on explicitly listed harmful data, but their generalization to compositional adversarial attacks (e.g., jailbreaks) remains substantially weaker than EU. In contrast, EU achieves high-fidelity forgetting even for unseen attack templates, demonstrating superior resilience and domain specialization.

Representation Analysis

t-SNE visualizations of model representations at the last-token position illustrate the mechanism of retention and forgetting. In EU-trained models, only inputs semantically or structurally proximal to the retained domain (e.g., medical queries) are preserved, while general, harmful, or cleverly crafted jailbreak prompts—regardless of surface similarity—are effectively forgotten.

Figure 2: EU compacts the representation of retained medical content and uniformly forgets both harmful and adversarially shifted (jailbreak) prompts.

Figure 3: Similar effects in OLMo-2-1B, showing that the method generalizes across architectures and domains.

Analysis of Self-Generated Forget Sets

A critical finding is that using model self-generated text as forget data yields more robust forgetting than using large external corpora (e.g., Wikipedia or The Pile). The diversity of a model's own output distribution ensures internal capabilities (including adversarial capacities) are exposed and forgotten, preventing the retention of rare but dangerous responses not present in external benchmarks.

Figure 4: Forgetting performance is strongest (lowest attack success rate) when self-generated data is used for unlearning compared to external corpora.

Hyperparameter and Design Sensitivity

• Choice of $\lambda$: Both retention quality and defensive forgetting are stable across broad $\lambda$ ranges.
• Self-generation diversity: Effective forgetting requires sufficiently diverse self-generated forget data, not overly deterministic nor excessively unconstrained.
• Pre-training dependency: EU is not a replacement for pretraining. Training from scratch on retention data yields unusable models; foundational linguistic and world knowledge must exist for EU to function effectively.
• Retention-forgetting trade-off: As a natural consequence of exclusive forgetting, all non-retained, including benign, capabilities are degraded—beneficial for strict, domain-specialized deployments where extra capabilities are potential liability.

Limitations

• Fine-tuning vulnerability: EU, being gradient-based, does not guarantee irreversibility; subsequent fine-tuning may re-enable previously forgotten harmful capabilities.
• Domain specificity: Application to general-purpose LLMs is limited, as EU removes all knowledge outside the explicit retention set.
• Retention of related benign content: Some general-domain prompts closely aligned (e.g., health-related general knowledge) may escape forgetting, as demonstrated in retained but semantically adjacent examples.

Implications and Future Work

EU recasts unlearning as a retention-centric paradigm, offering a scalable, data-efficient alternative in high-stakes domain applications (e.g., clinical, educational, financial) where exhaustive specification of harms is unattainable and safety requirements are strict. The exclusive approach addresses the fundamental limitation of enumeration-dependent unlearning—generalization to adversarial and unforeseen attacks—while providing a practical implementation framework.

However, robust parameter removal and theoretical guarantees against recoverability after further fine-tuning remain open research directions. Formal analysis of which semantic features can be inadvertently retained due to retention domain overlap, and the extension of the method toward dynamic or interactive domain expansion scenarios, are critical avenues for future work.

Conclusion

Exclusive Unlearning redefines the problem of machine unlearning for LLM safety by specifying only the retained domain and maximizing entropy elsewhere, thus achieving comprehensive and robust suppression of both known and unknown adversarial capabilities. This principle is empirically demonstrated to outperform enumerate-and-generalize methodologies in resisting jailbreak and harmful input attacks, provided the deployment context is that of a specialized, un-modified model. The EU framework is poised to form a foundational approach for domain-specific, safety-critical LLM applications requiring strict, provably limited model behavior.