---
title: 'LUNE: LoRA-based Unlearning with Negative Examples'
url: https://www.emergentmind.com/topics/lora-based-unlearning-with-negative-examples-lune
type: topic
---

# LUNE: LoRA-based Unlearning with Negative Examples

LoRA-based Unlearning with Negative Examples (LUNE) is a framework for targeted and efficient model unlearning that leverages Low-Rank Adaptation (LoRA) modules and negative example-driven supervision. LUNE aims to remove specific knowledge or behaviors from neural models—such as large language models (LLMs) or medical image predictors—without incurring the computational and generalization costs of full model retraining or direct backbone weight editing. By confining trainable updates to a low-dimensional subspace and employing adversarial or negative-only fine-tuning, LUNE achieves controlled knowledge removal while minimizing collateral utility loss on retained capabilities [2512.07375][2511.16574].

## 1. Motivation and Background

In both vision and language settings, neural models accumulate extensive domain and factual knowledge from large datasets. Regulatory and practical requirements—including privacy compliance, bias mitigation, and continual dataset revisions—increasingly demand mechanisms to remove or update specific content post-training. Prior approaches, such as full model retraining on curated datasets (excluding or down-weighting the target knowledge), or direct memory-editing of the model weights, entail prohibitive computational costs and can induce “catastrophic unlearning” by globally degrading performance on unrelated tasks.

LUNE addresses this gap by introducing a selective, lightweight approach that (i) restricts updates to LoRA adapters while freezing the original backbone, and (ii) guides those updates using negative examples—inputs paired with outputs that explicitly refute, contradict, or replace the target knowledge. This design produces localized edits in parameter space and targets the erasure of undesired content with high sample and compute efficiency [2512.07375].

## 2. Core Methodology and Architectural Design

LUNE adopts a parameter-efficient fine-tuning paradigm based on Low-Rank Adaptation (LoRA). LoRA introduces lightweight, trainable matrices into the structure (e.g., attention and feed-forward modules for LLMs; convolutional decoders in segmentation nets), leaving the bulk of pre-trained weights fixed. For a base weight matrix $W_0 \in \mathbb{R}^{d_{out} \times d_{in}}$, the adapted version is $W' = W_0 + AB^\top$, where $A \in \mathbb{R}^{d_{out}\times r}$ and $B \in \mathbb{R}^{d_{in}\times r}$, and $r \ll \min(d_{out}, d_{in})$.

### Integration in LLMs and Medical Models

- **LLMs:** LoRA adapters are inserted into query/key/value/output projections and feed-forward submodules of each Transformer block. All original model parameters $\theta$ are frozen, and only the LoRA parameters $\phi = \{A, B\}$ are updated using negative example supervision.
- **Medical Image Segmentation:** The LoRA adapters are applied to decoder convolution layers and the segmentation head. A teacher-student framework is used, freezing the full-capacity teacher and its weights or features as reference for the student’s updates [2511.16574].

A *strong unlearning* phase adversarially updates LoRA parameters to suppress, contradict, or increase uncertainty on the forget set, followed by a *gentle restoration* phase that recovers generalization on retained data by updating only the final head.

## 3. Negative Example Construction and Supervision

Negative example design is central to LUNE's effectiveness.

- **LLMs:** For each fact or behavior to be forgotten, negative completions are synthesized that either (a) state explicit contradictions, (b) propose plausible but incorrect alternatives, or (c) use paraphrased forms to maximize generalization. Candidate negatives are filtered to eliminate uncertain or hedged completions, and diversity is maintained to avoid paraphrase dominance [2512.07375]. No positive (“retain”) supervision is needed.
- **Medical Models:** The forget set comprises inputs whose outputs or features are to be erased. Supervisory losses include label flips (forcing the probability distribution away from the ground-truth), explicit teacher contradiction, and entropy maximization, all applied exclusively to the forget set $\mathcal{D}_f$ [2511.16574].

This negative-only approach enables targeted removal with minimal impact on unrelated functionality, as confirmed by ablations comparing LUNE with random or irrelevant negatives.

## 4. Training Procedures, Losses, and Hyperparameters

### LLMs

- Loss: Negative log-likelihood over negative pairs $(x, y^-)$,
  $$
  L(\phi) = -\sum_{(x, y^-) \in \mathcal{D}_{neg}} \log P_{\theta, \phi}(y^-|x).
  $$
- Optimization: AdamW with learning rate $2\times 10^{-4}$, weight decay $0.01$, and mixed precision.
- LoRA settings: Default rank $r=16$ ($r\in\{2,4,8,16,32\}$ tested in ablation), scaling $\alpha = r$, dropout $0.05$.
- Early stopping is based on convergence of the Unlearning Success Rate (USR) and General Utility Retention (GUR) metrics [2512.07375].

### Medical Segmentation

- **Strong unlearning loss** ($L_{asc}$) combines:
  - Label-flip ($L_{flip}$),
  - Teacher-contradiction ($L_{tc}$),
  - Entropy maximization ($L_{ent}$),
  - Feature repulsion ($L_{rep}$),
  - Mean-probability regularization ($L_{mean}$),
  - Total variation ($L_{tv}$).
- LoRA hyperparameters: rank $r=8$, dropout $0.05$, learning rates $\eta_\phi = 10^{-4}$ (LoRA), $\eta_\psi = 5\times10^{-5}$ (head).
- Batch sizes and step counts are dataset-dependent.
- The *gentle restoration* phase applies supervised, distillation, and guard losses only to the segmentation head [2511.16574].

## 5. Evaluation Metrics and Experimental Results

LUNE is evaluated using a suite of metrics quantifying both unlearning efficacy and utility retention.

| Metric  | Description | Reference |
|---------|-------------|-----------|
| USR     | Fraction of prompts where the undesired output is absent (LLM) | [2512.07375] |
| GUR     | Ratio of general-domain performance post-unlearning to baseline | [2512.07375] |
| APR     | Robustness to adversarial or paraphrased prompts | [2512.07375] |
| MIA     | Membership inference attack accuracy (lower is better) | [2512.07375] |
| $\Delta_\mathrm{forget}$ | IoU drop on the forget set (vision) | [2511.16574] |
| $\Delta_\mathrm{retain}$ | IoU drop on the retain set (vision) | [2511.16574] |

Quantitative results indicate that LUNE achieves state-of-the-art GUR across benchmarks (e.g., 95.1% on EDU-RELAT, 93.7% on RWKU), high USR (88–92%), and lowest MIA in most settings [2512.07375]. In medical segmentation, forget-set IoU drops from 0.875 to 0.509 (ISIC), while retain-set IoU remains stable (0.677 vs 0.647), demonstrating selective forgetting [2511.16574].

Computational costs are reduced by 10× compared to full fine-tuning, as LoRA parameters comprise $10^{-3}$–$10^{-2}$ of total model weights, with proportional reductions in optimizer state and gradient updates [2512.07375].

## 6. Theoretical Implications and Limitations

Restricting updates to a low-rank subspace (via LoRA) localizes parameter changes, minimizing the risk of catastrophic forgetting or unintended utility loss. Negative supervision targets only the desired conceptual region in function space, with empirical ablations showing clear superiority over random or non-contradictory negatives for all key metrics.

Limitations include:
- Over-suppression, where negative fine-tuning may marginally impair related knowledge (though GUR remains high).
- Current experiments focus on single-fact or attribute forgetting; extension to more complex, compositional, or multi-instance domains is ongoing.
- The robustness of unlearning depends on the quality and breadth of negative example construction, which may not capture all adversarial paraphrases or hidden dependencies [2512.07375].

*A plausible implication is that future LUNE derivations could incorporate automated relevance feedback, multi-instance unlearning, and more sophisticated low-rank regularization to further confine updates and scale to continual learning scenarios.*

## 7. Applications, Extensions, and Research Directions

Applications of LUNE span:
- **Medical imaging:** Selective erasure of sensitive anatomical features, lesion classes, or entire sample sets from segmentation/classification networks [2511.16574].
- **Language models:** Removal or suppression of specific facts, personal data, or biased constructs, with direct relevance for privacy and knowledge correction tasks [2512.07375].
- **Other vision/NLP domains:** Adaptation to object detection (e.g., zeroing confidence on specific bounding boxes) and attention-layer LoRA for targeted fact erasure.

Ongoing research explores automated negative dataset generation, continual unlearning within a single model instance, and theoretical analysis of how LoRA’s low-rank constraints bound functional drift and semantic “leakage.” Expanding LUNE to handle broader conceptual unlearning and abstract knowledge remains an open challenge.

**Key References:**  
- "Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks" [2511.16574]  
- "LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples" [2512.07375]

Source: https://www.emergentmind.com/topics/lora-based-unlearning-with-negative-examples-lune