UnGuide: Controlled Unlearning in Diffusion Models

• UnGuide is a machine unlearning method that combines LoRA and classifier-free guidance to selectively erase targeted concepts from diffusion models.
• It uses an adaptive inference mechanism, UnGuidance, to dynamically balance outputs between the base and LoRA-adapted models for optimal performance.
• Empirical evaluations show UnGuide achieves effective concept erasure with high image fidelity, outperforming traditional LoRA-only methods.

UnGuide is a machine unlearning method for text-to-image diffusion models that combines Low-Rank Adaptation (LoRA) with a dynamic inference mechanism called UnGuidance to erase specific concepts from a pretrained model while preserving performance on unrelated content. It is designed for settings in which diffusion models may generate harmful, misleading, or unauthorized content, and it addresses a central difficulty of LoRA-based unlearning: targeted concept removal often alters unrelated generations, reducing image fidelity and realism. UnGuide therefore treats concept erasure not only as a fine-tuning problem but also as an inference-time control problem, using Classifier-Free Guidance (CFG) to decide when the unlearned LoRA adapter should dominate and when the base model should remain in control (Polowczyk et al., 7 Aug 2025).

1. Problem setting and design goals

UnGuide is situated within machine unlearning for generative models, where the objective is to remove a specific concept from a pretrained model without degrading its broader generative capacity. In the text-to-image setting, the relevant concepts include objects, styles, and explicit content. The motivating concern is that large-scale diffusion models can synthesize harmful, misleading, or unauthorized content, which creates a need for selective erasure rather than wholesale retraining or coarse filtering (Polowczyk et al., 7 Aug 2025).

The method is explicitly framed against prior LoRA-based unlearning pipelines that require prompt embedding modification, external segmentation tools for accurate erasure localization, or multiple specialized LoRA modules and pipelines. UnGuide instead uses standard LoRA adapters, requires no segmentations and no prompt changes, and introduces an inference-time mechanism that adaptively balances the outputs of the base and LoRA models on a per-prompt and per-sample basis. This design emphasizes controlled forgetting rather than static deployment of an unlearned adapter alone (Polowczyk et al., 7 Aug 2025).

2. Architectural ingredients: LoRA and classifier-free guidance

The first component of UnGuide is LoRA, which adds small trainable low-rank matrices to the weights of a frozen pretrained model. In the formulation reported for UnGuide, the LoRA parameters are inserted into the cross-attention key and value matrices in the U-Net of Stable Diffusion, so only the LoRA weights are updated during fine-tuning. The weight modification is

$W' = W + \beta \cdot BA$

where $B$ and $A$ are low-rank matrices with $r \ll d,k$, and $\beta$ is a scaling factor (Polowczyk et al., 7 Aug 2025).

The second component is CFG, which is used as a steering mechanism during inference. Its standard form is

$$\hat\epsilon_{\theta^*}^{\mathrm{cfg}}(z_t, t, c) = \epsilon_{\theta^*}(z_t, t) + \alpha \left(\epsilon_{\theta^*}(z_t, t, c) - \epsilon_{\theta^*}(z_t, t)\right),$$

where $c$ is the conditional prompt. In UnGuide, CFG is not treated merely as a generic prompt-strength control. It becomes the basis of a selective arbitration mechanism between the original model and the LoRA-adapted unlearned model. This coupling is the distinctive architectural move of the method: LoRA encodes the unlearning objective, while guidance determines whether the unlearning signal should be emphasized for a given prompt (Polowczyk et al., 7 Aug 2025).

3. LoRA-based unlearning objective

UnGuide trains a LoRA adapter to suppress the target concept by replacing it with a mapping concept. Given a noisy latent $z_t$, a prompt $c$ corresponding to the concept to remove, and a mapping concept $c_m$, the method computes three noise predictions: the baseline prediction for the mapping concept, the baseline prediction for the target concept, and the LoRA-adapted prediction for the target concept. These are

$B$0

The LoRA fine-tuning objective is then

$B$1

where $B$2 controls repulsion from the forbidden concept. In the description of the method, this loss trains the LoRA model to suppress the noise features related to the concept to erase and replace them with a neutral or desired mapping concept. The result is not full-model retraining but a targeted adapter that encodes an erasure direction (Polowczyk et al., 7 Aug 2025).

This training formulation is important because UnGuide does not assume that unlearning should be represented as a simple deletion of capacity. Instead, the target concept is displaced toward another concept representation. A plausible implication is that the method is better understood as controlled redirection of denoising behavior than as mere removal of parameters or logits.

4. UnGuidance as adaptive inference-time control

The defining feature of UnGuide is UnGuidance, a dynamic inference mechanism that measures disagreement between the base model and the LoRA-adapted model early in the denoising process and uses that disagreement to decide which model should dominate. The procedure begins by sampling an initial noise vector $B$3, partially denoising using the base model to a step $B$4, and then evaluating both models on the partially denoised latent. The disagreement for a prompt $B$5 is quantified as

$B$6

This estimate is repeated with different seeds and averaged for robustness. A reference value is then established using a neutral or empty prompt $B$7,

$B$8

If the average disagreement for $B$9 exceeds the reference value for $A$0, the prompt is treated as likely containing a concept to erase, and the guidance weight $A$1 is set to prioritize the LoRA-adapted model. Otherwise, the base model is prioritized. Generation then proceeds with a blended prediction,

$A$2

The reported formulation combines the CFG outputs of both models but uses only conditional paths, not unconditional paths, for precision (Polowczyk et al., 7 Aug 2025).

The operational interpretation is direct. For prompts containing the erased concept, the LoRA module predominates and is counterbalanced by the base model; for unrelated prompts, the base model governs generation, preserving content fidelity. This mechanism addresses the common failure mode in which a LoRA adapter that successfully erases a concept also induces out-of-distribution artifacts or semantic drift on safe prompts. UnGuide therefore treats the first few denoising steps as a diagnostic phase for selective forgetting rather than applying a uniform unlearning strength to all inputs (Polowczyk et al., 7 Aug 2025).

5. Empirical evaluation and reported behavior

UnGuide is evaluated on two principal settings: object erasure and explicit content removal. For object erasure, the reported experiments use CIFAR-10 classes and evaluate efficacy $A$3, specificity $A$4, generality $A$5, and a harmonic mean $A$6. The reported outcome is that UnGuide achieves the highest harmonic mean $A$7 for object erasure across CIFAR-10 classes, combining better erasure with less impact elsewhere than LoRA-only and other unlearning methods (Polowczyk et al., 7 Aug 2025).

For explicit content removal, the evaluation uses I2P prompts, a NudeNet detector, and general-content FID and CLIP on MS-COCO. The reported result is that UnGuide yields the lowest number of inappropriate images, 31 out of 4703, while keeping FID and CLIP scores high. In the qualitative description, this is taken to show that controlled unlearning can be achieved without sacrificing safe-prompt fidelity (Polowczyk et al., 7 Aug 2025).

The paper also reports several qualitative properties. For non-target prompts, UnGuide produces images nearly indistinguishable from the original model, avoiding unwanted out-of-distribution artifacts or loss of diversity. Dynamic guidance is described as preventing over-erasure and preserving original generative power, which is not achieved when only the LoRA-unlearned model is deployed. The framework also supports simultaneous erasure of