Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks

Abstract: Machine unlearning for text-to-image diffusion models aims to selectively remove undesirable concepts from pre-trained models without costly retraining. Current unlearning methods share a common weakness: erased concepts return when the model is fine-tuned on downstream data, even when that data is entirely unrelated. We adapt Projected Gradient Unlearning (PGU) from classification to the diffusion domain as a post-hoc hardening step. By constructing a Core Gradient Space (CGS) from the retain concept activations and projecting gradient updates into its orthogonal complement, PGU ensures that subsequent fine-tuning cannot undo the achieved erasure. Applied on top of existing methods (ESD, UCE, Receler), the approach eliminates revival for style concepts and substantially delays it for object concepts, running in roughly 6 minutes versus the ~2 hours required by Meta-Unlearning. PGU and Meta-Unlearning turn out to be complementary: which performs better depends on how the concept is encoded, and retain concept selection should follow visual feature similarity rather than semantic grouping.

• The paper introduces PGU, a post-hoc method that projects fine-tuning gradients to block revival of erased concepts in diffusion models.
• It employs a Core Gradient Space derived from U-Net activations to eliminate style revival and delay object concept reactivation.
• Comparative analysis shows PGU is computationally efficient and complements Meta-Unlearning for enhanced model protection.

Projected Gradient Unlearning for Diffusion: Preventing Concept Revival under Fine-Tuning

Introduction

The proliferation of text-to-image diffusion models—exemplified by architectures such as Stable Diffusion and DALL·E—has intensified the demand for mechanisms that support selective erasure of undesirable or legally protected content from generative models. Standard practice employs machine unlearning approaches, but a persistent vulnerability has emerged: erased concepts can be reinstated via downstream fine-tuning, even on unrelated data, jeopardizing compliance with regulatory mandates such as the GDPR. "Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks" (2604.21041) reconceptualizes the defense paradigm by porting Projected Gradient Unlearning (PGU) from classification settings to diffusion models, introducing a geometry-driven, post-hoc hardening method that directly blocks the fine-tuning directions leading to concept revival.

Figure 1: Fine-tuning vulnerability and PGU defense. Top: without PGU, the Van Gogh style revives after fine-tuning on unrelated data. Bottom: with PGU hardening, erasure is preserved during downstream adaptation.

The Fine-Tuning Vulnerability in Diffusion Unlearning

Existing unlearning techniques, such as ESD, UCE, and Receler, typically suppress targeted content by parameter editing or introducing adapter modules. Prior analyses have exposed a universal flaw: erased concepts reliably "revive" when the model is subject to task-agnostic fine-tuning, threatening post-deployment guarantees. This revival transcends method details and reflects the fact that retained concepts in the model share latent features with erased targets, allowing gradient-based optimization during fine-tuning to reoccupy the erased subspace. Experimental evidence reveals the severity of the problem, particularly for compact, high-level object encodings, which can be reinstated with minimal fine-tuning data.

Projected Gradient Unlearning Adaptation: Core Gradient Space Projection

The PGU framework advances a geometric approach for defense. The central construction is the Core Gradient Space (CGS): a subspace of model parameter gradients spanned by per-layer activations on select "retain concepts"—specifically, visually similar prompts. Fine-tuning gradients are projected orthogonally to this space, guaranteeing that updates cannot reactivate the directions responsible for generating the erased concept.

Key technical adjustments are necessary for diffusion models: (1) PGU is applied post-hoc rather than as part of the main unlearning loop, (2) CGS is constructed from U-Net activations, requiring forward hooks and SVD-based eigenspace estimation, and (3) loss/optimization protocols are tailored for denoising objectives.

Empirical Evaluation: Robustness, Efficacy, Hyperparameter Sensitivity

Quantitative evaluation leverages Standard Diffusion v1.4 with both style ("Van Gogh") and object ("Golf Ball") concept benchmarks, assessed under strong curriculum-based fine-tuning attacks that traverse semantic and visual proximity. Detection of revival is operationalized via dual CLIP and ViT classifier thresholds.

Figure 2: Classifier accuracy vs. fine-tuning curriculum checkpoint (C0--C9) for Van Gogh (top) and Golf Ball (bottom); PGU eliminates Van Gogh revival entirely (all methods) and substantially delays Golf Ball revival, with Receler robust in both cases.

PGU achieves several salient outcomes:

• Vulnerable models: For ESD-U, Van Gogh revival is eliminated (peak accuracy drops from 71.4% to 28.6%), and Golf Ball revival is delayed 4-fold (from C1 to C4 in the curriculum), when visually selected retain concepts are used.
• Robust models: PGU does not degrade (and often improves) existing robustness, e.g., for models in which erasure already persists benignly.
• Failed erasure: Models where erasure did not occur (e.g., UCE Golf Ball) remain unprotected, confirming that PGU is strictly a defense, not an unlearning algorithm.

  Figure 3: ESD-U + PGU gamma ablation across checkpoints; $\gamma=0.7$ is optimal for style, $\gamma=0.9$ for object, with revival suppressed below threshold.

Hyperparameter ablation for the CGS coverage parameter $\gamma$ demonstrates that lower $\gamma$ (0.5--0.7) is optimal for distributed style encodings, while higher $\gamma$ (0.7--0.9) better suppresses revival for compact objects. Notably, visual similarity between the retain set and the erased concept is a primary determinant of defense effectiveness.

Figure 4: Hypothesis test: visual retain concepts (green) provide a $2\times$ delay in object concept revival compared to semantic retain; robust defense demands visual feature overlap.

Meta-Unlearning Comparison and Architectural Implications

Comparison with Meta-Unlearning, the main anticipatory alternative based on attack simulation and self-destruction during training, reveals a complementary performance profile. PGU surpasses Meta-Unlearning for distributed style concepts, owing to its broad geometric coverage, whereas Meta-Unlearning provides tighter protection for compact object encodings. Crucially, PGU's computational footprint is far lower (6 minutes, <12GB VRAM) versus Meta-Unlearning's bi-level optimization requirements (~2 hours, >40GB VRAM), enhancing its practical deployability.

Practical Guidelines and Theoretical Considerations

The analysis indicates:

• Retain concept selection: Visual feature overlap (texture, color, geometry) outperforms semantic relatedness for building the CGS, aligning with the underlying representation learning priors of diffusion architectures.
• Safe deployment: PGU preserves benign adaptation and does not interfere with utility in safe directions, a crucial property for deployed models subject to ongoing fine-tuning.
• Limitations: Revival can only be delayed, not guaranteed eliminated, for compact, high-level objects due to architectural constraints; coverage of the underlying feature manifold is bounded.
• Interoperability: PGU can harden any method with effective initial erasure and is modular by design; hybridization with Meta-Unlearning strategies could yield additive benefits.

Conclusion

Projected Gradient Unlearning, adapted for text-to-image diffusion models, represents a rigorous post-hoc defense against concept revival under arbitrary fine-tuning. The method leverages the geometry of model representations to block vulnerable gradient directions, ensures computational efficiency, and is effective provided initial erasure achieves suppression. While it is not a panacea for all concept structures, particularly compact object encodings, its complementary strengths with Meta-Unlearning indicate the potential for robust, scalable, and theoretically principled unlearning pipelines in high-stakes generative modeling contexts. Future directions include the automatic selection of retain prompts via visual similarity search, cross-architecture validation, and adaptive hybridization schemes.