Diffusion Model Relearning Attack (DiMRA)

Updated 10 December 2025

DiMRA is a method that exploits the rich diffusion priors to reconstruct erased or forbidden content by fine-tuning on auxiliary data.
It encompasses multiple attack scenarios, including white-box restoration, transferable black-box embedding, classifier-guided inversion, and gradient leakage, each targeting latent model vulnerabilities.
Empirical results show high restoration rates on datasets like CIFAR-10 and art styles, highlighting significant privacy risks and motivating advanced defense strategies.

Diffusion Model Relearning Attack (DiMRA) refers to a class of attacks that exploit diffusion-based generative models to restore, reconstruct, or recover information that has ostensibly been erased or is meant to remain private. These attacks present severe challenges for privacy, copyright compliance, and data trustworthiness in generative systems. DiMRA encompasses several scenarios: from regenerating “unlearned” or forbidden concepts in text-to-image diffusion models, to reconstructing private or unseen data via classifier guidance or leaked gradients, and to inverting unlearning objectives by exploiting the latent structure of the diffusion prior.

1. Threat Models and Attack Scenarios

The DiMRA framework encompasses several adversarial goals and capabilities, unified by their exploitation of the entangled structure and strong data priors embedded by diffusion models.

White-box restoration after machine unlearning: Given a diffusion model subjected to model unlearning (parameterized as θ₁), the adversary is assumed to have white-box or gradient access, and attempts to fine-tune θ₁ on auxiliary data—either from the original retain set or from a distributionally-aligned external set—to restore regenerative capacity for “forgotten” elements, such as sensitive classes, styles, or identities (Yuan et al., 3 Dec 2025).
Transferable black-box embedding attack: Even when the attacker lacks knowledge of the particular unlearning operation or has only black-box access, adversarial search can identify continuous token embeddings which, when injected into an unlearned text-to-image diffusion model, nonetheless yield the erased concept—achieving cross-method restoration and concept-level transferability (Han et al., 30 Apr 2024).
Classifier-guided inversion: Leveraging only query or gradient access to a victim classifier and a proxy diffusion model, an adversary can reconstruct high-confidence images for a private class, even in the absence of sample-level knowledge of the target distribution (Zheng, 2023).
Gradient-leakage-enabled high-resolution reconstruction: In federated or distributed training, per-example gradients can be eavesdropped, and a public diffusion prior can be fine-tuned to synthesize the original private image, even at previously unattainable resolutions (up to 512×512) (Meng et al., 13 Jun 2024).

These threat settings are characterized by minimal assumptions: often requiring only indirect observational access (gradients, classifier outputs), public diffusion priors, or public auxiliary datasets, and rarely demanding exact knowledge of the unlearned data or model-specific details.

2. Formal Objectives and Algorithms

At the core of DiMRA is the minimization of a loss that, under varying modalities, “pulls” the model or latent variables toward configurations that regenerate the forbidden, private, or erased content.

Restoration via fine-tuning: For a conditional diffusion model with parameters θ₁ (post-unlearning), DiMRA optimizes the original diffusion MSE loss on an auxiliary dataset Dₐᵤ to drive θ₁ back toward a basin in parameter space that encodes the unlearned concept:

$L_{\rm DiMRA}(θ) = \mathbb{E}_{t,\, (x_0, c)\in D_{a_u},\, \epsilon} \bigl\|\epsilon - \epsilon_{θ}(x_t \mid c)\bigr\|^2$

(Yuan et al., 3 Dec 2025)

Adversarial embedding search (min-max strategy): For black-box text-to-image scenarios, the attack seeks an embedding $v^*$ such that, across unlearned models $\widehat\theta$ , inserting $v^*$ into the text encoder restores generation of the target concept:

$v^* = \arg\min_v\, \mathbb{E}_{t, \epsilon} \bigl\|\epsilon - \epsilon_{\widehat\theta}(z_t,\, T_\text{text}(y'\Vert v),\, t).\bigr\|^2$

The search iteratively alternates between model erasure steps (maximizing a surrogate loss on the original model) and restoration steps (minimizing embedding error), pushing $v$ into low-density, unerased regions of the text embedding space (Han et al., 30 Apr 2024).

Classifier-guided DDPM inversion: The adversary optimizes the seed noise $x_T$ of a pre-trained diffusion model so that the generated image $\hat{x}$ maximizes the target class confidence under a victim classifier C:

$\min_{x_T} L(x_T) = \mathrm{CE}\left(C(f_\theta(x_T)),\, y^*\right) + \lambda\,\mathcal{R}(x_T)$

$f_\theta(\cdot)$ denotes the unrolled denoising map (Zheng, 2023).

Gradient-guided fine-tuning: The attacker uses the cosine similarity loss between true gradient $g_\text{tar}$ (from a leaked per-sample gradient) and synthetic $g_\text{gen}$ (from a generated sample), applied to the diffusion model’s parameters:

$L_\text{guided}(\theta) = 1 - \frac{ \langle g_\text{gen}(\theta),\, g_\text{tar} \rangle } { \| g_\text{gen}(\theta)\| \cdot \|g_\text{tar}\| }$

(Meng et al., 13 Jun 2024)

Common to all DiMRA variants is the exploitation of the diffusion prior’s convergence to a rich, data-encoding manifold, making restoration of subtle or erased features feasible with modest optimization effort and auxiliary data.

3. Empirical Results and Impact

Experiments across DiMRA variants demonstrate that diffusion models’ data retention and entanglement substantially undermine the efficacy of current unlearning and privacy-protection methods:

Restoration effectiveness after finetuning-based unlearning:
- On CIFAR-10, unlearning a class via methods such as Sfront or Salun yields near-zero appearance rates for the unlearned class (AR_MU≈0), but after DiMRA, recovery rates (AR_DiMRA) reach up to 96–100% for Sfront and up to 92% for Salun. By contrast, the defense method DiMUM showed AR_DiMRA as low as 2–3% (Yuan et al., 3 Dec 2025).
- On UnlearnCanvas (art-style unlearning), DiMRA achieves recovery rates as high as 0.6–1.0, while DiMUM keeps rates at or below 2% (Yuan et al., 3 Dec 2025).
Transferable adversarial probing: Black-box DiMRA embedding search restores concepts (objects, styles, identities) in unlearned Stable Diffusion with 45–95% success rates, drastically outperforming prompt-only and white-box baseline attacks, especially on narrow concepts and difficult targets such as celebrity identity or fine-grained artistic style (Han et al., 30 Apr 2024).
Classifier-guided inversion: On Olivetti Faces, DiMRA enables the synthesis of novel images classified with >90% confidence as a target identity, outperforming GAN and VAE-based inversion both qualitatively and quantitatively (Zheng, 2023).
Gradient leakage: Applying DiMRA to gradients in distributed learning, reconstructions at 256×256 and 512×512 resolution reach SSIM > 0.99 and PSNR > 25 dB, with time-to-leak far below previous attacks. Differential privacy mechanisms with moderate noise variance (<1e–1) do not prevent recognizable reconstruction (Meng et al., 13 Jun 2024).

The operational simplicity and transferability of DiMRA attacks expose diffusion models as highly vulnerable to both parametric and adversarial embedding attacks, even under restrictive black-box or privacy-enhanced scenarios.

4. Root Causes and Mechanistic Analysis

Several structural properties of diffusion models and unlearning objectives underpin DiMRA’s success:

Non-convergent unlearning objectives: Many finetuning-based unlearning approaches (e.g., maximizing loss or token replacement) produce parameter updates that do not settle at a new stable optimum but rather keep the model hovering near the original pre-trained region. DiMRA simply “re-attracts” the model towards its original trajectory via vanilla diffusion training on auxiliary data (Yuan et al., 3 Dec 2025).
Latent entanglement and concept leakage: Models retain benign concepts correlated with the erased targets; thus, small fine-tuning on related or even unrelated data sources can “unlock” recombinations that regenerate the forgotten content (Gao et al., 16 Oct 2024).
Insufficient erasure at the embedding or manifold level: Most unlearning methods shift specific mappings but do not remove the capacity of the model’s latent space or embedding space to encode or interpolate towards the target concept. DiMRA’s min-max optimization finds adversarial “hidden” embeddings in these subspaces (Han et al., 30 Apr 2024).
Powerful priors and smoothness: The strong image prior in DDPM/DDIM models and the low-dimensionality of their latent spaces facilitate high-fidelity restoration, even when the explicit target is absent from the auxiliary data or when only gradients are available (Meng et al., 13 Jun 2024).

This confluence of factors makes diffusion models particularly brittle to inversion, restoration, and unlearning-reversal attacks, often regardless of the fidelity of upstream unlearning methods.

5. Defenses and Robustness Enhancements

Emerging research proposes countermeasures targeting DiMRA vectors:

Meta-unlearning: Instead of only optimizing a standard unlearning loss, a bi-level meta-objective anticipates adversarial fine-tuning. Specifically, it penalizes reductions in attacker loss (on the forbidden concept) and enforces that any such reduction must induce an increase in loss on the retain set, causing “self-destruction” of benign features that could bootstrap restoration. This framework is compatible with existing unlearning methods and empirically reduces restoration of erased concepts by 30–50% post-attack, though with computational overhead (Gao et al., 16 Oct 2024).
Unlearning by Memorization (DiMUM): This approach redefines unlearning as memorizing alternative benign content in place of the targeted concept, with both retain and unlearning losses recast as convergent standard MSEs. This creates a true new local optimum, minimizing the risk of restoration: DiMUM drops AR_DiMRA to <5% on CIFAR-10 and <2% on UnlearnCanvas, outperforming Sfront, Salun, CA, and other baselines (Yuan et al., 3 Dec 2025).
Input obfuscation and gradient protections: In classifier-guided or gradient-leakage scenarios, restricting query or gradient access, adding strong noise (at the expense of utility), or employing secure aggregation are partial defenses. Traditional DP-SGD is insufficient at realistic noise budgets (Meng et al., 13 Jun 2024, Zheng, 2023).
Subspace erasure: Empirical findings suggest robust unlearning must delete information in entire latent subspaces rather than merely shifting token mappings, although practical, scalable methods remain an open topic (Han et al., 30 Apr 2024).

6. Limitations and Open Directions

DiMRA’s efficacy is subject to several caveats and ongoing areas of research:

White-box dependence: Many strong attacks require white-box model access for gradient computation and fine-tuning; fully black-box attacks are more challenging but achievable via embedding search (Yuan et al., 3 Dec 2025, Han et al., 30 Apr 2024).
Auxiliary data requirements: Restoration performance improves with access to distributionally-matched auxiliary datasets; in strong unlearning, required auxiliary quality may be a limiting factor (Yuan et al., 3 Dec 2025).
Model architecture and training regime: Most empirical results pertain to Stable Diffusion, DDPM/DDIM, and classifier-based models; generalization to other architectures and generative modalities is ongoing (Han et al., 30 Apr 2024, Meng et al., 13 Jun 2024).
Hyperparameter tuning: Defense methods such as meta-unlearning and DiMUM demand careful balancing of training and meta-robustness loss weights (Gao et al., 16 Oct 2024).
Scalability and practical deployment: Applying meta-unlearning or data-space memorization across large-scale, continuously updated models in production remains an open challenge. There is also a need for theoretical analysis of worst-case transferability and irreversible unlearning (Gao et al., 16 Oct 2024, Yuan et al., 3 Dec 2025).

DiMRA reveals systemic vulnerabilities in the assumptions of current unlearning and privacy defense methodologies for diffusion models, motivating rigorous architectural and procedural advances in generative model safety controls.

References (by arXiv id)

(Zheng, 2023): “Targeted Image Reconstruction by Sampling Pre-trained Diffusion Model” (2023)
(Han et al., 30 Apr 2024): “Probing Unlearned Diffusion Models: A Transferable Adversarial Attack Perspective” (2024)
(Meng et al., 13 Jun 2024): “Is Diffusion Model Safe? Severe Data Leakage via Gradient-Guided Diffusion Model” (2024)
(Gao et al., 16 Oct 2024): “Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts” (2024)
(Yuan et al., 3 Dec 2025): “Towards Irreversible Machine Unlearning for Diffusion Models” (2025)