Anti-Personalized Diffusion Models (APDM)

• Anti-Personalized Diffusion Models (APDM) are privacy-preserving defenses that introduce imperceptible perturbations to disrupt unauthorized fine-tuning in diffusion models.
• They employ both image-space and model-space techniques, including trajectory matching and controlled perturbation, to degrade personalized outputs while maintaining visual fidelity.
• Recent advancements shift from simple output degradation to targeted redirection and output immunization, enhancing robustness against adaptive attacks and cross-model transfer.

Searching arXiv for the cited APDM and related anti-personalization papers to ground the article in recent research. I’m checking arXiv for the core APDM paper and adjacent anti-personalization work. Anti-Personalized Diffusion Models (APDM) denote privacy-preserving defenses for diffusion-model personalization. In the image-side formulation, APDM add small, typically imperceptible perturbations to user images so that downstream personalization methods such as DreamBooth, Textual Inversion, or LoRA fail to learn the true identity, style, or object; in later formulations, the protection target shifts from images to model parameters, and in a related line of work the objective shifts to model-side output immunization against identity linkability in authorized personalization (Le et al., 2023, Liu et al., 2024, Lee et al., 3 Nov 2025, Dai et al., 1 Apr 2026). Across these uses, the common purpose is to prevent unauthorized or privacy-violating subject reproduction by diffusion-based generative systems.

1. Terminology, scope, and research trajectory

The literature uses the term APDM in a broad, method-family sense and in several more specific senses. One usage treats APDM as “protective perturbations,” namely small image-space perturbations added to private images so that a personalized diffusion model fine-tuned on them becomes “unlearnable” with respect to the protected subject (Liu et al., 2024). Another usage defines APDM as a privacy protection system for “general diffusion-based methods,” including both tuning-based personalization and editing-based abuse (Li et al., 7 Mar 2025). A later paper uses the same acronym for a model-level framework that “perturb[s] a model, not an image” (Lee et al., 3 Nov 2025). A related but distinct formulation, IDDM, explicitly positions itself beyond anti-personalization and addresses “model-side output immunization,” where personalization is authorized but public outputs should have reduced identity linkability (Dai et al., 1 Apr 2026).

Chronologically, one early formulation is Anti-DreamBooth, which studies user-side perturbations against DreamBooth-based personalization and evaluates transfer to Textual Inversion and LoRA (Le et al., 2023). DDAP then introduces a “Dual-Domain Anti-Personalization” strategy that alternates spatial and frequency perturbation learning under a localization mask (Yang et al., 2024). Subsequent work expands the design space toward prompt-agnostic protection and editing resistance, targeted redirection to user-specified concepts, latent-space perturbation, model-parameter protection, and anti-aesthetic optimization (Li et al., 7 Mar 2025, Lee et al., 11 Dec 2025, Devulapally et al., 3 Oct 2025, Wang et al., 2 Jul 2026).

Formulation	Core mechanism in source	Reported scope
Anti-DreamBooth	PGD on surrogate DreamBooth losses	DreamBooth, Textual Inversion, LoRA
DDAP	SPL + FPL + localization mask	Personalized generation on facial benchmarks
Anti-Diffusion	Prompt Tuning + URL + SDL + UNet update	Tuning and editing methods
TAFAP	Trajectory matching over fine-tuning checkpoints	Targeted Data Protection
Latent Diffusion Unlearning	Trajectory-shifted latent perturbation	Personalization and inversion robustness
APDM (model-side)	DPO + L2P on model parameters	Robust anti-personalization
IDDM	Two-stage identity-decoupled optimization	Output immunization
HAA	Global and Local Anti-Aesthetics	Facial privacy against customized diffusion

Taken together, these works suggest that APDM has evolved from untargeted degradation of downstream personalization toward controllable, trajectory-aware, and model-aware protection.

2. Threat models and optimization objectives

A standard threat model assumes that an adversary obtains user images $x$, fine-tunes a public diffusion backbone $\epsilon_\theta$ to embed a new token $S^*$, and then generates new images containing that subject at inference time (Yang et al., 2024). Anti-DreamBooth states the defender’s goal as crafting perturbations $\delta_i$ for a private image set $X=\{x_1,\dots,x_N\}$ so that any DreamBooth-based personalized generator fine-tuned on the perturbed set $X'=\{x_i+\delta_i\}$ produces low-quality or unrecognizable images of the user (Le et al., 2023).

The formalism is typically bilevel. Anti-DreamBooth writes

$$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$

with $\|\delta_i\|_p\le \eta$, and in practice replaces the abstract evaluation $A$ with maximization of the conditional diffusion loss $L_{cond}$ on a surrogate (Le et al., 2023). A closely related constrained maximization in the protective-perturbation literature is

$\epsilon_\theta$0

where $\epsilon_\theta$1 is the standard DreamBooth loss (Liu et al., 2024).

The objectives broaden in later work. TAFAP formulates Targeted Data Protection (TDP) with three explicit aims: Identity Concealing, Verifiability, and Traceability, and defines APDM as protection that causes downstream fine-tuning to learn and generate a user-specified target concept rather than merely degrading output quality (Lee et al., 11 Dec 2025). Anti-Diffusion enlarges the abuse surface from tuning-based personalization to editing-based misuse such as MasaCtrl and DiffEdit (Li et al., 7 Mar 2025). IDDM introduces a new setting in which the attacker does not see the private training set or weights, but only public generations and a black-box face recognition system, so the goal becomes low “identity linkability” under varied prompts rather than failed personalization per se (Dai et al., 1 Apr 2026).

Anti-DreamBooth also distinguishes three defender settings: convenient (white-box), adverse (gray-box), and uncontrolled (black-box), including the case where clean and perturbed images are mixed (Le et al., 2023). This taxonomy remains influential because later work repeatedly reports degradation under model, prompt, or architecture mismatch.

3. Image-space anti-personalization mechanisms

The earliest image-space APDM mechanisms optimize perturbations against surrogate personalization loops. Anti-DreamBooth introduces Fully-trained Surrogate Model Guidance (FSMG), Alternating Surrogate and Perturbation Learning (ASPL), targeted variants T-FSMG and T-ASPL, and ensemble extensions E-FSMG and E-ASPL (Le et al., 2023). The central empirical claim is that ASPL consistently outperforms FSMG by better simulating the attacker’s fine-tuning loop, while untargeted PGD on $\epsilon_\theta$2 is more effective than targeted attacks aimed at a single surrogate target (Le et al., 2023).

DDAP argues that prior defenses overlook intrinsic properties of diffusion models and therefore fail to achieve both effective defense and stealthiness. Its Spatial Perturbation Learning (SPL) exploits the fixed and perturbation-sensitive nature of the image encoder in personalized generation and maximizes a joint loss combining denoiser discrepancy and a latent-code gap: $\epsilon_\theta$3 Its Frequency Perturbation Learning (FPL) instead perturbs the 2D Fourier spectrum with a weighted learnable spectrum perturbation $\epsilon_\theta$4, emphasizing mid-to-high frequencies, and optimizes

$\epsilon_\theta$5

under a spectral-norm constraint (Yang et al., 2024). DDAP alternates these two updates and then applies a localization mask $\epsilon_\theta$6 derived from Diffusion Attentive Attribution Maps (DAAM), using DDIM inversion and cross-attention maps for the new token $\epsilon_\theta$7, so that perturbations are concentrated on attentive regions rather than the background (Yang et al., 2024).

Anti-Diffusion addresses two stated limitations of earlier image-side defenses: reliance on manually defined prompts and neglect of editing methods. Its Stage 1, Prompt Tuning (PT), optimizes a text embedding $\epsilon_\theta$8 starting from an empty prompt so that the embedding precisely describes the protected image. Stage 2 then maximizes a composite cost consisting of the UNet Reverse Loss (URL), which forces incorrect noise prediction and is intended to thwart tuning, and the Semantic Disturbance Loss (SDL), which drives cross-attention activations toward a zero target map and is intended to destroy semantic alignment for editing prompts. Stage 3 updates the UNet on the perturbed image and tuned prompt, thereby simulating a malicious tuning step (Li et al., 7 Mar 2025).

HAA introduces an aesthetic formulation. Rather than attacking only attention or feature distances, it degrades human-perceived aesthetic quality at two levels. The Global Anti-Aesthetics branch uses a frozen BLIP-based aesthetic scorer $\epsilon_\theta$9 on a single-step diffusion decode and optimizes a global anti-aesthetic reward; the Local Anti-Aesthetics branch detects a face box with RetinaFace and applies a frozen local reward model $S^*$0 to facial detail. The perturbation $S^*$1 is optimized under an $S^*$2-bounded PGD update on

$S^*$3

where $S^*$4 is the DreamBooth reconstruction plus prior-preservation loss (Wang et al., 2 Jul 2026).

4. Trajectory matching, targeted redirection, and latent-space protection

A major shift in APDM research is from perturbation objectives defined at a single fine-tuning state to objectives defined over the entire fine-tuning trajectory. TAFAP makes this explicit by arguing that snapshot-matching approaches have poor controllability because their protective influence is diluted as training progresses. It therefore builds an expert trajectory $S^*$5 from target-concept fine-tuning and matches the protected-data trajectory $S^*$6 to it with the normalized loss

$S^*$7

optimized over $S^*$8 with PGD-style sign updates (Lee et al., 11 Dec 2025). In this framework, APDM no longer merely degrades output quality; it actively redirects downstream fine-tuning toward a chosen decoy identity or even arbitrary chaotic masks, thereby supporting verifiable and traceable protection (Lee et al., 11 Dec 2025).

Latent Diffusion Unlearning moves the perturbation mechanism from pixel space to the latent trajectory of an LDM. It first encodes $S^*$9 to $\delta_i$0, inverts to a terminal latent $\delta_i$1, perturbs the terminal latent by a trainable transform $\delta_i$2, denoises for only $\delta_i$3 steps, and decodes back to an image $\delta_i$4 (Devulapally et al., 3 Oct 2025). The protection objective is a Lagrangian that maximizes personalization loss while imposing an imperceptibility constraint: $\delta_i$5 The paper presents this as “trajectory-shifted sampling,” with the stated benefit that perturbed images remain high-fidelity to the originals while resisting inversion and personalization (Devulapally et al., 3 Oct 2025).

These trajectory-based approaches differ in control objective. TAFAP aligns the full learning dynamics of protected-data fine-tuning to a target trajectory, whereas latent unlearning shifts the latent denoising trajectory itself. This suggests two distinct notions of trajectory in APDM research: parameter-space trajectory control and latent-state trajectory control.

5. Model-parameter protection and output immunization

The model-side APDM framework of “Perturb a Model, Not an Image” begins from a theoretical critique of naïve adversarial loss design. It considers the direct analogue of image-level poisoning,

$\delta_i$6

and states a gradient-alignment condition

$\delta_i$7

as necessary for a local minimizer. The paper then proves a no-convergence result for $\delta_i$8: under the alignment requirement, first-order inequalities for simultaneously increasing personalized loss and decreasing prior-preservation loss become contradictory (Lee et al., 3 Nov 2025).

To resolve this, the paper introduces Direct Protective Optimization (DPO), a preference-based loss adapted from Direct Preference Optimization. Given paired samples $\delta_i$9, where $X=\{x_1,\dots,x_N\}$0 is a generic sample to preserve and $X=\{x_1,\dots,x_N\}$1 is a protected sample to discourage, it defines reward differences relative to a frozen pretrained reference model $X=\{x_1,\dots,x_N\}$2 and optimizes a Bradley–Terry likelihood: $X=\{x_1,\dots,x_N\}$3 The protection loss is then

$X=\{x_1,\dots,x_N\}$4

Protection is made trajectory-aware through Learning to Protect (L2P), a dual-path optimization that alternates a simulated personalization path of $X=\{x_1,\dots,x_N\}$5 gradient steps with a protection path that accumulates $X=\{x_1,\dots,x_N\}$6 across all intermediate states before updating the protected model $X=\{x_1,\dots,x_N\}$7 (Lee et al., 3 Nov 2025).

IDDM addresses a different privacy problem. It assumes personalization is authorized but the generated outputs may still be linked back to the user by downstream face recognition systems. The method alternates short personalization updates on a fixed reference set $X=\{x_1,\dots,x_N\}$8 with identity-decoupled data updates on a protection set $X=\{x_1,\dots,x_N\}$9, using two losses: a fidelity-preserving denoising loss $X'=\{x_i+\delta_i\}$0 and an identity-decoupling loss

$X'=\{x_i+\delta_i\}$1

where $X'=\{x_i+\delta_i\}$2 is a weighted ensemble of cosine similarities from multiple face recognition systems (Dai et al., 1 Apr 2026). The privacy–utility trade-off is controlled by $X'=\{x_i+\delta_i\}$3, the fraction of PGD steps devoted to fidelity before switching to privacy minimization. Unlike classical anti-personalization, the aim is not to block the user’s own DreamBooth-style model, but to produce a personalized model whose public outputs are significantly harder to link back to the user (Dai et al., 1 Apr 2026).

6. Evaluation protocols and representative empirical findings

APDM evaluation is centered on facial personalization benchmarks, chiefly VGGFace2 and CelebA-HQ, with some work extending to WikiArt, DreamBooth objects, MS-COCO, and an editing benchmark called Defense-Edit (Le et al., 2023, Devulapally et al., 3 Oct 2025, Li et al., 7 Mar 2025). The common experimental pattern is to protect a small number of images per identity, fine-tune a personalized model such as DreamBooth or LoRA, and then score the generated outputs for identity leakage, face detectability, output distortion, or prompt alignment.

Metric	Definition in source	Direction
PSNR	$X'=\{x_i+\delta_i\}$4	higher = more imperceptible
LPIPS	Learned perceptual distance	lower = more imperceptible
FDFR	$X'=\{x_i+\delta_i\}$5	higher = more disruption
ISM	ArcFace cosine similarity	lower = more disruption
BRISQUE	No-reference naturalness score	source-dependent interpretation
SER-FIQ	Face quality via embedding robustness	source-dependent interpretation
FID	Fréchet distance between distributions	lower for generation quality; higher for divergence in some defense tables
CLIP Score	Alignment of edited image to prompt	lower = stronger editing defense

Anti-DreamBooth reports, in the convenient setting on VGGFace2 with Stable Diffusion v2.1 and the default prompt, that no defense yields FDFR≈0.07 and ISM≈0.63, whereas ASPL yields FDFR≈0.63 and ISM≈0.33; it also reports transfer to SD v1.4 and v1.5, disruption under novel prompts such as “a dslr portrait of sks person,” degradation of LoRA with FDFR≈0.64 and ISM≈0.23, and effectiveness when at least half of the reference images are perturbed in the uncontrolled setting (Le et al., 2023).

DDAP reports on VGGFace2 under the prompt “a photo of sks person” that Anti-DB attains PSNR $X'=\{x_i+\delta_i\}$6, LPIPS $X'=\{x_i+\delta_i\}$7, FDFR $X'=\{x_i+\delta_i\}$8, ISM $X'=\{x_i+\delta_i\}$9, BRISQUE $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$0, and SER-FIQ $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$1, while DDAP attains PSNR $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$2, LPIPS $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$3, FDFR $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$4, ISM $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$5, BRISQUE $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$6, and SER-FIQ $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$7; on CelebA-HQ and under a “DSLR portrait” prompt it improves FDFR by $$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$8–$$\min_{\Delta=\{\delta_i\}}\;A(\theta^*(\Delta),X) \quad\text{s.t.}\quad \theta^*(\Delta)=\arg\min_\theta \sum_{i=1}^{N_{db}} L_{db}(\theta; x_i+\delta_i),$$9 and PSNR by $\|\delta_i\|_p\le \eta$0 dB versus baselines (Yang et al., 2024).

Anti-Diffusion reports DreamBooth defense on VGGFace2 with APDM at PSNR $\|\delta_i\|_p\le \eta$1, FDFR $\|\delta_i\|_p\le \eta$2, ISM $\|\delta_i\|_p\le \eta$3, SER-FQA $\|\delta_i\|_p\le \eta$4, BRISQUE $\|\delta_i\|_p\le \eta$5, FID $\|\delta_i\|_p\le \eta$6, and NIQE $\|\delta_i\|_p\le \eta$7, outperforming Anti-DB and other baselines in that table; it also reports the strongest disruption on the Defense-Edit benchmark under both MasaCtrl and DiffEdit while preserving the best PSNR on the original image (Li et al., 7 Mar 2025).

HAA reports on CelebA-HQ with the prompt “a photo of sks person” that clean training yields FDSR $\|\delta_i\|_p\le \eta$8, FS $\|\delta_i\|_p\le \eta$9, IR $A$0, and FID $A$1, whereas HAA yields $A$2, $A$3, $A$4, and $A$5; on VGGFace2 with “a close-up photo of sks person, high details,” HAA reduces FDSR from $A$6 in CAAT to $A$7, FS from $A$8 to $A$9, IR from $L_{cond}$0 to $L_{cond}$1, and raises FID from $L_{cond}$2 to $L_{cond}$3 (Wang et al., 2 Jul 2026).

The model-side APDM paper reports DINO $L_{cond}$4 and BRISQUE $L_{cond}$5 averaged over person and dog protection, compared with substantially higher DINO for all image-poisoning baselines even when they are given no clean images, and preserves unconditional COCO generation at FID $L_{cond}$6, CLIP $L_{cond}$7, TIFA $L_{cond}$8, and GenEval $L_{cond}$9, compared with Stable Diffusion at FID $\epsilon_\theta$00, CLIP $\epsilon_\theta$01, TIFA $\epsilon_\theta$02, and GenEval $\epsilon_\theta$03 (Lee et al., 3 Nov 2025).

7. Limitations, counter-defenses, and open problems

A recurrent limitation is computational cost. DDAP notes that alternating SPL and FPL is costlier than single-domain attacks (Yang et al., 2024). TAFAP requires saving expert trajectories, with snapshots every iteration and approximately $\epsilon_\theta$04 MB per LoRA checkpoint, and also requires $\epsilon_\theta$05 PGD iterations with unrolled inner steps (Lee et al., 11 Dec 2025). The model-side APDM reports approximately $\epsilon_\theta$06 GPU hours of protection on a single NVIDIA RTX A6000 (Lee et al., 3 Nov 2025).

Another limitation is assumption mismatch. TAFAP explicitly lists a white-box assumption, although it also reports encouraging cross-model transfer (Lee et al., 11 Dec 2025). Latent Diffusion Unlearning requires access to the pretrained diffusion model in order to compute inversion and denoising operators (Devulapally et al., 3 Oct 2025). HAA reports weaker transfer to “very different architectures,” specifically transformer-based denoisers in SD-v3.0 and FLUX (Wang et al., 2 Jul 2026). DDAP states that black-box transferability degrades when the attacker’s prompt or model version mismatches (Yang et al., 2024).

A further issue is that image-side APDM are not unchallenged. “Rethinking and Defending Protective Perturbation in Personalized Diffusion Models” analyzes protective perturbations through shortcut learning and argues that they induce latent-space misalignment between images and prompts in CLIP space; it then proposes purification with off-the-shelf restoration, contrastive decoupling with noise tokens, and noise-free sampling with negative prompting, reporting IMS≈0.27 and $\epsilon_\theta$07 averaged across perturbations and only a small drop under an adaptive white-box attack on the purifier (Liu et al., 2024). This suggests that APDM can themselves become targets of restoration and adaptive defense.

The robustness of data poisoning in the presence of clean images is another point of contention. Anti-DreamBooth reports strong degradation when too many clean images leak into the reference set (Le et al., 2023). The model-side APDM paper criticizes earlier methods for relying on unrealistic assumptions and becoming ineffective in the presence of even a few clean images or simple image transformations, and uses this critique to motivate direct model perturbation (Lee et al., 3 Nov 2025).

Finally, complete identity removal is not guaranteed. HAA states explicitly that complete identity erasure is not guaranteed and that residual cues may persist under extreme prompt mismatch or new customization pipelines such as Textual Inversion (Wang et al., 2 Jul 2026). IDDM makes a related point from the opposite direction: even if anti-personalization succeeds at blocking unauthorized personalization, it does not solve the practical case where authorized personalization still produces publicly linkable outputs (Dai et al., 1 Apr 2026).

Open directions are therefore already visible in the literature: accelerating dual-domain and trajectory-based optimization, reducing checkpoint storage, broadening cross-model and cross-recipe robustness, extending protection beyond faces to objects and scenes, integrating traceability with watermarking or cryptographic proofs, and handling multi-subject or continual protection in model-side APDM (Yang et al., 2024, Lee et al., 11 Dec 2025, Lee et al., 3 Nov 2025). The current record indicates that APDM is no longer a single technique but a rapidly diversifying research area spanning image perturbation, latent manipulation, target-concept redirection, model-parameter protection, and privacy-preserving output immunization.