HAAD-KV: Adversarial Attack for Diffusion Models

• HAAD-KV is a refined variant of HAAD that targets key and value projections within the h-space, enabling precise anti-customization defenses.
• It protects personal images by generating imperceptible perturbations that poison training data, undermining few-shot personalization schemes.
• Empirical evaluations show HAAD-KV outperforms baselines by updating only 5.24M parameters, achieving robust protection with minimal visual degradation.

HAAD-KV denotes the more efficient variant of HAAD, an anti-customization adversarial attack for diffusion models that protects personal or private images from unauthorized few-shot personalization by generating imperceptibly perturbed training images that poison subsequent customization. In the formulation reported in “An h-space Based Adversarial Attack for Protection Against Few-shot Personalization,” HAAD attacks the diffusion model’s “h-space,” defined operationally as the deep features from the middle block of the U-Net denoiser, whereas HAAD-KV restricts perturbation construction to the key and value projection parameters of the cross-attention layer within that h-space (Xu et al., 23 Jul 2025). The method is evaluated against personalization pipelines such as DreamBooth, LoRA+DreamBooth, and Custom Diffusion, with the stated objective that a model fine-tuned on protected images fails to bind the user-specific token to the correct visual concept.

1. Definition and threat model

HAAD stands for h-space based Adversarial Attack for Diffusion models. It is described as an untargeted anti-customization attack that generates an adversarially perturbed image $x' = x + \delta$ by maximizing the latent diffusion reconstruction loss while updating only the h-space parameters of the diffusion U-Net during perturbation construction. HAAD-KV is a refinement of HAAD: the overall strategy is unchanged, but the trainable subset used during perturbation generation is reduced to the K and V projection parameters of cross-attention within the h-space (Xu et al., 23 Jul 2025).

The problem setting is few-shot personalization of text-to-image diffusion models. In that setting, an attacker obtains a small number of user images and fine-tunes a diffusion model so that a user-specific token, such as “sks,” becomes associated with a person, artwork, or concept. HAAD-KV is positioned as a data-poisoning defense against that procedure. The operational pipeline is: clean user images are perturbed; the attacker fine-tunes a personalization model on those protected images; and the resulting personalized model produces outputs that are semantically inconsistent, structurally degraded, or identity-mismatched when prompted with the learned token (Xu et al., 23 Jul 2025).

The method assumes that the defender can prepare protected images using a reference diffusion backbone and that the attacker later uses those images directly for standard personalization. The defense does not require modifying the attacker’s training loop. A plausible implication is that HAAD-KV is most directly applicable where image owners control the images that enter downstream customization datasets, rather than the fine-tuning procedure itself.

2. h-space and the KV-restricted mechanism

The conceptual basis of HAAD-KV is the paper’s claim that diffusion models exhibit a semantic latent space, called h-space, located at the middle block of the U-Net denoiser. That space is described as responsible for high-level semantic content and is motivated by properties such as homogeneity, linearity, robustness, and consistency across timesteps. The attack therefore moves anti-customization away from pixel-space heuristics and toward internal semantic representations (Xu et al., 23 Jul 2025).

HAAD-KV narrows that focus further. Instead of updating all trainable h-space parameters, it updates only the learned weight matrices that project features into the key and value representations inside the cross-attention module. The supplementary formulation is:

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$

$$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$

and the perturbed attention becomes

$$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$

The paper’s interpretation is that cross-attention is the mechanism that aligns prompt tokens with visual content, and that the key and value projections are especially important during personalization. On that basis, HAAD-KV is presented as a more focused attack locus than broad h-space optimization or broader cross-attention attacks such as CAAT (Xu et al., 23 Jul 2025).

A concise parameter comparison reported in the paper is as follows.

Method	Trainable parameters
AdvDM	859.52M
ACE / ACE+	123.06M
CAAT	19.17M
HAAD	97.03M
HAAD-KV	5.24M

The paper states that HAAD-KV updates only about 5.24M parameters, versus 97.03M for HAAD and 19.17M for CAAT. It also notes an inconsistency in its own percentage statements: the abstract and introduction describe HAAD-KV as about $\sim 0.05\%$ relative to HAAD, whereas the supplementary also describes it as about $\sim 5\%$ of the total parameters in h-space. The parameter-count table is the clearer statement (Xu et al., 23 Jul 2025).

3. Optimization formulation and perturbation construction

The paper first describes latent diffusion training in the usual form: an image $x$ is encoded to a latent $z_0 = E(x)$, and the denoiser predicts noise under conditioning $y$. The text extraction of the latent diffusion objective is corrupted, but it is explicitly described as the standard expected squared error between sampled noise $\epsilon$ and the U-Net’s predicted noise conditioned on timestep and text (Xu et al., 23 Jul 2025).

The generic adversarial objective is stated as

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$0

For HAAD, iterative PGD updates the image perturbation while also updating h-space weights:

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$1

with the joint update written as

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$2

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$3

$$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$4

HAAD-KV “operates similarly to HAAD in terms of optimization,” but constrains the internal updates to $$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$5 and $$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$6 only (Xu et al., 23 Jul 2025).

The practical procedure is reported in eight steps: initialize $$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$7; feed $$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$8 through the latent diffusion model; compute the latent diffusion reconstruction loss; backpropagate gradients with respect to both the image and the selected trainable parameters; update selected model weights with learning rate $$W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},$$9; update $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$0; project or clip to satisfy $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$1; and output the protected image $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$2 after $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$3 steps. The attack settings used for HAAD and HAAD-KV are batch size 1, learning rate $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$4, training steps 250, prompt “photo of a sks person/painting”, token “sks” initialized with “ktn”, PGD step size $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$5, and perturbation budget $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$6 in $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$7 (Xu et al., 23 Jul 2025).

The reference model for protection generation is Stable Diffusion 1.5, experiments are run on NVIDIA A40 GPU with bf16, and transferability experiments additionally consider SD1.4, SD1.5, SD2.1, with a preliminary study on SD3. The datasets are CelebA-HQ and WikiArt, each with 200 images total: 10 identities or artists, 20 images or paintings per identity or artist (Xu et al., 23 Jul 2025).

4. Empirical evaluation

The evaluation targets two customization pipelines: LoRA+DreamBooth (LoRA+DB) and Custom Diffusion (CD). Customization hyperparameters reported are: for LoRA+DB, train steps 1000, learning rate $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$8, batch size 1, and LoRA rank 4; for CD, train steps 250, learning rate $$W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}$$9, batch size 2. Baselines are No Attack, AdvDM, ACE, ACE+, and CAAT (Xu et al., 23 Jul 2025).

The principal metrics are CLIP-SIM (CS), CLIP-IQA (CI), FDFR, and ISM. The paper states that lower CS is better for protection, higher CI means lower generated quality and hence stronger protection, higher FDFR is better for protection, and lower ISM is better for identity obfuscation. Its central quantitative summary is that HAAD-KV is best in 10 out of 12 metric-dataset combinations (Xu et al., 23 Jul 2025).

On CelebA-HQ, LoRA+DB, the reported values are:

• No Attack: CI 21.32, CS 83.13, FDFR 0.005, ISM 0.6904
• CAAT: CI 30.96, CS 72.17, FDFR 0.080, ISM 0.5547
• HAAD: CI 29.10, CS 72.06, FDFR 0.085, ISM 0.5175
• HAAD-KV: CI 31.82, CS 71.91, FDFR 0.100, ISM 0.5083

On CelebA-HQ, CD, the values are:

• No Attack: CI 20.89, CS 85.64, FDFR 0.005, ISM 0.6907
• CAAT: CI 29.31, CS 73.26, FDFR 0.185, ISM 0.5763
• HAAD: CI 29.10, CS 73.35, FDFR 0.150, ISM 0.5657
• HAAD-KV: CI 29.52, CS 72.98, FDFR 0.185, ISM 0.5606

On WikiArt, LoRA+DB, the reported values are:

• No Attack: CI 34.62, CS 64.51
• CAAT: CI 37.52, CS 58.64
• HAAD: CI 37.66, CS 58.59
• HAAD-KV: CI 37.88, CS 58.26

On WikiArt, CD, the reported values are:

• No Attack: CI 34.22, CS 66.36
• CAAT: CI 51.17, CS 59.73
• HAAD: CI 52.14, CS 59.78
• HAAD-KV: CI 52.86, CS 59.52

The paper’s summary is that HAAD is already competitive and often second-best, but HAAD-KV consistently surpasses it and all baselines (Xu et al., 23 Jul 2025).

The qualitative evidence is aligned with the metric tables. For faces, HAAD-KV causes identity drift, facial feature distortion, hair or color changes, and stylistic changes; for artwork, the main subject may be replaced or compositions become incoherent; and for CD the stylistic and semantic drift is especially pronounced. Cross-attention visualizations are presented as mechanistic evidence: No Attack yields attention tightly concentrated on relevant semantic regions, HAAD produces more dispersed and partially misaligned attention, and HAAD-KV loses structure entirely, spreading diffusely to irrelevant regions and background (Xu et al., 23 Jul 2025).

5. Robustness, transfer, and representational analysis

The paper studies purification robustness under Gaussian noise, Gaussian blur, JPEG compression, resizing, and super-resolution. Representative CLIP-IQA values are reported as follows:

• Gaussian noise $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$0: HAAD 28.71, HAAD-KV 29.15
• Gaussian blur $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$1: HAAD 33.12, HAAD-KV 33.61
• JPEG $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$2: HAAD 28.87, HAAD-KV 29.26
• resizing $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$3: HAAD 26.37, HAAD-KV 27.88
• SR: HAAD 35.71, HAAD-KV 36.84

The paper also evaluates recent purification methods, Noisy Upscaling and Impress, and states that personalization still fails after purification. For example, for HAAD-KV itself, SSIM 0.9862, PSNR 59.95, CS 71.91; for Noisy Upscaling purified, SSIM 0.3463, PSNR 23.36, CS 77.62; and for Impress purified, SSIM 0.9209, PSNR 32.84, CS 74.64 (Xu et al., 23 Jul 2025).

Prompt invariance is tested on six prompts, including “a dslr portrait of sks person,” “looking at the mirror,” “sitting on a chair,” “sitting on the floor,” “wearing glasses,” and “talking on the phone.” For the first prompt, the reported values are:

• CAAT: CI 29.48, CS 74.88, FDFR 0.060, ISM 0.5917
• HAAD: CI 29.37, CS 73.85, FDFR 0.090, ISM 0.5899
• HAAD-KV: CI 30.37, CS 72.09, FDFR 0.115, ISM 0.5682

The supplementary states that across all six prompts, HAAD and HAAD-KV are consistently best or second best, with HAAD-KV usually strongest (Xu et al., 23 Jul 2025).

Transferability across Stable Diffusion versions is assessed using perturbations from SD1.4, SD1.5, or SD2.1, then evaluating on all three. Reported CLIP-IQA values on LoRA+DreamBooth are:

• No Attack: SD1.4 18.89, SD1.5 21.32, SD2.1 19.18
• Attacker SD1.4: 27.51, 29.38, 30.36
• Attacker SD1.5: 29.32, 29.10, 30.55
• Attacker SD2.1: 29.27, 26.81, 29.62

The paper concludes that the drop under transfer is small within the SD family and also reports preliminary qualitative transfer to SD3, while explicitly stating that those results are not yet conclusive (Xu et al., 23 Jul 2025).

A noise-budget study compares $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$4, $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$5, and $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$6. On CelebA-HQ, LoRA+DB, the reported values are:

• At $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$7: HAAD CI 29.10, FDFR 0.085, ISM 0.5175; HAAD-KV CI 29.52, FDFR 0.100, ISM 0.5083
• At $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$8: HAAD CI 37.70, FDFR 0.110, ISM 0.4517; HAAD-KV CI 40.01, FDFR 0.180, ISM 0.4384
• At $$\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.$$9: HAAD CI 46.11, FDFR 0.690, ISM 0.2905; HAAD-KV CI 48.41, FDFR 0.820, ISM 0.2357

The paper interprets this as a protection–perceptibility tradeoff: protection strengthens as budget rises, but perceptibility also rises (Xu et al., 23 Jul 2025).

Imperceptibility at the main budget is supported by a user study with 26 volunteers and 30 image pairs per budget on CelebA-HQ. Reported results are:

• $\sim 0.05\%$0: Mean Z-score -0.39, STD 0.25, Error Rate 0.650
• $\sim 0.05\%$1: Mean Z-score 0.28, STD 0.34, Error Rate 0.392
• $\sim 0.05\%$2: Mean Z-score 1.23, STD 0.29, Error Rate 0.117

The paper interprets $\sim 0.05\%$3 as effectively chance-level detectability (Xu et al., 23 Jul 2025).

The supplementary also provides a representation-level analysis of h-space using PCA and cosine similarity between top principal components of clean and HAAD-KV-protected images. Reported values are:

• $\sim 0.05\%$4: top-5 0.2961, top-10 0.1413, top-20 0.0668, top-50 0.0243
• $\sim 0.05\%$5: 0.2171, 0.1111, 0.0603, 0.0245
• $\sim 0.05\%$6: 0.1382, 0.0672, 0.0351, 0.0135

These low values are presented as evidence that HAAD-KV induces substantial semantic drift in h-space (Xu et al., 23 Jul 2025).

6. Nomenclature, related methods, and limitations

The term “HAAD-KV” can be misleading because the suffix “KV” has an established and unrelated usage in transformer inference for key-value cache compression or eviction. In HAAD-KV, however, “KV parameters” refers to the key and value projection parameters of the cross-attention layer within the h-space block of a diffusion U-Net, not to an autoregressive transformer’s persistent KV cache (Xu et al., 23 Jul 2025). This distinction matters because several contemporaneous methods with superficially similar names address different problems entirely.

In long-context LLM inference, Ada-KV studies head-wise adaptive budget allocation for KV-cache eviction and formulates the objective in terms of preserving attention mass under a fixed cache budget (Feng et al., 2024). Task-KV is a training-free, inference-time method that dynamically identifies semantically distinctive attention heads and assigns differentiated KV cache budgets across tasks (He et al., 25 Jan 2025). R-KV targets redundancy-aware decode-time KV compression for reasoning models and combines attention-based importance with redundancy penalties over key similarity (Cai et al., 30 May 2025). KV-Compress implements variable compression rates per attention head within a PagedAttention framework so that head-adaptive eviction yields physical memory savings rather than only theoretical compression (Rehg, 2024). HeadKV addresses autoregressive image generation by classifying heads as local or global and then applying asymmetric per-head cache budgets (Liang et al., 20 May 2026). These methods belong to the systems and inference literature on transformer KV caches, whereas HAAD-KV belongs to the adversarial protection literature for diffusion-model personalization.

Within anti-customization research, the paper positions HAAD-KV against AdvDM, ACE, ACE+, and CAAT. Its main conceptual distinction is “semantically grounded anti-customization by attacking the h-space, then refining further to the KV substructure of cross-attention where prompt-concept binding actually happens” (Xu et al., 23 Jul 2025). A plausible implication is that the novelty lies less in a new adversarial objective than in a tighter localization of the perturbation process to a semantically consequential subset of the model.

The reported limitations are explicit. Main experiments are on U-Net-based Stable Diffusion family backbones, especially SD1.5; transfer to SD3 or DiT-style backbones is only preliminary and not conclusive. Evaluation is focused on few-shot personalization methods such as LoRA+DB and Custom Diffusion. Purification is “not impossible”: purification changes image quality metrics and may partially reduce the effect, although the paper states that personalization still fails. The threat model assumes that the attacker uses the protected images directly for standard personalization and does not fully study adaptive countermeasures. The supplementary theoretical account is presented as intuition rather than as a formal proof, and the paper notes that some notation in the extracted text is noisy (Xu et al., 23 Jul 2025).

Taken together, HAAD-KV is best described as a semantically targeted anti-customization attack for diffusion models that generates imperceptible perturbations by restricting h-space optimization to the key and value projections of cross-attention. Its central technical claim is that this restriction improves both protection strength and computational efficiency relative to broader h-space or cross-attention attacks, while its principal empirical claim is that, at the strict budget $\sim 0.05\%$7, it is the strongest method in nearly all reported settings while updating only 5.24M trainable parameters (Xu et al., 23 Jul 2025).