---
title: 'HAAD-KV: Adversarial Attack for Diffusion Models'
url: https://www.emergentmind.com/topics/haad-kv
type: topic
---

# HAAD-KV: Adversarial Attack for Diffusion Models

HAAD-KV denotes the more efficient variant of HAAD, an anti-customization adversarial attack for diffusion models that protects personal or private images from unauthorized few-shot personalization by generating imperceptibly perturbed training images that poison subsequent customization. In the formulation reported in “An h-space Based Adversarial Attack for Protection Against Few-shot Personalization,” HAAD attacks the diffusion model’s “h-space,” defined operationally as the deep features from the middle block of the U-Net denoiser, whereas HAAD-KV restricts perturbation construction to the key and value projection parameters of the cross-attention layer within that h-space [2507.17554]. The method is evaluated against personalization pipelines such as DreamBooth, LoRA+DreamBooth, and Custom Diffusion, with the stated objective that a model fine-tuned on protected images fails to bind the user-specific token to the correct visual concept.

## 1. Definition and threat model

HAAD stands for **h-space based Adversarial Attack for Diffusion models**. It is described as an untargeted anti-customization attack that generates an adversarially perturbed image $x' = x + \delta$ by maximizing the latent diffusion reconstruction loss while updating only the h-space parameters of the diffusion U-Net during perturbation construction. HAAD-KV is a refinement of HAAD: the overall strategy is unchanged, but the trainable subset used during perturbation generation is reduced to the **K** and **V** projection parameters of cross-attention within the h-space [2507.17554].

The problem setting is few-shot personalization of text-to-image diffusion models. In that setting, an attacker obtains a small number of user images and fine-tunes a diffusion model so that a user-specific token, such as “sks,” becomes associated with a person, artwork, or concept. HAAD-KV is positioned as a data-poisoning defense against that procedure. The operational pipeline is: clean user images are perturbed; the attacker fine-tunes a personalization model on those protected images; and the resulting personalized model produces outputs that are semantically inconsistent, structurally degraded, or identity-mismatched when prompted with the learned token [2507.17554].

The method assumes that the defender can prepare protected images using a reference diffusion backbone and that the attacker later uses those images directly for standard personalization. The defense does not require modifying the attacker’s training loop. A plausible implication is that HAAD-KV is most directly applicable where image owners control the images that enter downstream customization datasets, rather than the fine-tuning procedure itself.

## 2. h-space and the KV-restricted mechanism

The conceptual basis of HAAD-KV is the paper’s claim that diffusion models exhibit a semantic latent space, called **h-space**, located at the middle block of the U-Net denoiser. That space is described as responsible for high-level semantic content and is motivated by properties such as homogeneity, linearity, robustness, and consistency across timesteps. The attack therefore moves anti-customization away from pixel-space heuristics and toward internal semantic representations [2507.17554].

HAAD-KV narrows that focus further. Instead of updating all trainable h-space parameters, it updates only the learned weight matrices that project features into the key and value representations inside the cross-attention module. The supplementary formulation is:

$$
W'_K=W_K+l \nabla_{W_K} \mathcal{L}_{\text{LDM}(x)},
$$

$$
W'_V=W_V+l \nabla_{W_V} \mathcal{L}_{\text{LDM}(x)}
$$

and the perturbed attention becomes

$$
\text{Attention}(Q, K', V') = \text{softmax}\left( \frac{QK'^\top}{\sqrt{d}} \right)V'.
$$

The paper’s interpretation is that cross-attention is the mechanism that aligns prompt tokens with visual content, and that the key and value projections are especially important during personalization. On that basis, HAAD-KV is presented as a more focused attack locus than broad h-space optimization or broader cross-attention attacks such as CAAT [2507.17554].

A concise parameter comparison reported in the paper is as follows.

| Method | Trainable parameters |
|---|---:|
| AdvDM | 859.52M |
| ACE / ACE+ | 123.06M |
| CAAT | 19.17M |
| HAAD | 97.03M |
| HAAD-KV | 5.24M |

The paper states that HAAD-KV updates only about **5.24M parameters**, versus **97.03M** for HAAD and **19.17M** for CAAT. It also notes an inconsistency in its own percentage statements: the abstract and introduction describe HAAD-KV as about **$\sim 0.05\%$** relative to HAAD, whereas the supplementary also describes it as about **$\sim 5\%$ of the total parameters in h-space**. The parameter-count table is the clearer statement [2507.17554].

## 3. Optimization formulation and perturbation construction

The paper first describes latent diffusion training in the usual form: an image $x$ is encoded to a latent $z_0 = E(x)$, and the denoiser predicts noise under conditioning $y$. The text extraction of the latent diffusion objective is corrupted, but it is explicitly described as the standard expected squared error between sampled noise $\epsilon$ and the U-Net’s predicted noise conditioned on timestep and text [2507.17554].

The generic adversarial objective is stated as

$$
\underset{x'}{argmax}\ \mathcal{L}_{\phi}(x')\; \quad s.t. \ ||x'-x||_{\infty} \leq \eta.
$$

For HAAD, iterative PGD updates the image perturbation while also updating h-space weights:

$$
x^{t+1} = \Pi_{\eta}\left(x^{t} + \alpha \cdot \text{sign} \left( \nabla_{x^t} \mathcal{L}_{\text{LDM}(x^t, W^t_{h})} \right)\right),
$$

with the joint update written as

$$
\nabla_{W_h}, \nabla_x \gets \nabla_{LDM}(W_h, x+\delta),
$$

$$
W_h \gets W_h - l \nabla_{W_h},
$$

$$
\delta \gets \delta + \alpha \, \mathrm{sgn}(\nabla_x).
$$

HAAD-KV “operates similarly to HAAD in terms of optimization,” but constrains the internal updates to $W_K$ and $W_V$ only [2507.17554].

The practical procedure is reported in eight steps: initialize $\delta$; feed $x+\delta$ through the latent diffusion model; compute the latent diffusion reconstruction loss; backpropagate gradients with respect to both the image and the selected trainable parameters; update selected model weights with learning rate $l$; update $\delta \leftarrow \delta + \alpha \,\mathrm{sgn}(\nabla_x)$; project or clip to satisfy $\|\delta\|_\infty \le \eta$; and output the protected image $x' = x+\delta$ after $N$ steps. The attack settings used for HAAD and HAAD-KV are batch size **1**, learning rate **$1 \times 10^{-5}$**, training steps **250**, prompt **“photo of a sks person/painting”**, token **“sks”** initialized with **“ktn”**, PGD step size **$\alpha = 5 \times 10^{-3}$**, and perturbation budget **$\eta = 4/255$** in $\ell_\infty$ [2507.17554].

The reference model for protection generation is **Stable Diffusion 1.5**, experiments are run on **NVIDIA A40 GPU** with **bf16**, and transferability experiments additionally consider **SD1.4**, **SD1.5**, **SD2.1**, with a preliminary study on **SD3**. The datasets are **CelebA-HQ** and **WikiArt**, each with **200 images total**: 10 identities or artists, 20 images or paintings per identity or artist [2507.17554].

## 4. Empirical evaluation

The evaluation targets two customization pipelines: **LoRA+DreamBooth (LoRA+DB)** and **Custom Diffusion (CD)**. Customization hyperparameters reported are: for LoRA+DB, train steps **1000**, learning rate **$5 \times 10^{-5}$**, batch size **1**, and LoRA rank **4**; for CD, train steps **250**, learning rate **$1 \times 10^{-5}$**, batch size **2**. Baselines are **No Attack**, **AdvDM**, **ACE**, **ACE+**, and **CAAT** [2507.17554].

The principal metrics are **CLIP-SIM (CS)**, **CLIP-IQA (CI)**, **FDFR**, and **ISM**. The paper states that lower **CS** is better for protection, higher **CI** means lower generated quality and hence stronger protection, higher **FDFR** is better for protection, and lower **ISM** is better for identity obfuscation. Its central quantitative summary is that HAAD-KV is best in **10 out of 12 metric-dataset combinations** [2507.17554].

On **CelebA-HQ, LoRA+DB**, the reported values are:

- No Attack: CI **21.32**, CS **83.13**, FDFR **0.005**, ISM **0.6904**
- CAAT: CI **30.96**, CS **72.17**, FDFR **0.080**, ISM **0.5547**
- HAAD: CI **29.10**, CS **72.06**, FDFR **0.085**, ISM **0.5175**
- HAAD-KV: CI **31.82**, CS **71.91**, FDFR **0.100**, ISM **0.5083**

On **CelebA-HQ, CD**, the values are:

- No Attack: CI **20.89**, CS **85.64**, FDFR **0.005**, ISM **0.6907**
- CAAT: CI **29.31**, CS **73.26**, FDFR **0.185**, ISM **0.5763**
- HAAD: CI **29.10**, CS **73.35**, FDFR **0.150**, ISM **0.5657**
- HAAD-KV: CI **29.52**, CS **72.98**, FDFR **0.185**, ISM **0.5606**

On **WikiArt, LoRA+DB**, the reported values are:

- No Attack: CI **34.62**, CS **64.51**
- CAAT: CI **37.52**, CS **58.64**
- HAAD: CI **37.66**, CS **58.59**
- HAAD-KV: CI **37.88**, CS **58.26**

On **WikiArt, CD**, the reported values are:

- No Attack: CI **34.22**, CS **66.36**
- CAAT: CI **51.17**, CS **59.73**
- HAAD: CI **52.14**, CS **59.78**
- HAAD-KV: CI **52.86**, CS **59.52**

The paper’s summary is that HAAD is already competitive and often second-best, but HAAD-KV consistently surpasses it and all baselines [2507.17554].

The qualitative evidence is aligned with the metric tables. For faces, HAAD-KV causes identity drift, facial feature distortion, hair or color changes, and stylistic changes; for artwork, the main subject may be replaced or compositions become incoherent; and for CD the stylistic and semantic drift is especially pronounced. Cross-attention visualizations are presented as mechanistic evidence: No Attack yields attention tightly concentrated on relevant semantic regions, HAAD produces more dispersed and partially misaligned attention, and HAAD-KV loses structure entirely, spreading diffusely to irrelevant regions and background [2507.17554].

## 5. Robustness, transfer, and representational analysis

The paper studies purification robustness under **Gaussian noise**, **Gaussian blur**, **JPEG compression**, **resizing**, and **super-resolution**. Representative **CLIP-IQA** values are reported as follows:

- Gaussian noise $\sigma=8$: HAAD **28.71**, HAAD-KV **29.15**
- Gaussian blur $k=5$: HAAD **33.12**, HAAD-KV **33.61**
- JPEG $Q=70$: HAAD **28.87**, HAAD-KV **29.26**
- resizing $0.5\times$: HAAD **26.37**, HAAD-KV **27.88**
- SR: HAAD **35.71**, HAAD-KV **36.84**

The paper also evaluates recent purification methods, **Noisy Upscaling** and **Impress**, and states that personalization still fails after purification. For example, for HAAD-KV itself, **SSIM 0.9862**, **PSNR 59.95**, **CS 71.91**; for Noisy Upscaling purified, **SSIM 0.3463**, **PSNR 23.36**, **CS 77.62**; and for Impress purified, **SSIM 0.9209**, **PSNR 32.84**, **CS 74.64** [2507.17554].

Prompt invariance is tested on six prompts, including “a dslr portrait of sks person,” “looking at the mirror,” “sitting on a chair,” “sitting on the floor,” “wearing glasses,” and “talking on the phone.” For the first prompt, the reported values are:

- CAAT: CI **29.48**, CS **74.88**, FDFR **0.060**, ISM **0.5917**
- HAAD: CI **29.37**, CS **73.85**, FDFR **0.090**, ISM **0.5899**
- HAAD-KV: CI **30.37**, CS **72.09**, FDFR **0.115**, ISM **0.5682**

The supplementary states that across all six prompts, HAAD and HAAD-KV are consistently best or second best, with HAAD-KV usually strongest [2507.17554].

Transferability across Stable Diffusion versions is assessed using perturbations from **SD1.4**, **SD1.5**, or **SD2.1**, then evaluating on all three. Reported **CLIP-IQA** values on LoRA+DreamBooth are:

- No Attack: SD1.4 **18.89**, SD1.5 **21.32**, SD2.1 **19.18**
- Attacker SD1.4: **27.51**, **29.38**, **30.36**
- Attacker SD1.5: **29.32**, **29.10**, **30.55**
- Attacker SD2.1: **29.27**, **26.81**, **29.62**

The paper concludes that the drop under transfer is small within the SD family and also reports preliminary qualitative transfer to **SD3**, while explicitly stating that those results are not yet conclusive [2507.17554].

A noise-budget study compares $\eta = 4/255$, $8/255$, and $16/255$. On **CelebA-HQ, LoRA+DB**, the reported values are:

- At $4/255$: HAAD CI **29.10**, FDFR **0.085**, ISM **0.5175**; HAAD-KV CI **29.52**, FDFR **0.100**, ISM **0.5083**
- At $8/255$: HAAD CI **37.70**, FDFR **0.110**, ISM **0.4517**; HAAD-KV CI **40.01**, FDFR **0.180**, ISM **0.4384**
- At $16/255$: HAAD CI **46.11**, FDFR **0.690**, ISM **0.2905**; HAAD-KV CI **48.41**, FDFR **0.820**, ISM **0.2357**

The paper interprets this as a protection–perceptibility tradeoff: protection strengthens as budget rises, but perceptibility also rises [2507.17554].

Imperceptibility at the main budget is supported by a user study with **26 volunteers** and **30 image pairs per budget** on CelebA-HQ. Reported results are:

- $4/255$: Mean Z-score **-0.39**, STD **0.25**, Error Rate **0.650**
- $8/255$: Mean Z-score **0.28**, STD **0.34**, Error Rate **0.392**
- $16/255$: Mean Z-score **1.23**, STD **0.29**, Error Rate **0.117**

The paper interprets $4/255$ as effectively chance-level detectability [2507.17554].

The supplementary also provides a representation-level analysis of h-space using PCA and cosine similarity between top principal components of clean and HAAD-KV-protected images. Reported values are:

- $4/255$: top-5 **0.2961**, top-10 **0.1413**, top-20 **0.0668**, top-50 **0.0243**
- $8/255$: **0.2171**, **0.1111**, **0.0603**, **0.0245**
- $16/255$: **0.1382**, **0.0672**, **0.0351**, **0.0135**

These low values are presented as evidence that HAAD-KV induces substantial semantic drift in h-space [2507.17554].

## 6. Nomenclature, related methods, and limitations

The term “HAAD-KV” can be misleading because the suffix “KV” has an established and unrelated usage in transformer inference for **key-value cache** compression or eviction. In HAAD-KV, however, “KV parameters” refers to the **key and value projection parameters of the cross-attention layer within the h-space block** of a diffusion U-Net, not to an autoregressive transformer’s persistent KV cache [2507.17554]. This distinction matters because several contemporaneous methods with superficially similar names address different problems entirely.

In long-context LLM inference, **Ada-KV** studies head-wise adaptive budget allocation for KV-cache eviction and formulates the objective in terms of preserving attention mass under a fixed cache budget [2407.11550]. **Task-KV** is a training-free, inference-time method that dynamically identifies semantically distinctive attention heads and assigns differentiated KV cache budgets across tasks [2501.15113]. **R-KV** targets redundancy-aware decode-time KV compression for reasoning models and combines attention-based importance with redundancy penalties over key similarity [2505.24133]. **KV-Compress** implements variable compression rates per attention head within a PagedAttention framework so that head-adaptive eviction yields physical memory savings rather than only theoretical compression [2410.00161]. **HeadKV** addresses autoregressive image generation by classifying heads as local or global and then applying asymmetric per-head cache budgets [2605.20600]. These methods belong to the systems and inference literature on transformer KV caches, whereas HAAD-KV belongs to the adversarial protection literature for diffusion-model personalization.

Within anti-customization research, the paper positions HAAD-KV against **AdvDM**, **ACE**, **ACE+**, and **CAAT**. Its main conceptual distinction is “semantically grounded anti-customization by attacking the h-space, then refining further to the KV substructure of cross-attention where prompt-concept binding actually happens” [2507.17554]. A plausible implication is that the novelty lies less in a new adversarial objective than in a tighter localization of the perturbation process to a semantically consequential subset of the model.

The reported limitations are explicit. Main experiments are on U-Net-based Stable Diffusion family backbones, especially **SD1.5**; transfer to **SD3** or DiT-style backbones is only preliminary and not conclusive. Evaluation is focused on few-shot personalization methods such as **LoRA+DB** and **Custom Diffusion**. Purification is “not impossible”: purification changes image quality metrics and may partially reduce the effect, although the paper states that personalization still fails. The threat model assumes that the attacker uses the protected images directly for standard personalization and does not fully study adaptive countermeasures. The supplementary theoretical account is presented as intuition rather than as a formal proof, and the paper notes that some notation in the extracted text is noisy [2507.17554].

Taken together, HAAD-KV is best described as a semantically targeted anti-customization attack for diffusion models that generates imperceptible perturbations by restricting h-space optimization to the key and value projections of cross-attention. Its central technical claim is that this restriction improves both protection strength and computational efficiency relative to broader h-space or cross-attention attacks, while its principal empirical claim is that, at the strict budget $\eta = 4/255$, it is the strongest method in nearly all reported settings while updating only **5.24M** trainable parameters [2507.17554].

Source: https://www.emergentmind.com/topics/haad-kv