---
title: Identity Guidance in Diffusion Models
url: https://www.emergentmind.com/topics/identity-guidance
type: topic
---

# Identity Guidance in Diffusion Models

Identity guidance in diffusion models denotes a family of conditioning and control mechanisms that steer generation toward a specified identity-related objective while constraining other semantic factors. In "Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection," identity guidance is defined as a gradient-based control signal injected into the reverse diffusion process in latent space so that a generated face is recognized as a chosen target identity by face recognition systems while remaining visually natural and structurally close to the source face [2504.21646]. More broadly, recent work uses the same term for substantially different operations, including adaptive guidance origins for identity-preserving editing, adaptive negative conditioning for identity-consistent synthesis, decoupled subject/context guidance for personalization, and training-centric disentanglement for anonymization [2602.03826][2603.29569][2607.00766][2510.24213].

## 1. Formal task and threat model

In DiffAIM, the task is adversarial identity manipulation for facial privacy protection. Given a source face \(I^{src}\) and a target face \(I^{tgt}\), the objective is to generate an adversarial image \(I^{adv}\) that preserves human-perceived identity and visual naturalness with respect to \(I^{src}\), yet impersonates the target identity for machine face recognition systems [2504.21646]. For an FR model \(\mathcal{M}\) with embedding similarity \(Sim(\cdot,\cdot)\), the target conditions are:
- for verification,
  \[
  Sim(\mathcal{M}(I^{adv}), \mathcal{M}(I^{tgt})) > \tau
  \]
- for identification,
  \[
  Sim(\mathcal{M}(I^{adv}), \mathcal{M}(I^{tgt})) > Sim(\mathcal{M}(I^{adv}), \mathcal{M}(I^{src}))
  \]

The optimization is posed as
\[
\begin{aligned}
\max_{I^{adv}} \; & \big[ Sim(\mathcal{M}(I^{adv}), \mathcal{M}(I^{tgt})) - Sim(\mathcal{M}(I^{adv}), \mathcal{M}(I^{src})) \big] \\
\text{s.t. } & \mathcal{H}(I^{adv}, I^{src}) \le \epsilon,
\end{aligned}
\]
where \(\mathcal{H}\) measures unnaturalness relative to the source [2504.21646].

The threat model is transfer-based and targets black-box FR systems at test time. The attacked systems include IRSE50, IR152, FaceNet, MobileFace, Face++, and Aliyun, treated as black-boxes during attack execution. Training uses a set of white-box surrogate FR models \(\{\mathcal{M}_i\}\) with available gradients, and the attack mode is impersonation rather than simple obfuscation [2504.21646]. Within that setting, identity guidance occupies a precise role: it modifies the diffusion trajectory so that the generated sample exhibits identity convergence toward the target, semantic divergence from the source, and structure preservation.

This formulation differs from identity-consistency objectives in synthesis or restoration. For example, AdaptDiff uses positive and negative identity conditions \(p^+\) and \(p^-\) to improve inter-class separability in identity-conditioned face synthesis [2603.29569], whereas ID\(^2\)Face learns an identity-decoupled latent space for anonymization and samples a new identity vector at inference time [2510.24213]. DiffAIM instead uses identity guidance to induce targeted machine-level impersonation under a black-box transfer setting [2504.21646].

## 2. Latent diffusion framework and the definition of identity guidance

DiffAIM uses a latent diffusion model with a VAE encoder \(\mathcal{E}\) and decoder \(\mathcal{D}\). A source face \(I^{src}\) is encoded into a latent
\[
x_0 = \mathcal{E}(I^{src}),
\]
and diffusion is performed in latent space rather than pixel space in order to leverage the natural image prior and the efficiency of latent diffusion models [2504.21646]. The forward process is
\[
x_t = \sqrt{\bar{\alpha}_t} \, x_0 + \sqrt{1 - \bar{\alpha}_t} \, \tilde{\epsilon}_t,\quad t=1,\dots,T,
\]
with \(\tilde{\epsilon}_t \sim \mathcal{N}(0,I)\), and the standard reverse step is
\[
x_{t-1} = \hat{\mu}_t(x_t) + \sigma_t z_t,\quad t=T,\dots,1.
\]

Identity guidance is defined on the adversarial latent \(\hat{x}_t\). At each diffusion step, DiffAIM predicts the corresponding clean latent
\[
\hat{x}_0^{t} = \frac{\hat{x}_t - \sqrt{1-\bar{\alpha}_t} \hat{\epsilon}_\theta(\hat{x}_t)}{\sqrt{\bar{\alpha}_t}},
\]
decodes it into image space,
\[
I^{(t)}_{adv} = \mathcal{D}(\hat{x}_0^{t}),
\]
and forms a total loss over target identity, source divergence, and structure [2504.21646]. The guidance vector is then
\[
\mathcal{G}_t = \nabla_{\hat{x}_t} \mathcal{L}_{total}\big(\mathcal{D}(\hat{x}_0^{t}), I^{tgt}, I^{src}\big).
\]
This gradient is the paper’s adversarial identity guidance.

Facial identity is represented by an ensemble of pretrained FR models \(\{\mathcal{M}_i\}_{i=1}^{N_m}\), such as IRSE50, IR152, FaceNet, and MobileFace. These models output embedding vectors, and similarity is cosine similarity:
\[
Sim(v_1, v_2) = \cos(v_1, v_2) = \frac{v_1^\top v_2}{\|v_1\|\|v_2\|}.
\]
The target identity is represented directly by target embeddings \(\mathcal{M}_i(I^{tgt})\), without a classification head or cross-entropy loss [2504.21646].

In this formulation, identity guidance is neither text conditioning nor class-label conditioning. That distinction is central to the paper’s conceptual framing. Whereas classifier guidance uses \(\nabla_{x_t}\log p(y|x_t)\) and classifier-free guidance interpolates conditional and unconditional score estimates, DiffAIM uses gradients of a customized identity-oriented objective computed from decoded images and feature spaces external or internal to the diffusion backbone [2504.21646]. This suggests that, in DiffAIM, identity is treated as a metric relation in embedding space rather than as a discrete label.

## 3. Loss design: identity convergence, semantic divergence, and structure preservation

The central adversarial term in DiffAIM is not a naive difference between FR similarity to target and FR similarity to source. The paper notes that an explicit FR-based divergence term tends to create artifacts, and therefore decouples the objective into identity convergence in FR embedding space and semantic divergence in U-Net feature space [2504.21646].

The semantic divergence term is defined using the deepest or intermediate features from the frozen diffusion U-Net, denoted \(\mathcal{U}_\theta^{\text{mid}(\cdot)}\). Let \(x_t\) be the benign inverted latent and \(\hat{x}_t\) the adversarial latent. Then
\[
\mathcal{L}_{sem-div}
= -\, Sim\Big(\mathcal{U}_\theta^{\text{mid}(\hat{x}_t)},
               \mathcal{U}_\theta^{\text{mid}(x_t)}\Big).
\]
This pushes the adversarial trajectory away from the benign trajectory in the learned generative semantic space while remaining visually plausible [2504.21646].

Identity convergence is defined for each FR model by
\[
\mathcal{L}_{id-conv,i}
= Sim\big(\mathcal{M}_i(I^{(t)}_{adv}), \mathcal{M}_i(I^{tgt})\big).
\]
Over an ensemble, the adversarial loss becomes
\[
\mathcal{L}_{adv} = \mathcal{L}_{sem-div} + \sum_{i=1}^{N_m} w_i \cdot Score_i,
\]
where
\[
Score_i = Sim\big(\mathcal{M}_i(I^{(t)}_{adv}), \mathcal{M}_i(I^{tgt})\big).
\]
The ensemble weights are adaptively updated so that harder surrogate models receive larger weights:
\[
w_i = \frac{e^{1 - Score_i}}{\sum_{j=1}^{N_m} e^{1 - Score_j}}.
\]
This adaptive ensemble is intended to improve transferability to unknown black-box FR systems by emphasizing models that are currently least fooled [2504.21646].

Structure-preserving regularization is defined through U-Net self-attention maps. Let \(\mathcal{S}\) denote the self-attention layers and \(\mathcal{A}_j(x)\) the attention map at layer \(j\) for latent \(x\). Then
\[
\mathcal{L}_{str}
= -\sum_{j \in \mathcal{S}}
\big\|
\mathcal{A}_j(\hat{x}_t) - \mathcal{A}_j(x_t)
\big\|_2^2.
\]
Although written with a negative sign in the paper’s notation, it is minimized in practice so as to keep attention differences small [2504.21646]. The total loss used for identity guidance is
\[
\mathcal{L}_{total} = \mathcal{L}_{adv} + \lambda \mathcal{L}_{str},
\]
with \(\lambda = 0.1\) in the reported implementation.

The resulting decomposition is technically specific. Identity is pulled toward the target in FR space, pushed away from the source in U-Net semantic space, and constrained in self-attention space to preserve structure. A plausible implication is that DiffAIM treats realism as a property of the generative manifold rather than as an external perceptual constraint.

## 4. Injection into reverse diffusion and identity-sensitive scheduling

DiffAIM integrates identity guidance directly into the reverse diffusion update. Starting from the benign reverse step
\[
x_{t-1} = \hat{\mu}_t(x_t) + \sigma_t z_t,
\]
the adversarial process maintains an adversarial latent \(\hat{x}_t\) and first computes the base denoising step with consistent noise:
\[
x_{t-1} = \hat{\mu}_t(\hat{x}_t) + \sigma_t z_t.
\]
It then performs an inner loop of \(N_a\) gradient steps per timestep to accumulate a bounded guidance vector [2504.21646].

The accumulated guidance update is
\[
\mathcal{G}_t^{k}
= \Pi_{\kappa}\big(\mathcal{G}_t^{k-1}
  + \eta \cdot \nabla_{\hat{x}_t} \mathcal{L}_{total}\big),
\]
initialized with \(\mathcal{G}_t^0=0\), where \(\eta\) is the step size and \(\Pi_\kappa\) projects into an \(\ell_2\)-ball of radius \(\kappa\). After \(N_a\) inner iterations,
\[
\hat{x}_{t-1} = x_{t-1} + \mathcal{G}_t^{N_a},
\]
or equivalently,
\[
\hat{x}_{t-1}
= \hat{\mu}_t(\hat{x}_t) + \sigma_t z_t + \mathcal{G}_t.
\]
The paper explicitly relates this to classifier guidance in spirit, but the guidance objective and gradients are defined in latent space by the custom adversarial loss rather than by class probabilities [2504.21646].

A further design choice is identity-sensitive timestep truncation. DiffAIM reports that applying guidance at all timesteps harms global structure, because early timesteps define low-frequency global structure while later timesteps refine high-frequency identity-sensitive regions. The method therefore runs benign reverse diffusion from \(T\) down to a truncated timestep \(t_s\), and applies adversarial identity guidance only from \(t_s\) down to \(1\) [2504.21646]. The reported optimal regime is \(t_s \in [0.2T, 0.3T]\); with \(T=100\), the implementation uses \(t_s=20\).

This scheduling choice places identity guidance in later denoising phases, where FR models’ Grad-CAM attention becomes localized on identity-relevant regions such as eyes, nose, and mouth [2504.21646]. Similar timestep asymmetry appears in other identity-guidance formulations. AdaptDiff increases negative identity guidance toward later sampling steps because early steps contain little identity information, while later steps benefit from stronger repulsive conditioning relative to negative identities [2603.29569]. This suggests a broader pattern: identity-sensitive control in diffusion models is often most effective after coarse structure has been established.

## 5. Empirical behavior, transferability, and practical trade-offs

DiffAIM reports that, on verification tasks, average attack success rate over IRSE50, IR152, FaceNet, and MobileFace on CelebA-HQ and LADN improves to \(86.13\%\), compared with \(81.27\%\) for GIFT and \(77.88\%\) for DiffAM [2504.21646]. On identification tasks, average Rank-1 targeted ASR is \(35.0\%\) and Rank-5 is \(84.2\%\), and on commercial APIs Face++ and Aliyun the method attains the highest and most stable confidence scores, around \(72\) and \(55\) respectively [2504.21646]. The paper further reports that adding the U-Net-based semantic divergence term improves robustness and ASR.

The implementation uses Stable Diffusion with DDIM sampling and \(T=100\) steps. The latent is the standard Stable Diffusion VAE latent, exemplified as \(4 \times 64 \times 64\) for \(512 \times 512\) images. Guidance hyperparameters include \(t_s=20\), \(N_a=10\) inner-loop iterations per guided timestep, step size \(\eta=3\), and structure regularization weight \(\lambda=0.1\) [2504.21646].

The reported trade-offs are explicit. Larger guidance strength, larger \(\kappa\), or larger \(t_s\) improve attack success rate but risk artifacts and structural drift; smaller or later guidance improves visual quality but reduces attack success [2504.21646]. Timing is also dataset- and model-dependent, and the method incurs substantial computation cost because it performs multiple inner-loop gradient steps at many timesteps.

The same article reports that removing \(\mathcal{L}_{str}\) increases artifacts and worsens PSNR, SSIM, and FID, emphasizing that identity guidance alone is not sufficient for naturalness [2504.21646]. A plausible implication is that the empirical strength of DiffAIM derives less from any single loss term than from the coordination of metric-space target attraction, semantic-space source repulsion, attention-map stabilization, and late-stage scheduling.

## 6. Relation to other identity-guidance formulations

The term identity guidance is used more broadly in recent diffusion and flow literature, but with different meanings and technical implementations. In AdaOr, identity guidance appears as an identity-conditioned adaptive origin for editing: a special token \((\text{id})\) is trained to reconstruct the input, and the guidance origin is interpolated between the null instruction and the identity instruction as a function of edit strength \(a\) [2602.03826]. The central update is
\[
\epsilon_{w,a}(z_t; c_I, c_T, t)
= O(a)
+ a \cdot w \big( \epsilon(z_t; c_I, c_T, t) - \epsilon(z_t; c_I, \varnothing, t) \big),
\]
with \(O(a)\) defined from the null and identity predictions. Here identity guidance preserves the source under continuous editing, rather than inducing impersonation.

In AdaptDiff, identity guidance is implemented through positive and negative identity conditions in identity-conditioned synthesis:
\[
\hat{\epsilon}_\theta(x_t, t, p^+, p^-)
= (1 + w_t)\,\epsilon_\theta(x_t,t,p^+) - w_t\,\epsilon_\theta(x_t,t,p^-),
\]
with a linear schedule
\[
w_t = w_{\max}\left(1 - \frac{t}{T}\right).
\]
This mechanism is designed to improve inter-class separability while preserving intra-class variation in synthetic face datasets [2603.29569].

In DeGu, identity guidance is decoupled from context guidance by routing fidelity and editability through separate conditioning streams:
\[
\tilde{\boldsymbol{\epsilon}(\mathbf{z}_t)}
= \boldsymbol{\epsilon}(\mathbf{z}_t)
+ \gamma_e \big(\boldsymbol{\epsilon}(\mathbf{z}_t, t, c_e) - \boldsymbol{\epsilon}(\mathbf{z}_t)\big)
+ \gamma_f \big(\boldsymbol{\epsilon}(\mathbf{z}_t, t, c_f) - \boldsymbol{\epsilon}(\mathbf{z}_t)\big),
\]
with an optional spatial mask \(\mathbf{M}_t\) that confines fidelity guidance to subject regions [2607.00766]. In that formulation, identity guidance is a plug-and-play fidelity stream for personalization.

Training-centric variants also exist. ID\(^2\)Face treats identity as an explicit latent variable in an identity-decoupled anonymization framework and fuses identity-aware and non-identity-aware features via the Identity-Guided Latent Harmonizer:
\[
f_{i+1} = m_i \cdot f_i^{\text{id}} + (1 - m_i)\cdot f_i^{\text{non-id}},
\]
while Orthogonal Identity Mapping further enforces orthogonality between source and sampled identity vectors in the learned identity space [2510.24213]. In DiffFace, identity guidance is externalized as a face-recognition energy term during diffusion sampling,
\[
\mathcal{G}_{\mathrm{id}}
= 1 - \cos\Big( \mathcal{D}_I(\mathbf{x}_{\mathrm{src}}),\; \mathcal{D}_I(\widehat{\mathbf{x}_0})\Big),
\]
combined with semantic and gaze guidance for face swapping [2212.13344].

These comparisons clarify a common misconception: identity guidance is not a single standardized operator. In DiffAIM it denotes adversarial latent-space control for targeted impersonation against FR systems [2504.21646]; in editing it often denotes identity preservation against undesired edits [2602.03826][2607.00766]; in synthesis it may refer to positive or negative identity conditioning [2603.29569]; and in anonymization it may be built into the latent factorization itself [2510.24213].

## 7. Limitations, broader significance, and open directions

DiffAIM identifies timing sensitivity, surrogate-model dependence, and computational cost as its main limitations. Guidance applied too early harms structure, guidance applied too late weakens attacks, and the optimal truncation timestep depends on the dataset and model [2504.21646]. Because the attack is transfer-based, distribution shifts or substantially different FR architectures may reduce transferability. The need for multiple gradient refinements at many diffusion steps also makes the method expensive.

The paper suggests several future directions, including more adaptive or learned timestep schedules for guidance, better identity-space regularizers that further decouple human-perceived identity from FR identity, and extensions to more controllable attributes such as explicit pose or expression preservation and multi-target impersonation [2504.21646]. These proposals align with broader trends in the literature. AdaOr generalizes the notion of an identity origin into a wider class of semantic origin tokens [2602.03826]. DeGu motivates more explicit conditional independence between identity and context features [2607.00766]. TIGER, in face video restoration, combines an identity prior, a geometry prior, and a generative prior to control identity fidelity and temporal stability jointly [2606.24336]. Lookahead Anchoring reframes future keyframes as directional beacons that preserve identity in autoregressive human animation, with lookahead distance controlling the balance between expressivity and consistency [2510.23581].

Within this broader landscape, DiffAIM occupies a distinct position. Its identity guidance is adversarial rather than restorative or preservative; it leverages the generative prior of a latent diffusion model not to keep identity fixed, but to make targeted impersonation visually natural and transferable [2504.21646]. This suggests a dual significance. On one hand, it advances privacy-protection research by exposing vulnerabilities of black-box FR systems, including commercial APIs. On the other hand, it sharpens the technical distinction between human-perceived identity, generative semantics, and machine-recognized identity, a distinction that recurs across recent work on editing, synthesis, personalization, anonymization, and restoration [2504.21646][2602.03826][2510.24213].

Source: https://www.emergentmind.com/topics/identity-guidance