---
title: 'NegFaceDiff: Negative Guidance in Face Synthesis'
url: https://www.emergentmind.com/topics/negfacediff
type: topic
---

# NegFaceDiff: Negative Guidance in Face Synthesis

NegFaceDiff is a sampling method for identity-conditioned latent diffusion models that introduces explicit negative conditions into the denoising process. Its central mechanism is to steer synthesis toward a desired identity while simultaneously pushing it away from another identity, with the stated objective of improving inter-class separability without sacrificing intra-class consistency. Implemented atop IDiff-Face, it leaves the original DDPM/LDM training regimen unchanged and modifies only sampling; reported experiments show identity separability, measured by the Fisher Discriminant Ratio, increasing from 2.427 to 5.687, with corresponding gains in downstream face recognition systems trained on the generated data [2508.09661].

## 1. Problem setting and motivation

NegFaceDiff is situated in the use of synthetic face data for face recognition development. The underlying motivation is that synthetic data can reduce privacy, ethical, and practical concerns associated with collecting and using authentic face datasets, while also offering scalability and controllable diversity. Within this setting, diffusion models are presented as the preferred generator over GANs because of superior realism and identity preservation, but identity-conditioned models that rely on positive conditions only can still exhibit identity overlap across classes, especially when generating many classes [2508.09661].

The method is designed to address a specific failure mode of positive-only identity conditioning: unintended feature leakage between identities. According to the reported formulation, positive-only guidance may allow unwanted features to persist in generated samples, which degrades inter-class separability and, consequently, downstream face recognition performance. NegFaceDiff targets this issue by injecting negative conditions into the denoising trajectory so that the model is guided not only toward a positive identity context but also away from an undesired one [2508.09661].

This design makes the method primarily a synthetic data generation technique for face recognition rather than a generic image editing procedure. Its intended effect is geometric in embedding space: shrinking overlapping regions between classes while preserving the identity manifold of the positive class. A plausible implication is that NegFaceDiff treats identity consistency and class separation as distinct optimization pressures at sampling time, instead of assuming that stronger positive conditioning alone is sufficient.

## 2. Diffusion formulation and negative guidance

NegFaceDiff uses an identity-conditioned latent diffusion model with a U-Net denoiser operating in the latent space of a pre-trained autoencoder. The conditioning signal is an identity context $p^{+}$, typically an embedding produced by a pre-trained face recognition network $f(\cdot)$, such as an ArcFace or ElasticFace backbone. For new synthetic identities, IDiff-Face samples $p^{+}$ from a normal distribution, and this context is fixed per identity while random seeds provide intra-identity variability [2508.09661].

The method retains the standard forward diffusion process:
$$
q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),
$$
$$
q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),
$$
with $\alpha_t = 1-\beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^{t}\alpha_i$.

The reverse distribution conditioned on context $c$ is parameterized as
$$
p_\theta(x_{t-1} \mid x_t, c) = \mathcal{N}(\mu_\theta(x_t, t, c), \sigma_t^2 I),
$$
with mean
$$
\mu_\theta(x_t, t, c) =
\frac{1}{\sqrt{\alpha_t}}
\left[
x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t, c)
\right].
$$

Training is unchanged relative to the underlying identity-conditioned diffusion model. The loss remains the conditional $\epsilon$-prediction objective with positive context only:
$$
L(\theta) =
\mathbb{E}_{t,x_0,\epsilon}
\left[
\left\|
\epsilon -
\epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, t, p^{+})
\right\|_2^2
\right],
$$
with $\epsilon \sim \mathcal{N}(0, I)$ [2508.09661].

The distinctive change appears at inference. At each step, the denoiser is evaluated twice, once under the positive context $p^{+}$ and once under the negative context $p^{-}$. The combined guidance is
$$
\hat{\epsilon}_\theta(x_t, t, p^{+}, p^{-}) =
(1+w)\epsilon_\theta(x_t, t, p^{+}) - w \epsilon_\theta(x_t, t, p^{-}),
$$
where $w \ge 0$ is the negative guidance strength. The reverse update becomes
$$
x_{t-1} = \mu_\theta(x_t, t, p^{+}, p^{-}) + \sigma_t \zeta_t,
$$
$$
\mu_\theta(x_t, t, p^{+}, p^{-}) =
\frac{1}{\sqrt{\alpha_t}}
\left[
x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\hat{\epsilon}_\theta(x_t, t, p^{+}, p^{-})
\right],
$$
with $\zeta_t \sim \mathcal{N}(0, I)$ [2508.09661].

Under this formulation, the negative condition is not an auxiliary loss or a post hoc filter. It directly modifies the conditional score estimate used by the sampler. This suggests that NegFaceDiff is best understood as a sampling-time guidance rule for identity-conditioned diffusion, analogous in structure to dual-condition guidance, but specialized for inter-class separation in synthetic face generation.

## 3. Negative context construction, selection strategies, and implementation

Negative conditions are constructed as other identity embeddings $p^{-}$ selected from the pool of identity contexts. Selection is based on normalized Euclidean distance in embedding space, and the paper studies five strategies: Far-Neg, in which $p^{-}$ is the farthest identity embedding from $p^{+}$; Close-Neg, in which it is the nearest impostor; Mid-Neg, in which it is the median impostor by distance; Rand-Neg, in which it is selected randomly from the set of impostors; and Null, in which $p^{-}$ is the zero vector and serves as a reference negative-prompt condition [2508.09661].

The reported sampling algorithm fixes one negative context per identity for all samples of that identity. For each identity, the sampler initializes $x_T \sim \mathcal{N}(0,I)$ and, for $t=T,\ldots,1$, computes $\epsilon_{\text{pos}}$, $\epsilon_{\text{neg}}$, combines them through the weighted subtraction rule, forms the corresponding reverse mean, and updates the latent with Gaussian noise. After the final step, $x_0$ is decoded to an image through the pre-trained decoder. Multi-negative extensions are described as possible but are not studied [2508.09661].

Several practical settings are specified. The guidance weight $w$ is constant across timesteps, and the reported tested values are $0.25$, $0.5$, and $1.0$, with $w=0.5$ giving the best average downstream face recognition accuracy among those values. For ablations, DDIM with 200 steps is used, while final state-of-the-art comparisons use DDPM with 1000 steps for parity with prior work. DDIM sampling is reported to speed evaluation by approximately $5\times$ with minimal performance loss [2508.09661].

The implementation is presented as architecture-agnostic in the sense that it requires only the ability to evaluate the denoiser under two different conditions. It therefore applies to conditional latent diffusion models with U-Net backbones whether conditioning enters via cross-attention or feature modulation such as FiLM. The computational overhead is approximately $2\times$ denoiser calls per step, while memory overhead is described as minimal. The base model is IDiff-Face, trained in the latent space of a pre-trained autoencoder, with contextual partial dropout at $25\%$. Reported diffusion training settings include DDPM with $T=1000$ steps, a linear variance schedule, and the standard $L_2$ $\epsilon$-prediction loss; NegFaceDiff itself does not alter this training stage. Reported training data for the diffusion model include FFHQ with 70k images and an additional variant trained on CASIA-WebFace with approximately 500k images [2508.09661].

## 4. Evaluation protocol and empirical results

Evaluation is organized around two categories: identity separability and downstream face recognition performance. Genuine and impostor similarity distributions are computed from embeddings extracted by a ResNet100 with ElasticFace trained on MS1MV2, and separability is summarized by the Fisher Discriminant Ratio
$$
\mathrm{FDR} = \frac{(\mu_g - \mu_i)^2}{\sigma_g^2 + \sigma_i^2},
$$
where $\mu_g, \sigma_g$ are the mean and standard deviation of genuine scores and $\mu_i, \sigma_i$ are those of impostor scores [2508.09661].

For the synthetic-data training study, the reported dataset size is 10k identities with 50 images per identity, totaling 500k images. The face recognition backbone is ResNet50 trained with CosFace using margin $m=0.35$ and scale $s=64$, SGD with batch size $512$, momentum $0.9$, and weight decay $5\times 10^{-4}$, together with RandAugment using 4 operations and magnitude 16 unless otherwise noted. Small-scale evaluation uses LFW, AgeDB-30, CFP-FP, CA-LFW, and CP-LFW, while large-scale evaluation uses IJB-C under 1:1 mixed verification at TAR@FAR $10^{-5}$ and $10^{-4}$ [2508.09661].

| Measure | Positive-only baseline / reference | NegFaceDiff result |
|---|---:|---:|
| FDR | 2.427 | 5.687 with Far-Neg |
| EER | 0.130 | 0.039 with Far-Neg |
| Small-scale average accuracy, DDIM 200, no augmentation | 83.95% (IDiff-Face) | 85.22% (Far-Neg) |
| IJB-C TAR@1e−5, FFHQ-trained DM | 20.60% | 58.09% |
| IJB-C TAR@1e−4, FFHQ-trained DM | 62.60% | 73.93% |
| IJB-C TAR@1e−5, C-WF-trained DM | 23.44% | 77.38% |
| IJB-C TAR@1e−4, C-WF-trained DM | 69.69% | 86.11% |

The reported separability gains are accompanied by shifts in score distributions: the genuine mean increases from 0.226 to 0.361, while the impostor mean remains low at approximately 0.019–0.020. The Null strategy achieves an even higher FDR of 6.361, but the paper notes less favorable face recognition training outcomes, indicating that maximal separability and optimal downstream utility do not coincide automatically [2508.09661].

The small-scale benchmark results show a nuanced pattern. Under DDIM with 200 steps and no augmentation, average accuracy improves from 83.95% for IDiff-Face and 84.07% for Null to 85.22% for Far-Neg. Under DDPM with 1000 steps and augmentation, Far-Neg achieves a small-scale average of 88.50%, with per-benchmark results of LFW 97.97, AgeDB 86.87, CA-LFW 90.77, CFP-FP 85.83, and CP-LFW 81.07. For the CASIA-WebFace-trained diffusion model, the small-scale average improves from 91.47% to 92.23%, with gains across AgeDB, CFP-FP, CA-LFW, and CP-LFW [2508.09661].

A fairness assessment on RFW is also reported. With CASIA-WebFace training, NegFaceDiff reaches 85.61% average accuracy, with Indian 86.38, Caucasian 89.40, Asian 84.23, and African 83.20, alongside lower standard deviation at 2.74 and lower SER at 1.58 among diffusion-based approaches. Classical image-level metrics such as FID, KID, and LPIPS are explicitly not the focus and are not reported; identity consistency and diversity are instead characterized through score distributions, genuine standard deviation, and downstream face recognition performance [2508.09661].

## 5. Position within adjacent research and terminological distinctions

NegFaceDiff belongs to a line of work on identity-conditioned diffusion for synthetic face generation and is positioned against positive-only identity-conditioned diffusion methods such as IDiff-Face and ID$^3$, as well as dual condition diffusion, text-to-image negative prompts and guidance techniques, and GAN-based synthetic face recognition systems such as SFace, SynFace, and IDNet. Its distinctive claim is that explicit subtraction of negative identity features at each denoising step improves inter-class separability while preserving intra-class variability, and that this can be achieved as a training-free plug-in at sampling time [2508.09661].

The method should be distinguished from diffusion-based face swapping. DiffFace is a face swapping framework composed of ID conditional DDPM training, sampling with facial guidance, and a target-preserving blending strategy; it uses off-the-shelf facial expert models to transfer source identity while preserving target attributes and background. NegFaceDiff does not address swapping between a source and target image, and it does not rely on facial guidance or target-preserving blending. Its problem formulation is synthetic identity generation for face recognition training, and its core intervention is negative conditioning in the denoising trajectory rather than attribute-preserving source-to-target transfer [2212.13344].

It should also be distinguished from deepfake detection framed through pairwise facial differencing. DiffFake treats detection as an anomaly detection problem over combined embeddings from pairs of real images of the same person, with a Gaussian Mixture Model trained on real same-identity pairs and anomaly scores defined by negative log-likelihood. NegFaceDiff is not a detector, does not model video-level pairwise anomalies, and does not use combined operators such as $|\mathbf{z}_1-\mathbf{z}_2|$ or $(\mathbf{z}_1-\mathbf{z}_2)^2$ for classification. The overlap in terminology is therefore superficial: DiffFake concerns differential anomaly detection, whereas NegFaceDiff concerns negative-context guidance for generation [2502.16247].

A common misconception is that negative guidance necessarily reduces diversity by collapsing classes into narrower modes. The reported evidence is more specific. The Null negative maximizes separability but can reduce intra-class diversity and hurt downstream face recognition generalization, whereas Far-Neg is reported to improve separability while increasing genuine standard deviation. This suggests that the effect of negativity depends materially on how the negative context is selected, not merely on the presence of a subtractive term.

## 6. Limitations, trade-offs, and ethical considerations

The most immediate trade-off is computational. Because NegFaceDiff evaluates the denoiser twice per step, sampling incurs approximately twice the denoiser calls of the positive-only baseline. This is partly mitigated by DDIM, which reduces sampling time significantly relative to DDPM, but the method remains slower than single-condition sampling [2508.09661].

The method also depends on access to a set of identity contexts for negative selection. Far-Neg is reported to provide the best face recognition results and strong separability, but it presumes that distances to other identity embeddings can be computed. Rand-Neg is described as practical when the full set of contexts is unknown a priori, for example in single-identity generation. This implies a deployment distinction between closed-world synthetic dataset generation, where precomputed context pools are available, and open-ended sampling, where only weaker selection strategies may be feasible [2508.09661].

A second limitation concerns over-pruning. If the guidance strength is too large, negative guidance can suppress not only unwanted overlap but also useful identity variation, reducing diversity or realism. The reported ablation supports this: average small-scale accuracy peaks at $w=0.5$ and drops at $w=1.0$. Similarly, the Null negative yields the highest separability but less favorable downstream training outcomes. These results indicate that stronger repulsion is not monotonically better [2508.09661].

The paper also notes broader concerns of bias and distribution shift. Although NegFaceDiff reduces measured bias relative to baselines in the reported RFW evaluation, synthetic-to-real domain gaps can persist, particularly under extreme pose, lighting, occlusion, or age differences. Ethical safeguards are therefore emphasized: synthetic identity generation should avoid leakage of real identities, respect data protection laws, and be transparently documented. In that sense, NegFaceDiff addresses one technical bottleneck in synthetic face generation—identity overlap—but does not eliminate the broader representational and governance issues associated with deploying synthetic data in face recognition systems [2508.09661].

Source: https://www.emergentmind.com/topics/negfacediff