NegFaceDiff: Negative Guidance in Face Synthesis

• NegFaceDiff is a diffusion-based method that injects negative identity conditions during sampling to steer synthesis away from undesired features.
• The technique improves inter-class separability, as evidenced by an increase in the Fisher Discriminant Ratio from 2.427 to 5.687.
• It integrates seamlessly with existing frameworks like IDiff-Face by modifying only the sampling process while preserving intra-class consistency.

NegFaceDiff is a sampling method for identity-conditioned latent diffusion models that introduces explicit negative conditions into the denoising process. Its central mechanism is to steer synthesis toward a desired identity while simultaneously pushing it away from another identity, with the stated objective of improving inter-class separability without sacrificing intra-class consistency. Implemented atop IDiff-Face, it leaves the original DDPM/LDM training regimen unchanged and modifies only sampling; reported experiments show identity separability, measured by the Fisher Discriminant Ratio, increasing from 2.427 to 5.687, with corresponding gains in downstream face recognition systems trained on the generated data (Caldeira et al., 13 Aug 2025).

1. Problem setting and motivation

NegFaceDiff is situated in the use of synthetic face data for face recognition development. The underlying motivation is that synthetic data can reduce privacy, ethical, and practical concerns associated with collecting and using authentic face datasets, while also offering scalability and controllable diversity. Within this setting, diffusion models are presented as the preferred generator over GANs because of superior realism and identity preservation, but identity-conditioned models that rely on positive conditions only can still exhibit identity overlap across classes, especially when generating many classes (Caldeira et al., 13 Aug 2025).

The method is designed to address a specific failure mode of positive-only identity conditioning: unintended feature leakage between identities. According to the reported formulation, positive-only guidance may allow unwanted features to persist in generated samples, which degrades inter-class separability and, consequently, downstream face recognition performance. NegFaceDiff targets this issue by injecting negative conditions into the denoising trajectory so that the model is guided not only toward a positive identity context but also away from an undesired one (Caldeira et al., 13 Aug 2025).

This design makes the method primarily a synthetic data generation technique for face recognition rather than a generic image editing procedure. Its intended effect is geometric in embedding space: shrinking overlapping regions between classes while preserving the identity manifold of the positive class. A plausible implication is that NegFaceDiff treats identity consistency and class separation as distinct optimization pressures at sampling time, instead of assuming that stronger positive conditioning alone is sufficient.

2. Diffusion formulation and negative guidance

NegFaceDiff uses an identity-conditioned latent diffusion model with a U-Net denoiser operating in the latent space of a pre-trained autoencoder. The conditioning signal is an identity context $p^{+}$, typically an embedding produced by a pre-trained face recognition network $f(\cdot)$, such as an ArcFace or ElasticFace backbone. For new synthetic identities, IDiff-Face samples $p^{+}$ from a normal distribution, and this context is fixed per identity while random seeds provide intra-identity variability (Caldeira et al., 13 Aug 2025).

The method retains the standard forward diffusion process:

$$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$

$$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$

with $\alpha_t = 1-\beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^{t}\alpha_i$.

The reverse distribution conditioned on context $c$ is parameterized as

$$p_\theta(x_{t-1} \mid x_t, c) = \mathcal{N}(\mu_\theta(x_t, t, c), \sigma_t^2 I),$$

with mean

$$\mu_\theta(x_t, t, c) = \frac{1}{\sqrt{\alpha_t}} \left[ x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_\theta(x_t, t, c) \right].$$

Training is unchanged relative to the underlying identity-conditioned diffusion model. The loss remains the conditional $f(\cdot)$0-prediction objective with positive context only:

$f(\cdot)$1

with $f(\cdot)$2 (Caldeira et al., 13 Aug 2025).

The distinctive change appears at inference. At each step, the denoiser is evaluated twice, once under the positive context $f(\cdot)$3 and once under the negative context $f(\cdot)$4. The combined guidance is

$f(\cdot)$5

where $f(\cdot)$6 is the negative guidance strength. The reverse update becomes

$f(\cdot)$7

$f(\cdot)$8

with $f(\cdot)$9 (Caldeira et al., 13 Aug 2025).

Under this formulation, the negative condition is not an auxiliary loss or a post hoc filter. It directly modifies the conditional score estimate used by the sampler. This suggests that NegFaceDiff is best understood as a sampling-time guidance rule for identity-conditioned diffusion, analogous in structure to dual-condition guidance, but specialized for inter-class separation in synthetic face generation.

3. Negative context construction, selection strategies, and implementation

Negative conditions are constructed as other identity embeddings $p^{+}$0 selected from the pool of identity contexts. Selection is based on normalized Euclidean distance in embedding space, and the paper studies five strategies: Far-Neg, in which $p^{+}$1 is the farthest identity embedding from $p^{+}$2; Close-Neg, in which it is the nearest impostor; Mid-Neg, in which it is the median impostor by distance; Rand-Neg, in which it is selected randomly from the set of impostors; and Null, in which $p^{+}$3 is the zero vector and serves as a reference negative-prompt condition (Caldeira et al., 13 Aug 2025).

The reported sampling algorithm fixes one negative context per identity for all samples of that identity. For each identity, the sampler initializes $p^{+}$4 and, for $p^{+}$5, computes $p^{+}$6, $p^{+}$7, combines them through the weighted subtraction rule, forms the corresponding reverse mean, and updates the latent with Gaussian noise. After the final step, $p^{+}$8 is decoded to an image through the pre-trained decoder. Multi-negative extensions are described as possible but are not studied (Caldeira et al., 13 Aug 2025).

Several practical settings are specified. The guidance weight $p^{+}$9 is constant across timesteps, and the reported tested values are $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$0, $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$1, and $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$2, with $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$3 giving the best average downstream face recognition accuracy among those values. For ablations, DDIM with 200 steps is used, while final state-of-the-art comparisons use DDPM with 1000 steps for parity with prior work. DDIM sampling is reported to speed evaluation by approximately $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$4 with minimal performance loss (Caldeira et al., 13 Aug 2025).

The implementation is presented as architecture-agnostic in the sense that it requires only the ability to evaluate the denoiser under two different conditions. It therefore applies to conditional latent diffusion models with U-Net backbones whether conditioning enters via cross-attention or feature modulation such as FiLM. The computational overhead is approximately $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$5 denoiser calls per step, while memory overhead is described as minimal. The base model is IDiff-Face, trained in the latent space of a pre-trained autoencoder, with contextual partial dropout at $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$6. Reported diffusion training settings include DDPM with $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$7 steps, a linear variance schedule, and the standard $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$8 $$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, \beta_t I),$$9-prediction loss; NegFaceDiff itself does not alter this training stage. Reported training data for the diffusion model include FFHQ with 70k images and an additional variant trained on CASIA-WebFace with approximately 500k images (Caldeira et al., 13 Aug 2025).

4. Evaluation protocol and empirical results

Evaluation is organized around two categories: identity separability and downstream face recognition performance. Genuine and impostor similarity distributions are computed from embeddings extracted by a ResNet100 with ElasticFace trained on MS1MV2, and separability is summarized by the Fisher Discriminant Ratio

$$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$0

where $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$1 are the mean and standard deviation of genuine scores and $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$2 are those of impostor scores (Caldeira et al., 13 Aug 2025).

For the synthetic-data training study, the reported dataset size is 10k identities with 50 images per identity, totaling 500k images. The face recognition backbone is ResNet50 trained with CosFace using margin $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$3 and scale $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$4, SGD with batch size $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$5, momentum $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$6, and weight decay $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$7, together with RandAugment using 4 operations and magnitude 16 unless otherwise noted. Small-scale evaluation uses LFW, AgeDB-30, CFP-FP, CA-LFW, and CP-LFW, while large-scale evaluation uses IJB-C under 1:1 mixed verification at TAR@FAR $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$8 and $$q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),$$9 (Caldeira et al., 13 Aug 2025).

Measure	Positive-only baseline / reference	NegFaceDiff result
FDR	2.427	5.687 with Far-Neg
EER	0.130	0.039 with Far-Neg
Small-scale average accuracy, DDIM 200, no augmentation	83.95% (IDiff-Face)	85.22% (Far-Neg)
IJB-C TAR@1e−5, FFHQ-trained DM	20.60%	58.09%
IJB-C TAR@1e−4, FFHQ-trained DM	62.60%	73.93%
IJB-C TAR@1e−5, C-WF-trained DM	23.44%	77.38%
IJB-C TAR@1e−4, C-WF-trained DM	69.69%	86.11%

The reported separability gains are accompanied by shifts in score distributions: the genuine mean increases from 0.226 to 0.361, while the impostor mean remains low at approximately 0.019–0.020. The Null strategy achieves an even higher FDR of 6.361, but the paper notes less favorable face recognition training outcomes, indicating that maximal separability and optimal downstream utility do not coincide automatically (Caldeira et al., 13 Aug 2025).

The small-scale benchmark results show a nuanced pattern. Under DDIM with 200 steps and no augmentation, average accuracy improves from 83.95% for IDiff-Face and 84.07% for Null to 85.22% for Far-Neg. Under DDPM with 1000 steps and augmentation, Far-Neg achieves a small-scale average of 88.50%, with per-benchmark results of LFW 97.97, AgeDB 86.87, CA-LFW 90.77, CFP-FP 85.83, and CP-LFW 81.07. For the CASIA-WebFace-trained diffusion model, the small-scale average improves from 91.47% to 92.23%, with gains across AgeDB, CFP-FP, CA-LFW, and CP-LFW (Caldeira et al., 13 Aug 2025).

A fairness assessment on RFW is also reported. With CASIA-WebFace training, NegFaceDiff reaches 85.61% average accuracy, with Indian 86.38, Caucasian 89.40, Asian 84.23, and African 83.20, alongside lower standard deviation at 2.74 and lower SER at 1.58 among diffusion-based approaches. Classical image-level metrics such as FID, KID, and LPIPS are explicitly not the focus and are not reported; identity consistency and diversity are instead characterized through score distributions, genuine standard deviation, and downstream face recognition performance (Caldeira et al., 13 Aug 2025).

5. Position within adjacent research and terminological distinctions

NegFaceDiff belongs to a line of work on identity-conditioned diffusion for synthetic face generation and is positioned against positive-only identity-conditioned diffusion methods such as IDiff-Face and ID$\alpha_t = 1-\beta_t$0, as well as dual condition diffusion, text-to-image negative prompts and guidance techniques, and GAN-based synthetic face recognition systems such as SFace, SynFace, and IDNet. Its distinctive claim is that explicit subtraction of negative identity features at each denoising step improves inter-class separability while preserving intra-class variability, and that this can be achieved as a training-free plug-in at sampling time (Caldeira et al., 13 Aug 2025).

The method should be distinguished from diffusion-based face swapping. DiffFace is a face swapping framework composed of ID conditional DDPM training, sampling with facial guidance, and a target-preserving blending strategy; it uses off-the-shelf facial expert models to transfer source identity while preserving target attributes and background. NegFaceDiff does not address swapping between a source and target image, and it does not rely on facial guidance or target-preserving blending. Its problem formulation is synthetic identity generation for face recognition training, and its core intervention is negative conditioning in the denoising trajectory rather than attribute-preserving source-to-target transfer (Kim et al., 2022).

It should also be distinguished from deepfake detection framed through pairwise facial differencing. DiffFake treats detection as an anomaly detection problem over combined embeddings from pairs of real images of the same person, with a Gaussian Mixture Model trained on real same-identity pairs and anomaly scores defined by negative log-likelihood. NegFaceDiff is not a detector, does not model video-level pairwise anomalies, and does not use combined operators such as $\alpha_t = 1-\beta_t$1 or $\alpha_t = 1-\beta_t$2 for classification. The overlap in terminology is therefore superficial: DiffFake concerns differential anomaly detection, whereas NegFaceDiff concerns negative-context guidance for generation (Stamnas et al., 22 Feb 2025).

A common misconception is that negative guidance necessarily reduces diversity by collapsing classes into narrower modes. The reported evidence is more specific. The Null negative maximizes separability but can reduce intra-class diversity and hurt downstream face recognition generalization, whereas Far-Neg is reported to improve separability while increasing genuine standard deviation. This suggests that the effect of negativity depends materially on how the negative context is selected, not merely on the presence of a subtractive term.

6. Limitations, trade-offs, and ethical considerations

The most immediate trade-off is computational. Because NegFaceDiff evaluates the denoiser twice per step, sampling incurs approximately twice the denoiser calls of the positive-only baseline. This is partly mitigated by DDIM, which reduces sampling time significantly relative to DDPM, but the method remains slower than single-condition sampling (Caldeira et al., 13 Aug 2025).

The method also depends on access to a set of identity contexts for negative selection. Far-Neg is reported to provide the best face recognition results and strong separability, but it presumes that distances to other identity embeddings can be computed. Rand-Neg is described as practical when the full set of contexts is unknown a priori, for example in single-identity generation. This implies a deployment distinction between closed-world synthetic dataset generation, where precomputed context pools are available, and open-ended sampling, where only weaker selection strategies may be feasible (Caldeira et al., 13 Aug 2025).

A second limitation concerns over-pruning. If the guidance strength is too large, negative guidance can suppress not only unwanted overlap but also useful identity variation, reducing diversity or realism. The reported ablation supports this: average small-scale accuracy peaks at $\alpha_t = 1-\beta_t$3 and drops at $\alpha_t = 1-\beta_t$4. Similarly, the Null negative yields the highest separability but less favorable downstream training outcomes. These results indicate that stronger repulsion is not monotonically better (Caldeira et al., 13 Aug 2025).

The paper also notes broader concerns of bias and distribution shift. Although NegFaceDiff reduces measured bias relative to baselines in the reported RFW evaluation, synthetic-to-real domain gaps can persist, particularly under extreme pose, lighting, occlusion, or age differences. Ethical safeguards are therefore emphasized: synthetic identity generation should avoid leakage of real identities, respect data protection laws, and be transparently documented. In that sense, NegFaceDiff addresses one technical bottleneck in synthetic face generation—identity overlap—but does not eliminate the broader representational and governance issues associated with deploying synthetic data in face recognition systems (Caldeira et al., 13 Aug 2025).