DiffusionFF: Diffusion-Based Face Forgery Detection

• DiffusionFF is a face forgery detection framework that employs a conditional denoising diffusion model to generate fine-grained DSSIM maps for precise artifact localization.
• It fuses high-level semantic features from a pretrained detector with diffusion-generated artifact maps to improve both classification accuracy and explainability.
• Empirical evaluations demonstrate that DiffusionFF achieves high detection AUC and superior localization metrics across datasets through detailed ablations and robust training protocols.

DiffusionFF is a face forgery detection framework that couples binary deepfake classification with diffusion-based artifact localization. Its central mechanism is a conditional denoising diffusion probabilistic model that generates fine-grained Structural Dissimilarity (DSSIM) maps, which are then fused with high-level semantic features from a pretrained forgery detector to improve both detection accuracy and localization fidelity. In the formulation reported by the original work, the generated DSSIM map is not an auxiliary visualization detached from the decision process; it is a learned artifact representation that materially contributes to the final real-versus-fake prediction (Peng et al., 3 Aug 2025).

1. Concept and problem formulation

DiffusionFF is situated in face forgery detection, where the conventional task is binary classification of a face image or video frame as real or fake. The method is motivated by the claim that modern deepfake detection should not only predict whether an image has been manipulated, but also localize where the evidence of manipulation lies. The stated reasons are explainability and trustworthiness: localization provides visual evidence of the detector’s decision and can make a prediction less opaque (Peng et al., 3 Aug 2025).

A central distinction in the framework is between mask-based localization and DSSIM-based localization. Mask-based methods predict binary manipulated regions and are described as often coarse. DiffusionFF instead uses DSSIM maps, which are continuous, fine-grained maps intended to capture subtle local inconsistencies between original and manipulated appearances at the pixel level. The framework argues that these maps can expose blending artifacts, texture inconsistencies, boundary anomalies, and structural mismatches that are weak or blurred in RGB space alone.

The method’s main hypothesis is that better localization improves detection. Existing DSSIM-based approaches are characterized as relying on direct regression and therefore tending to produce blurry maps. DiffusionFF replaces direct regression with iterative denoising, using a diffusion model to generate higher-quality DSSIM maps and then feeding those maps back into the classifier through an artifact-aware fusion pathway.

2. DSSIM supervision and artifact representation

The supervision signal for the localization module is a ground-truth DSSIM map computed from aligned real and manipulated face pairs. After aligning the face crops from original and manipulated videos, the framework computes local structural dissimilarity with a sliding window. The paper gives

$$\begin{aligned} \text{DSSIM}(i, j) &= 1 - \text{SSIM}(x, y), \ \text{SSIM}(x, y) &= \frac{(2\mu_x\mu_y+C_1)(2\sigma_{xy}+C_2)}{(\mu_x^2+\mu_y^2+C_1)(\sigma_x^2+\sigma_y^2+C_2)}. \end{aligned}$$

Here $(i,j)$ denotes pixel location; $x$ and $y$ are local square windows centered at $(i,j)$ in the original and manipulated images; $\mu_x,\mu_y$ are local means; $\sigma_x,\sigma_y$ are local variances; $\sigma_{xy}$ is the covariance; and $C_1,C_2$ are numerical-stability constants (Peng et al., 3 Aug 2025).

Ground-truth DSSIM maps are generated with a $7 \times 7$ local window. They are computed only for manipulated images. For real images, the target DSSIM map is a pure black map, representing zero dissimilarity. This design makes the localization target structurally asymmetric: forged inputs are supervised with fine-grained spatial dissimilarity, whereas authentic inputs are supervised toward an all-zero artifact field.

This representation differs from binary segmentation-style supervision in an important way. A DSSIM map is continuous rather than categorical. That suggests a localization objective centered on structural inconsistency rather than region occupancy. The framework interprets this as especially useful for subtle manipulations, where the forged evidence may be distributed across boundaries, textures, or low-amplitude distortions rather than confined to a sharply delineated region.

3. Architecture and diffusion-based localization pipeline

DiffusionFF begins with a pretrained forgery detector. The primary backbone is ConvNeXt-B pretrained on FF++, and the detector is frozen during both training stages. It has four hierarchical stages, and the framework extracts multi-resolution features from those stages. These features serve two functions: they provide high-level semantic cues about manipulations and they condition the diffusion model that generates the DSSIM map (Peng et al., 3 Aug 2025).

The localization module is a conditional DDPM with a standard U-Net backbone and timestep encoding. In the supplement, the U-Net is specified as seven stages with channel dimensions

$(i,j)$0

Because detector features and U-Net stages differ in shape and channel count, DiffusionFF uses conditioning projectors to transform detector features before injecting them into the corresponding U-Net encoder stages. The paper states that jointly training detector and diffusion from scratch leads to instability or collapse; the frozen detector is therefore treated as a stable provider of forgery priors.

The forward diffusion process uses the standard DDPM closed-form noising equation, with the clean signal $(i,j)$1 interpreted here as the ground-truth DSSIM map: $(i,j)$2 where $(i,j)$3. The denoising objective is

$(i,j)$4

The reverse update is printed in the paper as

$(i,j)$5

As noted in the source, this expression appears typographically incomplete relative to standard DDPM notation, but its stated role is the iterative denoising step (Peng et al., 3 Aug 2025).

At inference time, the pipeline proceeds from a face image through the frozen detector, into the conditioned diffusion U-Net, which starts from pure noise $(i,j)$6 and generates a single-channel DSSIM map through iterative denoising. That map is then passed through an artifact feature extractor. The resulting artifact-aware features are aligned with the detector’s final-stage semantic features and fused by a gating mechanism. A linear projection layer finally produces the real/fake classification score.

4. Data processing, training protocol, and implementation

The preprocessing and testing protocol follows SBI. RetinaFace is used for face detection and bounding box generation. During training, each detected face is cropped with a random margin from 4% to 20%; during inference, a fixed 12.5% margin is used. Dlib 81-point landmarks are used only during training. During training, 32 frames are extracted from each real video, 8 frames from each fake video, and 8 frames from each real video are used to synthesize pseudo-fakes using SBI. During testing, 32 frames per video are sampled uniformly. If multiple faces appear in one frame, the classifier is run on each face and the highest forgery score is taken as the frame-level confidence (Peng et al., 3 Aug 2025).

The training set is enriched with SBI-generated pseudo-fake images. Because these are aligned with the corresponding real images, they support accurate ground-truth DSSIM generation. Augmentations include RGBShift, HueSaturationValue, RandomBrightnessContrast, Downscale, Sharpen, and ImageCompression.

Training is explicitly two-stage. In stage 1, the pretrained forgery detector is frozen, while the conditioning projectors and diffusion model are trained with the DDPM MSE denoising loss. In stage 2, the detector, conditioning projectors, and diffusion model are all frozen, and only the artifact feature extractor and classifier are trained, using standard Cross-Entropy loss. The framework states that generation and classification are significantly different tasks and that joint optimization is suboptimal or unstable.

The implementation details reported are specific. Stage 1 uses 100 epochs, AdamW, batch size 96, initial learning rate $(i,j)$7, cosine decay, and total denoising steps $(i,j)$8. Stage 2 uses 5 epochs, AdamW, batch size 128, and a fixed learning rate $(i,j)$9. The system is implemented in PyTorch and trained on 8 NVIDIA RTX 3090 GPUs. The noise schedule is linear, with beta increasing from 0.02 to 0.4. The paper studies $x$0, selects $x$1 as the best quality-efficiency tradeoff, and reports that training fails to converge when $x$2 (Peng et al., 3 Aug 2025).

5. Empirical performance

The reported evaluation spans intra-dataset detection on FF++, cross-dataset detection on CDF2, DFDC, DFDCP, and FFIW, and localization-quality assessment through PSNR, SSIM, LPIPS, and FID of generated DSSIM maps. Detection is measured by video-level AUC, where video score is the average of frame-level predictions (Peng et al., 3 Aug 2025).

Benchmark	DiffusionFF result	Setting
FF++ overall	99.94	Intra-dataset AUC
CDF2	97.24	Cross-dataset AUC
DFDC	85.05	Cross-dataset AUC
DFDCP	92.56	Cross-dataset AUC
FFIW	88.56	Cross-dataset AUC

On FF++, DiffusionFF reports AUC values of 100 on DF, 100 on F2F, 99.90 on FS, and 99.86 on NT, with an overall FF++ AUC of 99.94. On cross-dataset evaluation, it achieves the best AUC among the compared methods on all four reported benchmarks: 97.24 on CDF2, 85.05 on DFDC, 92.56 on DFDCP, and 88.56 on FFIW.

Localization results show equally large gains relative to prior DSSIM-based approaches.

Dataset	PSNR / SSIM	LPIPS / FID
FF++	26.376 / 0.718	0.198 / 43.093
CDF2	30.697 / 0.546	0.376 / 98.982
FFIW	32.828 / 0.529	0.394 / 191.798

On FF++, DiffusionFF improves localization to PSNR 26.376, SSIM 0.718, LPIPS 0.198, and FID 43.093. The source highlights the magnitude of the FID reduction relative to prior regression-based outputs. Cross-dataset localization remains substantially stronger than the compared methods, with CDF2 at 30.697 PSNR, 0.546 SSIM, 0.376 LPIPS, and 98.982 FID, and FFIW at 32.828 PSNR, 0.529 SSIM, 0.394 LPIPS, and 191.798 FID.

The qualitative analysis reported in the paper is consistent with the quantitative findings. DiffusionFF is described as generating more precise, more fine-grained, and less blurry DSSIM maps than LRL and LiSiam, highlighting subtle manipulated regions across all four FF++ forgery types. The paper’s interpretation is that the gain in localization quality is not merely explanatory; it also improves downstream detection.

6. Ablations, limitations, and broader context

The ablation studies isolate the contribution of diffusion-based localization, conditioning design, fusion strategy, detector choice, denoising horizon, and training schedule. In the diffusion-design ablation, direct regression yields PSNR 24.341, SSIM 0.584, LPIPS 0.252, and FID 172.213; latent-space diffusion improves FID to 56.384 but is still below the full design; final-stage conditioning reaches FID 43.812; decoder conditioning gives FID 46.329; and the proposed configuration attains 26.376 PSNR, 0.718 SSIM, 0.198 LPIPS, and 43.093 FID. The paper interprets this as evidence that diffusion outperforms direct regression, pixel-space diffusion outperforms latent-space diffusion for this task, and multi-stage conditioning is superior to conditioning only the final stage (Peng et al., 3 Aug 2025).

The fusion ablation compares addition, Hadamard product, concatenation, cross-attention, and a gating mechanism. The gating mechanism gives the best average cross-dataset AUC, 90.85, narrowly ahead of cross-attention at 90.71. Backbone ablations show that DiffusionFF consistently improves EfficientNet-B4, Swin-B, and ConvNeXt-B, with ConvNeXt-B + DiffusionFF performing best: CDF2 97.24, DFDC 85.05, DFDCP 92.56, and FFIW 88.56.

The denoising-step ablation is similarly specific. $x$3 is chosen as the best quality-efficiency tradeoff. $x$4 slightly improves LPIPS and FID, to 0.193 and 40.363 respectively, but hurts PSNR and SSIM and nearly doubles computation. $x$5 fails to converge. The training-strategy ablation further shows that two-stage training dominates single-stage training: CDF2 97.24 versus 95.47, DFDC 85.05 versus 83.54, DFDCP 92.56 versus 90.27, and FFIW 88.56 versus 87.69. A further result of note is that replacing generated DSSIM maps with ground-truth DSSIM maps in stage 2 reduces CDF2 AUC from 97.24% to 96.72%, which the authors attribute to distribution mismatch between ground-truth maps and diffusion-generated maps at inference.

Several limitations are explicit or strongly implied. Ground-truth DSSIM supervision depends on aligned real/fake pairs and is therefore available for FF++, CDF2, and FFIW, but not for DFDC or DFDCP. The diffusion module introduces additional cost through iterative denoising and increased parameter count; for example, ConvNeXt-B grows from 89M parameters to 102M with DiffusionFF. The method is architecturally tied to hierarchical feature extractors, which is why plain ViT is excluded. Joint training of detector and diffusion from scratch can collapse. The paper also reports that a ControlNet/Stable Diffusion approach collapsed, attributing this to mismatch between natural-image RGB priors and the target single-channel DSSIM-map domain.

Within the broader diffusion literature, DiffusionFF occupies a distinct position. Unlike federated diffusion frameworks such as FedDiffuse and FedDDPM, which adapt DDPM training to distributed, non-IID data and emphasize parameter aggregation or server-side synthetic correction (Goede et al., 2024); (Peng et al., 9 Mar 2025), DiffusionFF uses diffusion as a conditional artifact-localization module inside a supervised deepfake detector. Its diffusion model is therefore not a general-purpose image generator, but a structured mechanism for generating evidential maps that sharpen both localization and classification.