MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization

Abstract: The swift advancement in photo-realistic face generation technology has sparked considerable concerns across society and academia, emphasizing the requirement of generalizable face forgery detection and localization methods. Prior works tend to capture face forgery patterns across multiple domains using image modality, other modalities like fine-grained texts are not comprehensively investigated, which restricts the generalization capability of models. Besides, they usually analyze facial images created by GAN, but struggle to identify and localize those synthesized by diffusion. To solve the problems, in this paper, we devise a novel multi-domain fine-grained vision-language reconstruction (MFVLR) model, which explores comprehensive and diverse visual forgery traces via language-guided face forgery representation learning, to achieve generalizable diffusion-synthesized face forgery detection and localization (DFFDL). Specifically, we devise a fine-grained language transformer that studies general fine-grained language embeddings using language reconstruction. We propose a multi-domain vision encoder to capture general and complementary visual forgery patterns across the image and residual domains. A vision decoder is designed to reconstruct image appearance and achieve forgery localization. Besides, we propose an innovative plug-and-play vision injection module to enhance the interaction between the vision and language embeddings. Extensive experiments and visualizations demonstrate that our network outperforms the state of the art on different settings like cross-generator, cross-forgery, and cross-dataset evaluations.

• The paper introduces a novel MFVLR framework that fuses multi-domain visual residuals with fine-grained language embeddings to enhance face forgery detection.
• It leverages residual analysis and a vision injection module to precisely localize forged regions, especially in diffusion-based synthetic faces.
• Experimental results demonstrate significant improvements in AUC and mIoU, underscoring the model's robustness against image perturbations and cross-domain variations.

Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization

Introduction

The increasing fidelity of synthetic face images generated by diffusion models and other generative methods necessitates robust, generalizable detection approaches. Existing face forgery detection and localization (FFDL) methods commonly focus on spatial or frequency domains, often disregarding cross-modal signals such as language. Furthermore, their efficacy on diffusion-based generative models remains limited. "MFVLR: Multi-domain Fine-grained Vision-Language Reconstruction for Generalizable Diffusion Face Forgery Detection and Localization" (2605.10071) addresses these deficits via a multi-domain, fine-grained, vision-language framework designed for robust synthetic face detection and localization, especially for content synthesized by advanced diffusion models.

Figure 1: Architecture overview of MFVLR, contrasting traditional space/frequency visual forgery detectors and the proposed multi-domain, vision-language reconstruction approach.

Model Architecture and Methodology

MFVLR integrates multi-domain image-residual visual features with fine-grained, class-aware language embeddings through end-to-end reconstruction and contrastive learning:

• Multi-domain Vision Encoder (MVE): Encodes both global image and residual features via shared transformer blocks. Residuals, computed as differences between reconstructed and original images, provide highly discriminative cues distinguishing real from synthetic content, particularly for diffusion-generated faces.
• Vision Decoder (VD): Performs dual decoding for both image appearance (to aid reconstruction and supervision) and pixel-level manipulation localization.
• Fine-grained Language Transformer (FLT): Introduces a language encoder/decoder stack with a plug-and-play Vision Injection Module (VIM). VIM enables word-level cross-attention, letting global visual forensics representations directly inform linguistic reconstruction and vice versa.

Figure 3: Complete workflow highlighting image-residual feature extraction, image reconstruction, residual computation, fine-grained text encoding, VIM interaction, and multi-objective learning.

• Vision Injection Module (VIM): Facilitates efficient and precise integration between the class token of vision features and word-level language tokens. This module can be attached to arbitrary Vision-LLMs (VLMs) such as CLIP, enhancing generalization with negligible parameter overhead.

Figure 5: VIM interaction pipeline, where vision features are projected as keys/values and queried by language tokens at the word level.

Residual Analysis

MFVLR demonstrates that residual maps derived from diffusion-synthesized faces exhibit notably higher discriminative power (as assessed by FID) than those from GAN-based forgeries or from frequency/edge-based priors.

Figure 2: Cross-generator visualization and FID score analysis of authentic and forged residuals, edges, high-frequency priors—diffusion residuals yield maximal domain separability.

This motivates explicit residual encoding in the detection pipeline, resulting in improved generalization to unseen generators.

Language-guided Feature Alignment and Reconstruction

A crucial insight revealed via t-SNE is that language guidance mitigates the modality gap among diffusion generators, enhancing clustering between real and fake distributions.

Figure 4: t-SNE of visual features with and without language prompts, showing reduced domain shift and tighter cluster alignment under language-guided embeddings.

FLT reconstructs input prompts, enforcing semantic alignment and supplying complementary supervision. Comprehensive ablation supports that both encoder and decoder contributions substantively impact detection and localization accuracy.

Experimental Results

Generalization to Unseen Manipulations and Generators

MFVLR establishes state-of-the-art results across multiple challenging protocols: cross-forgery, cross-generator, and cross-dataset. Notably, under cross-diffusion evaluation on GenFace, MFVLR consistently surpasses both traditional and vision-language competitors, delivering:

• Higher AUC and mIoU across unseen diffusions (DDPM, LatDiff, CollDiff, DiffFace, Diffae)
• Significant improvements over strong baselines (e.g., up to +10% AUC vs. M2TR on Diffae, +5% mIoU over MSCCNet on DiffFace)

Robustness to Image Perturbations

The model substantially outperforms existing detectors under a suite of image corruptions (noise, blur, pixelation, compression, etc.), as shown by maintained AUC scores across distortion intensities.

Figure 7: Comparative AUC of MFVLR and existing models under diverse image distortions—MFVLR exhibits greater robustness and stability.

Localization Performance

MFVLR achieves precise localization, as reflected in improved mIoU and qualitative mask visualizations. It outperforms HiFi-Net and MSCCNet in capturing manipulated regions, even on challenging diffusion-based fakes.

Figure 6: Module heatmaps visualizing progressive improvement in forged region localization across MFVLR architectural variants.

Figure 10: Mask visualizations for various detectors, with MFVLR highlighting forgery regions more comprehensively and accurately.

Ablations and Analysis

Key findings include:

• Residual encoding and VD/MD modules: Substantial gains in AUC/mIoU for both detection and localization.
• Language Decoder and hierarchical prompts: Fine-grained textual supervision enhances both vision-language alignment and cross-domain robustness.
• VIM integration: Plug-and-play VIM in both language encoder and decoder maximizes benefit, especially with cross-attention fusion at the word level.
• Thresholding and block depth: Mask threshold and number of transformer blocks are sensitive hyperparameters for localization; optimal settings maximize generalizability.
• Efficiency: Using class token as key-value in VIM achieves linear attention computation, providing strong trade-off between performance and complexity.

Practical and Theoretical Implications

MFVLR’s results underscore the benefit of multi-modal, residual-guided, and language-augmented learning for robust face forgery detection across revolutionizing generative paradigms. Practically, the architecture is directly extensible to other forms of AIGC (beyond diffusion/GAN faces), and can be seamlessly fused with existing VLM-based detection and reasoning systems. Theoretically, the model demonstrates that vision-language fine-grained reconstruction is an effective mechanism for bridging synthetic-real domain shifts and exposes new avenues for semantic/physical forgery source disentanglement.

Future developments may include scaling toward large multi-modal foundation models—potentially enabling explainable, interactive forensic analysis and visual question answering on synthetic media. Direct application toward broader synthetic media detection (e.g., video, audio, cross-modal manipulations) is anticipated.

Conclusion

MFVLR introduces a principled, modular architecture for generalizable diffusion face forgery detection and localization, demonstrating robust cross-domain generalization, resilience to perturbation, and precise localization by integrating residual and visual features with fine-grained language modeling. The plug-and-play VIM module broadens applicability across vision-language systems. This work advances both the practice and understanding of multi-modal, semantically informed deepfake detection, and provides a strong baseline for future research in robust AIGC forensics.

Figure 11: Example residual visualization, highlighting strong separability between real and diffusion-generated fakes—the principal prior exploited by MFVLR.