Harnessing the Power of Large Vision Language Models for Synthetic Image Detection (2404.02726v1)
Abstract: In recent years, the emergence of models capable of generating images from text has attracted considerable interest, offering the possibility of creating realistic images from text descriptions. Yet these advances have also raised concerns about the potential misuse of these images, including the creation of misleading content such as fake news and propaganda. This study investigates the effectiveness of using advanced vision-LLMs (VLMs) for synthetic image identification. Specifically, the focus is on tuning state-of-the-art image captioning models for synthetic image detection. By harnessing the robust understanding capabilities of large VLMs, the aim is to distinguish authentic images from synthetic images produced by diffusion-based models. This study contributes to the advancement of synthetic image detection by exploiting the capabilities of visual LLMs such as BLIP-2 and ViTGPT2. By tailoring image captioning models, we address the challenges associated with the potential misuse of synthetic images in real-world applications. Results described in this paper highlight the promising role of VLMs in the field of synthetic image detection, outperforming conventional image-based detection techniques. Code and models can be found at https://github.com/Mamadou-Keita/VLM-DETECT.
- “Generative adversarial nets,” Advances in neural information processing systems, vol. 27, 2014.
- “Zero-shot text-to-image generation,” in International Conference on Machine Learning. PMLR, 2021, pp. 8821–8831.
- “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” arXiv preprint arXiv:2112.10741, 2021.
- “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in Neural Information Processing Systems, vol. 35, pp. 36479–36494, 2022.
- “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695.
- “On the detection of synthetic images generated by diffusion models,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
- “Towards the detection of diffusion model deepfakes,” arXiv preprint arXiv:2210.14571, 2022.
- “De-fake: Detection and attribution of fake images generated by text-to-image diffusion models,” arXiv preprint arXiv:2210.06998, 2022.
- “Detecting images generated by diffusers,” arXiv preprint arXiv:2303.05275, 2023.
- “Dire for diffusion-generated image detection,” arXiv preprint arXiv:2303.09295, 2023.
- “Detecting images generated by deep diffusion models using their local intrinsic dimensionality,” arXiv preprint arXiv:2307.02347, 2023.
- “Exposing the fake: Effective diffusion-generated images detection,” arXiv preprint arXiv:2307.06272, 2023.
- “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023.
- “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
- “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365, 2015.
- “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021.
- “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
- “Improved denoising diffusion probabilistic models,” in International Conference on Machine Learning. PMLR, 2021, pp. 8162–8171.
- “Pseudo numerical methods for diffusion models on manifolds,” arXiv preprint arXiv:2202.09778, 2022.
- “Deep residual learning for image recognition,” 2015.
- Francois Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
- “Training data-efficient image transformers & distillation through attention,” 2021.
- “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
- “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
- Mamadou Keita (4 papers)
- Wassim Hamidouche (62 papers)
- Hassen Bougueffa (1 paper)
- Abdenour Hadid (28 papers)
- Abdelmalik Taleb-Ahmed (24 papers)