Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Characteristics and prevalence of fake social media profiles with AI-generated faces (2401.02627v2)

Published 5 Jan 2024 in cs.CY, cs.AI, and cs.SI

Abstract: Recent advancements in generative AI have raised concerns about their potential to create convincing fake social media accounts, but empirical evidence is lacking. In this paper, we present a systematic analysis of Twitter (X) accounts using human faces generated by Generative Adversarial Networks (GANs) for their profile pictures. We present a dataset of 1,420 such accounts and show that they are used to spread scams, spam, and amplify coordinated messages, among other inauthentic activities. Leveraging a feature of GAN-generated faces -- consistent eye placement -- and supplementing it with human annotation, we devise an effective method for identifying GAN-generated profiles in the wild. Applying this method to a random sample of active Twitter users, we estimate a lower bound for the prevalence of profiles using GAN-generated faces between 0.021% and 0.044% -- around 10K daily active accounts. These findings underscore the emerging threats posed by multimodal generative AI. We release the source code of our detection method and the data we collect to facilitate further investigation. Additionally, we provide practical heuristics to assist social media users in recognizing such accounts.

PDF HTML Abstract

Analysis of Fake Social Media Profiles with AI-Generated Faces

In the paper "Characteristics and prevalence of fake social media profiles with AI-generated faces," the authors provide a comprehensive examination of the use of AI-generated images in the creation of fake social media profiles. The paper particularly focuses on Twitter accounts utilizing Generative Adversarial Networks (GANs) to fabricate realistic human faces for profile pictures. The paper introduces a dataset of 1,353 Twitter accounts identified as fake, which engage in activities such as spreading scams, spam, and propagating coordinated disinformation campaigns.

The authors emphasize the growing sophistication of generative AI tools, particularly GANs, in producing highly realistic images that are increasingly indistinguishable from genuine human photographs. The paper leverages a unique characteristic of GAN-generated faces — precise and consistent positioning of eyes — to develop a method for detecting such profiles. This technique, coupled with manual human annotation, forms the basis of their detection methodology.

Using this approach, the researchers estimate the prevalence of profiles with GAN-generated faces among active Twitter users to be between 0.021% and 0.044%, translating to approximately 10,000 daily active accounts. This figure is compelling, highlighting a tangible presence of AI-generated profiles despite their ostensibly low proportion. The paper provides a nuanced understanding of the implications of generative AI misuse, suggesting that these artificial profiles are utilized not just for deception but as instruments for coordinated activities that challenge the authenticity of social media interactions.

The paper's findings illuminate the broader implications of AI in the digital information ecosystem. As AI-generated content proliferates, the potential for its misuse in creating sophisticated inauthentic accounts becomes a significant concern for social media platforms and users alike. This underscores the necessity for advanced detection systems to safeguard digital spaces.

The authors explicitly present the challenges in detecting AI-generated images with existing technological tools, advocating for improved algorithms that consider the nuanced aspects of facial landmark placements characteristic of GAN-generated content. Additionally, the paper briefly touches upon the regulatory and educational strategies necessary to mitigate the risks posed by the proliferation of such deceptive profiles. Engaging social media platforms in policy-making and user education appears crucial for developing resilience against AI-enhanced disinformation.

In summary, this paper offers substantial insights into the tactics and prevalence of GAN-generated profiles on social media platforms, with implications for future research and technological development. The authors provide open access to their source code and datasets, inviting further exploration and refinement of detection methodologies across varying contexts and platforms. Looking forward, as generative technologies continue to evolve, the paper accentuates the importance of cross-disciplinary collaboration in addressing the potential threats to digital information integrity.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Trueface: A dataset for the detection of synthetic face images from social networks. In 2022 IEEE International Joint Conference on Biometrics, pages 1–7. IEEE, 2022.
  2. On the opportunities and risks of foundation models. arXiv:2108.07258, 2021.
  3. The value of ai guidance in human examination of synthetically-generated faces. Proceedings of the AAAI Conference on Artificial Intelligence, 37(5):5930–5938, Jun. 2023. doi: 10.1609/aaai.v37i5.25734. URL https://ojs.aaai.org/index.php/AAAI/article/view/25734.
  4. Testing human ability to detect ‘deepfake’ images of human faces. Journal of Cybersecurity, 9(1):tyad011, 2023.
  5. A comprehensive survey of AI-generated content (AIGC): A history of generative AI from GAN to ChatGPT. arXiv:2303.04226, 2023.
  6. The coming age of adversarial social bot detection. First Monday, 2021.
  7. Deep learning based computer generated face identification using convolutional neural network. Applied Sciences, 8(12):2610, 2018.
  8. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  9. The rise of social bots. Communications of the ACM, 59(7):96–104, 2016.
  10. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv:2301.04246, 2023.
  11. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  12. The emerging threat of AI-driven cyber attacks: A review. Applied Artificial Intelligence, 36(1):2037254, 2022.
  13. Eyes tell all: Irregular pupil shapes reveal GAN-generated faces. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2904–2908. IEEE, 2022.
  14. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  15. Exposing GAN-generated faces using inconsistent corneal specular highlights. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2500–2504. IEEE, 2021.
  16. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120, 2023.
  17. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  18. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  19. One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1867–1874, 2014.
  20. More real than real: A study on human visual perception of synthetic faces [applications corner]. IEEE Signal Processing Magazine, 39(1):109–116, 2021.
  21. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.
  22. Detection of GAN-generated fake images over social networks. In 2018 IEEE conference on multimedia information processing and retrieval, pages 384–389. IEEE, 2018.
  23. Addressing the harms of AI-generated inauthentic content. Nature Machine Intelligence, 5:679–680, 2023. doi: 10.1038/s42256-023-00690-w. URL https://rdcu.be/dgGfk.
  24. Exposing GAN-generated profile photos from compact embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 884–892, 2023.
  25. Ganbot: a GAN-based framework for social bot detection. Social Network Analysis and Mining, 12:1–11, 2022.
  26. Detecting GAN generated fake images using co-occurrence matrices. arXiv:1903.06836, 2019.
  27. Ai-synthesized faces are indistinguishable from real faces and more trustworthy. Proceedings of the National Academy of Sciences, 119(8):e2120481119, 2022.
  28. Uncovering coordinated networks on social media: Methods and case studies. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 455–466, 2021.
  29. Just another day on twitter: a complete 24 hours of twitter data. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 1073–1081, 2023.
  30. Hierarchical text-conditional image generation with CLIP latents. arXiv:2204.06125, 1(2):3, 2022.
  31. Towards the detection of diffusion model deepfakes. arXiv:2210.14571, 2022.
  32. Detection of novel social bots by ensembles of specialized classifiers. In Proceedings of 29th ACM International Conference on Information & Knowledge Management (CIKM), pages 2725–2732, 2020. doi: 10.1145/3340531.3412698. URL https://doi.org/10.1145/3340531.3412698.
  33. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  34. GAN-generated faces detection: A survey and new perspectives. arXiv:2202.07145, 2022.
  35. The rise and potential of large language model based agents: A survey. arXiv:2309.07864, 2023.
  36. Weaponized AI for cyber attacks. Journal of Information Security and Applications, 57:102722, 2021.
  37. Exposure to social bots amplifies perceptual biases and regulation propensity. Scientific Reports, 13(1):20707, 2023.
  38. Anatomy of an AI-powered malicious social botnet. arXiv:2307.16336, 2023.
  39. Arming the public with artificial intelligence to counter social bots. Human Behavior and Emerging Technologies, 1(1):48–61, 2019a.
  40. Scalable and Generalizable Social Bot Detection through Data Selection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):1096–1103, 2020. ISSN 2374-3468.
  41. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1–39, 2023.
  42. Exposing GAN-synthesized faces using landmark locations. In Proceedings of the ACM workshop on information hiding and multimedia security, pages 113–118, 2019b.
  43. A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models. arXiv:2303.10420, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Kai-Cheng Yang (29 papers)
  2. Danishjeet Singh (2 papers)
  3. Filippo Menczer (102 papers)
Citations (6)
View on Semantic Scholar