SocialDF: Social Media Deepfake Benchmark
- SocialDF is a benchmark dataset and detection framework addressing social media deepfakes by verifying identity, speech, and contextual cues.
- It comprises 1,071 real and 1,055 manipulated short videos, annotated via manual search, automated scraping, and consensus-based verification including sentiment analysis.
- The detection model employs a two-stage multimodal pipeline using face detection, audio transcription, and multi-agent LLM reasoning to overcome limitations of traditional lip-sync detectors.
Searching arXiv for "SocialDF" and closely related references to ground the article in current papers. SocialDF most specifically denotes the benchmark dataset and detection framework introduced in “SocialDF: Benchmark Dataset and Detection Model for Mitigating Harmful Deepfake Content on Social Media Platforms” (Batra et al., 5 Jun 2025). In that usage, it addresses harmful deepfake content as it appears on social-media platforms rather than in clean laboratory-style settings: short, fast-consumed, highly contextual clips that may combine speech, faces, overlays, scene cuts, multiple speakers, background noise, and social-media-style editing. Its central premise is that realistic moderation requires more than lip-sync inspection or low-level visual artifact detection; instead, the system treats a video as a multimedia claim about who appears, what was said, and whether the surrounding context supports authenticity (Batra et al., 5 Jun 2025).
1. Scope and nomenclature
Within the current literature, SocialDF is most clearly the proper name of the 2025 deepfake benchmark and detection model for social media (Batra et al., 5 Jun 2025). In surrounding research discussions, however, the same label has also been used as a convenience reference for distinct social-data problems, including denoised social recommendation via the Denoised Self-Augmented Learning paradigm (Wang et al., 2023), decentralized social data management via Prometheus (Kourtellis et al., 2012), and socially-aware DHT placement for decentralized online social networks (Nasir et al., 2015). This suggests that the term is not self-disambiguating and should be interpreted from the paper title and problem setting.
The formal SocialDF benchmark is situated in the deepfake-detection literature but is motivated by a mismatch between conventional datasets and real social-media content. Earlier benchmarks such as FakeAVCeleb, LAV-DF, AV-Deepfake1M, Celeb-DF, DFDC, and DeepSpeak are treated as useful reference points, yet they are described as generally easier than real social-media content because they typically contain clearer subjects, cleaner shots, and more controlled recording conditions (Batra et al., 5 Jun 2025). SocialDF therefore redefines the operative task from “detect synthetic artifacts” to a broader multimodal verification problem centered on identity, utterance plausibility, factual correctness, and contextual support.
2. Dataset design and annotation protocol
SocialDF contains 2,126 short-form videos: 1,071 genuine and 1,055 manipulated, together with 869 “potential deepfake targets,” described as public figures who are plausible subjects for impersonation (Batra et al., 5 Jun 2025). The data are drawn from publicly accessible social-media content, especially Instagram Reels and Stories, as well as related social media accounts and hashtag-driven searches. Collection combines manual search and automated scraping. Manual search uses keywords such as “deepfake,” “face swap,” and “parody,” explores Instagram accounts known for posting face manipulations, and intentionally prioritizes videos that are difficult to classify by eye. Automated routines scrape posts from relevant hashtags and user profiles (Batra et al., 5 Jun 2025).
The labeling process is explicitly pragmatic and evidence-driven. Manipulated content is identified through uploader-provided indicators such as hashtags like #deepfake, captions explicitly stating that the video was generated or altered, and mentions of tools such as Parrot AI. Real videos are taken from credible or verified accounts without mention of synthetic modification. When labels are ambiguous, annotators review comments and use zero-shot classification to resolve uncertainty, and final labels are obtained through consensus-based verification. The dataset also includes user comment sentiment analysis, which is used to assess how often viewers misclassify deepfakes (Batra et al., 5 Jun 2025).
A major design distinction is contextual richness. SocialDF includes comments, sentiment signals, popularity indicators, uploader cues, and hashtag or caption information, so it functions not only as a media benchmark but also as a social-media context benchmark. The released resource distributes references or URLs rather than redistributed media, and for each target person the collection process gathers up to 20 images to support facial matching. The experimental train-test split is stratified to preserve the near-balanced class distribution (Batra et al., 5 Jun 2025).
| Dataset | Composition | Characterization |
|---|---|---|
| FakeAVCeleb | 570 real, 25,000 fake | Generated with public software; low quality |
| LAV-DF | 36,431 real, 99,873 fake | Generated with public software; low quality |
| AV-Deepfake1M | 500,000 real, 500,000 fake | Large scale; largely generated with public software |
| DeepSpeak v1.0 | 6,226 real, 6,226 fake | Good quality; less variety and fewer camera angles |
| SocialDF | 1,071 real, 1,055 fake | Realistic, real-world deepfakes; very high quality |
The contrast in this table is central to the benchmark’s purpose. SocialDF is not the largest corpus in the comparison, but it is presented as a realism-first dataset intended to capture borderline cases that are difficult for humans as well as machines (Batra et al., 5 Jun 2025).
3. Detection framework and multimodal reasoning pipeline
The SocialDF detection model is a two-stage multimodal fact-checking pipeline (Batra et al., 5 Jun 2025). Stage 1 identifies who appears in the video and transcribes what is said. Each frame is processed with YOLO for face detection; detected face regions are cropped; each crop is passed through FaceNet, which produces a 512-dimensional face embedding; and the embedding is compared against a database of 869 known personalities using cosine similarity. The paper gives the matching rule as
where and are embedding vectors (Batra et al., 5 Jun 2025). In parallel, the audio track is transcribed with Whisper, specifically Whisper Large V3 Turbo in the experiments. The output of Stage 1 is a list of identified individuals and the transcript .
This first stage effectively converts raw video into a structured claim: the identified person or persons said transcript (Batra et al., 5 Jun 2025). That representation is then evaluated by Stage 2, a multi-agent LLM pipeline with three conceptual modules. LLM Agent-1 performs plausibility attribution, checking whether the identified individuals could plausibly have made the statements and whether stylistic, contextual, and semantic alignment is present. LLM Agent-2 performs factual and ethical verification, using web search to retrieve supporting or contradicting evidence, checking whether the claims in are factually accurate, and assessing whether the statements propagate misinformation or harmful content. A final LLM consolidator synthesizes the two analyses and outputs the authenticity or deepfake decision (Batra et al., 5 Jun 2025).
A notable design choice is that the LLM receives only the transcript and identified individuals, not video titles or descriptions, in order to avoid leakage from web results. The method is therefore not merely a face-voice matching system. It can flag a video even when the face is real and the lips appear synchronized if the attributed speech is implausible for that person or the claims are contradicted by external evidence. In the paper’s own formulation, the framework incorporates linguistic, behavioral or style, and contextual or factual signals, making it closer to multimedia claim verification than to conventional artifact spotting (Batra et al., 5 Jun 2025).
4. Experimental evaluation and performance profile
The experimental section benchmarks SocialDF with LipFD as a strong lip-sync detector and then evaluates the proposed LLM-based framework (Batra et al., 5 Jun 2025). LipFD is trained from scratch on SocialDF using a 90/10 train-test split that is stratified and constructed so that there is no overlap in clips or subjects. For the LLM system, the experiments compare Llama 3.3 8B, Qwen 2.5 7B, and DeepSeek R-1 Llama 8B, while varying temperature values of 0.3, 0.5, and 0.7. A temperature of 0.5 is reported as the best setting because it balances determinism and diversity. DuckDuckGo is used for web search, and Whisper Large V3 Turbo is used for transcription (Batra et al., 5 Jun 2025).
| Method | Accuracy | F1 |
|---|---|---|
| LipFD | 50.94%–51.64% validation accuracy | Not reported |
| Llama 3.3 8B | 89.5% | 0.90 |
| Qwen 2.5 7B | 87.4% | 0.89 |
| DeepSeek R-1 Llama 8B | 90.4% | 0.93 |
LipFD performs near chance and plateaus at roughly 50% to 51% validation accuracy. Its false positive rate is reported as 0.00%, while its false negative rate is about 98.38% to 99.80%, indicating that it is strongly biased toward predicting “real” (Batra et al., 5 Jun 2025). The paper interprets this failure as a consequence of the benchmark’s construction: scene cuts, multiple speakers, occlusion, heavy text overlays, non-frontal viewpoints, and noisy editing violate the assumptions of lip-sync-centered detectors.
By contrast, the LLM-based framework substantially outperforms the lip-sync baseline. DeepSeek R-1 Llama 8B yields the strongest result at 90.4% accuracy and 0.93 F1, followed by Llama 3.3 8B and Qwen 2.5 7B. The authors attribute DeepSeek’s performance to stronger reasoning and self-reflection capabilities derived from reinforcement learning and chain-of-thought-style reasoning (Batra et al., 5 Jun 2025). More broadly, the result supports the benchmark’s central claim: on realistic social-media deepfakes, semantic and contextual verification can exceed the effectiveness of narrow audio-visual synchronization models.
5. Conceptual significance, deployment implications, and limitations
The principal contribution of SocialDF is a reframing of deepfake detection on social media. Traditional detectors often assume continuous visibility of a single frontal face, stable audio-visual synchronization cues, and relatively clean video composition. SocialDF instead treats authenticity as a composite property involving identity verification, transcript extraction, plausibility attribution, factual verification, and contextual analysis (Batra et al., 5 Jun 2025). This is why the framework is described as a fact-checking system rather than a narrow lip-sync classifier.
That reframing has clear deployment implications. The paper presents potential use cases that include automatic flagging of manipulated public-figure content, severity classification for moderation, real-time mobile fact-checking, and misinformation awareness tools (Batra et al., 5 Jun 2025). At the same time, the paper also identifies several practical constraints. The method depends on web search and external reasoning, which may not always be available or reliable in real-time deployment. It may struggle when faces are not visible enough for identity recognition, when speech is unclear or heavily edited, when search results are misleading or insufficient, or when content appears in languages and domains not well covered by the LLM (Batra et al., 5 Jun 2025).
The benchmark itself is also bounded in scope. It focuses on celebrities and other high-profile individuals, so generalization to ordinary users is uncertain. It is platform-dependent, and the authors note that social-media trends may shift over time. The dataset reflects demographic and language biases of social platforms, adversarial robustness is recognized but not yet fully tested, and the experimental comparison has not yet covered many other state-of-the-art detectors beyond LipFD (Batra et al., 5 Jun 2025). The paper further notes a moderation risk: because the method can flag content on the basis of contextual implausibility rather than perceptual artifacts alone, imperfect external reasoning could over-flag satire, parody, or ambiguous speech. A common misconception is therefore that SocialDF is simply a larger deepfake corpus; the paper’s actual contribution is broader and closer to multimodal authenticity assessment.
6. Relation to broader social-data research
In adjacent arXiv literature, the problems that cluster around the SocialDF label often revolve around one recurring theme: social signals are informative but noisy, and systems must decide when to trust them. In social recommendation, “Denoised Self-Augmented Learning for Social Recommendation” argues that social information is often noisy because users connect for reasons unrelated to preference, and it proposes a dual-view framework in which the interaction view judges whether a social edge is reliable (Wang et al., 2023). “Score-based Generative Diffusion Models for Social Recommendations” takes a related position under low social homophily, using score-based generative diffusion to generate denoised social representations aligned with collaborative signals (Liu et al., 2024). By contrast, “Social Data Analysis: A Study on Friend Rating Influence” reports weak linear association between a user’s rating and friends’ average rating on Yelp, with and , and an even weaker relationship for review volume, with and , concluding that ratings are primarily shaped by the user’s own experience (Gupta et al., 2017).
A second neighboring line of work concerns social data infrastructure rather than recommendation or media forensics. “Enabling Social Applications via Decentralized Social Data Management” presents Prometheus as a socially-aware peer-to-peer service that aggregates fragmented social information into a directed, weighted, labeled multigraph, exposes non-trivial inference APIs, and enforces user-defined access policies (Kourtellis et al., 2012). “Socially-Aware Distributed Hash Tables for Decentralized Online Social Networks” rearranges DHT identifiers so socially close users become overlay-close, reporting almost 30% lower lookup latency and nearly 10% better communication reliability through trusted contacts (Nasir et al., 2015). A third strand uses social data for research support and cross-domain modeling: one platform explicitly combines social network dynamics, forum features, and gamification while giving researchers unrestricted data access under consent and a compromise term (Caldas et al., 2018), whereas “Social Knowledge for Cross-Domain User Preference Modeling” projects users into a pretrained social embedding space learned from Twitter/X and reports effective zero-shot personalization across 14 topical categories (Lotan et al., 10 Mar 2026).
Taken together, these works show that the label “SocialDF” sits at the intersection of several technical agendas: realistic social-media deepfake detection, denoised social recommendation, decentralized social-data management, and social-signal-based user modeling. The formal benchmark named SocialDF (Batra et al., 5 Jun 2025) belongs to the deepfake branch, but it also inherits a broader research lesson that appears across the surrounding literature: raw social context is rarely a ground-truth signal, and effective systems must evaluate its reliability before using it for inference.
