Fakeddit: Reddit's Multimodal Misinformation Benchmark

• Fakeddit is a large, Reddit-derived multimodal benchmark designed for fake news and misinformation research, offering over 1 million text, image, and metadata samples.
• It supports flexible hierarchical classification tasks (2-way, 3-way, and 6-way) to facilitate both binary detection and fine-grained misinformation categorization.
• Researchers use Fakeddit in varied applications such as stream learning, graph-based outlier detection, and domain-transfer studies to evaluate models on noisy, real-world data.

Fakeddit is a Reddit-derived benchmark for fake news and misinformation research that was introduced as a large-scale, weakly supervised, multimodal dataset with over 1 million samples, paired text and images, metadata, comments, and hierarchical 2-way, 3-way, and 6-way labels assigned through distant supervision (Nakamura et al., 2019). In the literature, it functions both as an original multimodal benchmark and as a source corpus that later work repeatedly reformulates for binary fake-versus-real classification, fine-grained misinformation categorization, chronologically ordered stream learning, graph-based outlier detection, and domain-transfer studies (Alonso-Bartolome et al., 2021, Zyblewski et al., 2024, Liu et al., 26 Nov 2025). This dual role makes Fakeddit less a single fixed task than a benchmark family whose methodological significance depends heavily on which subset, label mapping, modality configuration, and evaluation protocol are adopted.

1. Dataset conception and scope

Fakeddit was created to address three limitations identified in earlier fake news resources: many were primarily text-only, many were relatively small, and many supported only coarse labels such as true versus false (Nakamura et al., 2019). The dataset was designed explicitly for multimodal fake news research, especially text+image learning, while also enabling finer-grained categorization beyond binary detection. The original paper also frames it as a benchmark not only for fake news classification, but for related tasks such as implicit fact-checking and image-caption verification (Nakamura et al., 2019).

The source domain is Reddit. The data were collected from 22 subreddits chosen to correspond to different truthfulness or misinformation-related themes, spanning categories labeled True, Satire/Parody, Misleading Content, False Connection, Imposter Content, and Manipulated Content (Nakamura et al., 2019). The corpus is broad rather than narrowly political, which is reflected in later work that treats Fakeddit as more heterogeneous and less curated than fact-checking datasets such as Politifact or GossipCop (Kheddache et al., 26 Sep 2025). This breadth is one reason later studies repeatedly use Fakeddit as a stress test for models expected to operate on noisy, user-generated, social-media-style content rather than on professionally curated claims (Kheddache et al., 26 Sep 2025).

The original statistics reported for Fakeddit include 1,063,106 total samples, 682,996 multimodal samples, 22 subreddits, 358,504 unique users, 24,203 unique domains, a timespan from 3/19/2008 to 10/24/2019, mean words per submission of 8.27, mean comments per submission of 17.94, and vocabulary size 175,566 (Nakamura et al., 2019). The paper also reports 628,501 fake samples and 527,049 true samples, while noting counts that appear somewhat inconsistent with the stated total if interpreted as disjoint categories; this is a documented reporting inconsistency rather than a later reinterpretation (Nakamura et al., 2019).

2. Collection pipeline, modalities, and labeling

The data were collected with the pushshift.io API from Reddit submissions and associated content (Nakamura et al., 2019). For each submission, the collected fields include the submission title, image, comments, and metadata such as score, author username, subreddit source, source domain, number of comments, and upvote/downvote ratio (Nakamura et al., 2019). A special case is r/photoshopbattles: original submissions contain real images, while comments often contain manipulated derivative images plus associated text, and these comments were treated as submission data so that manipulated multimodal examples could be included in the benchmark (Nakamura et al., 2019).

The dataset is multimodal in a broad sense. The primary textual field used in the original experiments is the Reddit submission title. Images are submission thumbnails or associated images. Metadata and comments are included in the release but were not used in the original baseline experiments (Nakamura et al., 2019). Later work repeatedly exploits this latent breadth differently: some studies use only text and image, some use text only, some add comments, and graph-oriented work converts comments or comment users into propagation nodes (Jun-hao et al., 2024, Zyblewski et al., 2024, Liu et al., 26 Nov 2025).

Labeling is based on distant supervision rather than item-level fact-checking. Each subreddit is assigned one label in each of the 2-way, 3-way, and 6-way schemes, and all posts from that subreddit inherit those labels (Nakamura et al., 2019). The review process described in the original paper includes subreddit moderation as a first refinement layer, filtering out submissions with score < 1, a manual audit of 10 posts per subreddit with removal of subreddits that did not match their intended theme, text cleaning to reduce label leakage, and a manual agreement check on 150 text-image pairs for the 6-way task (Nakamura et al., 2019). The reported agreement is Cohen’s kappa = 0.54, interpreted there as moderate agreement (Nakamura et al., 2019).

This means Fakeddit is not a gold-standard item-level fact-checking corpus. The labels are proxies derived from subreddit theme, and the original paper is explicit that some items plausibly fit more than one category even though only one 6-way label is assigned (Nakamura et al., 2019). A common misconception is therefore that Fakeddit provides definitive truth judgments for each post. The original benchmark instead trades item-level factual certainty for scale, multimodality, and label granularity (Nakamura et al., 2019).

3. Task formulations and benchmark structure

Fakeddit supports hierarchical 2-way, 3-way, and 6-way classification (Nakamura et al., 2019). The 2-way setup distinguishes True from Fake. The 3-way setup separates Completely true, Fake and contains text that is true, and Fake with false text. The 6-way taxonomy contains True, Satire/Parody, Misleading Content, Imposter Content, False Connection, and Manipulated Content (Nakamura et al., 2019). This taxonomy is central to the benchmark’s identity, and later work that reduces Fakeddit to binary classification is using only one slice of the original design.

The original release also specifies splits of 878,218 training, 92,444 validation, 92,444 released test, and 92,444 unreleased set (Nakamura et al., 2019). However, later papers frequently do not retain this exact benchmark configuration. Some use custom subsets, some retain only multimodal examples, and some define new train/test partitions. This suggests that “results on Fakeddit” are not automatically comparable across papers unless the subset, label mapping, and split protocol are stated explicitly.

A concise summary of core benchmark properties reported in the original dataset paper is given below.

Property	Reported value	Source
Total samples	1,063,106	(Nakamura et al., 2019)
Multimodal samples	682,996	(Nakamura et al., 2019)
Subreddits	22	(Nakamura et al., 2019)
Classification schemes	2-way, 3-way, 6-way	(Nakamura et al., 2019)
Training split	878,218	(Nakamura et al., 2019)
Validation split	92,444	(Nakamura et al., 2019)
Released test split	92,444	(Nakamura et al., 2019)
Unreleased set	92,444	(Nakamura et al., 2019)

Several later studies illustrate how variable the operational task can become. Fine-grained multimodal work treats Fakeddit as a 6-class problem over post title and image (Alonso-Bartolome et al., 2021). Transfer-learning work uses a 5,997-sample multimodal subset with 3,523 true and 2,474 misinformation samples, evaluating binary, 3-class, and 6-class variants (Jager et al., 2023). Stream-learning work uses 682,996 multimodal samples, discards the image modality during classification, sorts the examples by creation timestamp, and constructs a binary stream of 2,731 chunks of size 250 (Zyblewski et al., 2024). Large multimodal model evaluation uses a 300-sample multimodal subset with 50 fake and 250 real items as a zero-/few-shot evaluation benchmark rather than a training dataset (Kheddache et al., 26 Sep 2025). These are all Fakeddit-based tasks, but not identical benchmarks.

4. Early modeling results and multimodal baselines

The original Fakeddit paper benchmarked text-only, image-only, and multimodal hybrid models, showing that multimodal models outperform text-only models, which in turn outperform image-only models across 2-way, 3-way, and 6-way classification (Nakamura et al., 2019). The strongest text encoder in that study was BERT-Large, Uncased, with test accuracies of 0.8644 for 2-way, 0.8580 for 3-way, and 0.7677 for 6-way classification (Nakamura et al., 2019). The strongest image-only encoder was ResNet50, with 0.8070, 0.7988, and 0.7549 on the same tasks (Nakamura et al., 2019). The best overall multimodal model was BERT + ResNet50 + maximum fusion, achieving 0.8909 for 2-way, 0.8890 for 3-way, and 0.8588 for 6-way (Nakamura et al., 2019).

A later fine-grained study on six-way Fakeddit classification similarly concludes that exploiting both text and image is beneficial, though with a different architecture and evaluation narrative (Alonso-Bartolome et al., 2021). In that work, BERT is the best unimodal text model with 0.78 accuracy, while a multimodal CNN reaches 0.87 accuracy on the 6-class task (Alonso-Bartolome et al., 2021). The paper emphasizes that some categories, especially Manipulated content, Satire/Parody, and False connection, benefit strongly from images, whereas Imposter content remains difficult and does not benefit from images in the same way (Alonso-Bartolome et al., 2021). This aligns with the original benchmark’s motivation: some misinformation categories are inherently cross-modal rather than purely textual (Nakamura et al., 2019).

Later multimodal architectures continue to use Fakeddit as a headline benchmark for stronger fusion or graph-enhanced methods. MAGIC, a graph-based multimodal model over post text, image, and comments, reports 98.72 accuracy, 98.68 precision, 98.44 recall, and 98.56 F1 on a 2-way Fakeddit setting, and 97.60 across all four metrics on a 3-way setting, but on a small subset of 3,127 examples with a custom 80/20 split (Jun-hao et al., 2024). GAMED, a knowledge-adaptive multi-experts decoupling model using 2-way labels on the standard large-scale split, reports 93.93 accuracy and 93.63 F1 on Fakeddit (Shen et al., 2024). E-CaTCH, an event-centric cross-modal attention model with temporal consistency and class-imbalance handling, reports 95.5 accuracy, 0.955 F1, and 0.975 AUC on Fakeddit (Mousavi et al., 15 Aug 2025). MOMENTA reports 0.965 accuracy, 0.959 F1, 0.982 AUC, and 0.918 MCC on Fakeddit in binary form (Abdollahinejad et al., 17 Apr 2026).

These results are not directly interchangeable. Some use the original large-scale split, some use reduced subsets, some use binary remappings, and some introduce event-centric or temporal aggregation over timestamp-sorted posts. A plausible implication is that Fakeddit should be treated less as a single leaderboard and more as a common source dataset supporting multiple non-equivalent benchmark instantiations.

5. Reinterpretations across later research

A major feature of Fakeddit’s research afterlife is that later work repeatedly redefines the benchmark to target different methodological questions. This is not incidental; it reflects the fact that the dataset contains multimodal content, timestamps, metadata, comments, and hierarchical labels, making it unusually adaptable.

In data stream learning, Fakeddit is used as a chronologically ordered text stream rather than a static classification corpus. One study keeps only multimodal samples, uses only text, sorts the data by creation timestamp, partitions the stream into 2,731 chunks of 250 samples, and applies a Test-Then-Train protocol (Zyblewski et al., 2024). The proposed Streaming Sentence Space method converts each short text into an image-like matrix and uses ResNet-18 in a batch-based stream setting. The paper reports that this method is the only one maintaining an average balanced accuracy of about 80% on the Fakeddit stream (Zyblewski et al., 2024). This reformulation shifts the benchmark from multimodal detection to temporally ordered, dynamically imbalanced stream classification.

In transfer-learning work, Fakeddit serves as both a benchmark and a non-local source domain. A multimodal model called MMiC is evaluated on a 5,997-sample multimodal Fakeddit subset and obtains test F1 values of 0.88 for binary, 0.88 for 3-class, and 0.83 for 6-class classification (Jager et al., 2023). The same paper then uses Fakeddit as the non-local training source for transfer to a South African misinformation dataset, concluding that Fakeddit-only training transfers poorly and that local South African data substantially improves performance (Jager et al., 2023). This suggests that Fakeddit is useful for predeployment experimentation but insufficient by itself for culturally specific deployment settings.

In multimodal fact verification and knowledge transfer, Fakeddit is treated as a large two-label multimodal target benchmark whose performance can improve through intra-domain dataset mixtures (Lee et al., 2024). That work reports that CLIP-large-336 trained on a mixture of Mocheg, Vitamin-C, and Fakeddit reaches 93.42 F1, compared with 91.87 F1 for the same model trained on Fakeddit only, and argues that data diversity can outperform scale alone (Lee et al., 2024). LEMMA, which augments large vision-LLMs with external knowledge retrieval, reports 0.824 accuracy and 0.854 F1 on binary Fakeddit, with the paper interpreting this as an approximately 13% relative accuracy improvement over Direct GPT-4V (Xuan et al., 2024). UNITE-FND goes further by converting Fakeddit images into generated text with Gemini 1.5 Pro, creating a family of six transformed Uni-Fakeddit-55k datasets and reporting 92.52% accuracy for binary classification with a text-only downstream model (Mukherjee et al., 16 Feb 2025). This suggests that Fakeddit also serves as a testbed for whether multimodal evidence can be translated into text without catastrophic loss.

In graph-oriented work, Fakeddit becomes a source for propagation-graph benchmarks. TAGFN incorporates a Fakeddit subset as text-attributed propagation graphs, where each original Fakeddit sample becomes a root news node and comment users become child nodes carrying textual attributes (Liu et al., 26 Nov 2025). In that construction, the Fakeddit subset contains 7,249,803 nodes, 6,683,699 edges, and 566,104 graphs, with 59.6% fake labels (Liu et al., 26 Nov 2025). This is a conceptual shift from multimodal post classification to graph-level binary outlier detection over text-attributed graphs (Liu et al., 26 Nov 2025).

A further derivative benchmark isolates a specific distinction internal to Fakeddit’s 6-way taxonomy. WISE constructs a balanced 20,000-sample dataset from Fakeddit for satire-versus-fake classification, using 10,000 Satire/Parody samples and 10,000 fake-news samples formed by combining 5,000 Misleading Content and 5,000 Manipulated Content items (Chhetri et al., 30 Dec 2025). The benchmark uses clean titles longer than 50 characters and stratified 5-fold cross-validation, with MiniLM reaching 87.58% accuracy and RoBERTa-base reaching 95.42% ROC-AUC (Chhetri et al., 30 Dec 2025). This shows that Fakeddit’s fine-grained labels can support focused subproblems not reducible to ordinary real-versus-fake detection.

6. Methodological limitations and recurring controversies

The central methodological caveat is the distant-supervision label design. Because labels are inherited from subreddit membership rather than fact-checked item by item, models trained on Fakeddit may learn subreddit style, community conventions, lexical norms, or domain correlations rather than fully generalizable notions of fake news (Nakamura et al., 2019). The original authors attempted to reduce trivial leakage by removing explicit subreddit-revealing words, but that does not eliminate all community-specific signals (Nakamura et al., 2019). This remains the most important limitation of Fakeddit as a benchmark.

A second limitation is that later papers often redefine the benchmark in ways that complicate comparison. Some retain only multimodal samples, some discard images, some use custom binary mappings, and some create new splits. MAGIC, for example, evaluates on a 3,127-sample subset with a custom 80/20 split rather than an official large-scale split (Jun-hao et al., 2024). MMiC uses a 5,997-sample multimodal subset (Jager et al., 2023). UNITE-FND introduces a 55,000-sample transformed subset family (Mukherjee et al., 16 Feb 2025). This means that nominally higher or lower “Fakeddit results” may refer to substantively different evaluation objects.

A third limitation is that Fakeddit can be exceptionally challenging for prompt-only large multimodal models in noisy in-the-wild settings. A study of GPT-4o-based multimodal disinformation detection reports that Fakeddit is the hardest of five evaluation datasets, with Recall 20%, Specificity 40%, Rejected 12%, and Accuracy 23% on a 300-sample multimodal subset (Kheddache et al., 26 Sep 2025). The paper attributes this poor performance to Fakeddit’s noisy Reddit posts, broad topical diversity, varied writing styles, uneven image quality, and ambiguous text-image relations (Kheddache et al., 26 Sep 2025). It also notes an internal inconsistency in the decision rule for the sixth scoring criterion, creating a reproducibility ambiguity for that experiment (Kheddache et al., 26 Sep 2025). This illustrates a broader point: Fakeddit can expose weaknesses in models that perform much better on more structured, curated, fact-checking datasets.

A fourth issue concerns benchmark identity. Fakeddit is often described as “multimodal,” but many later papers use only text, or use text plus graph structure instead of text plus image, or convert the visual modality into generated text (Zyblewski et al., 2024, Liu et al., 26 Nov 2025, Mukherjee et al., 16 Feb 2025). This does not invalidate those studies, but it means that Fakeddit’s role in the literature is often as a source corpus rather than as a fixed multimodal protocol. A plausible implication is that reproducibility on Fakeddit requires specifying not only the source dataset, but also the exact operational benchmark induced from it.

Overall, Fakeddit is best understood as a large, weakly supervised, Reddit-native benchmark ecosystem rather than a single immutable dataset task. Its enduring value lies in the unusual combination of scale, multimodality, fine-grained labels, timestamps, comments, and social context that allows it to support fine-grained multimodal classification, binary misinformation detection, transfer learning, stream evaluation, graph reformulation, and focused subproblems such as satire-versus-fake discrimination (Nakamura et al., 2019, Zyblewski et al., 2024, Liu et al., 26 Nov 2025). Its central tradeoff remains the same as in the original release: broad coverage and methodological flexibility at the cost of noisier supervision and substantial room for benchmark drift across later work.