TAGFN: A Text-Attributed Graph Dataset for Fake News Detection in the Age of LLMs

Abstract: LLMs have recently revolutionized machine learning on text-attributed graphs, but the application of LLMs to graph outlier detection, particularly in the context of fake news detection, remains significantly underexplored. One of the key challenges is the scarcity of large-scale, realistic, and well-annotated datasets that can serve as reliable benchmarks for outlier detection. To bridge this gap, we introduce TAGFN, a large-scale, real-world text-attributed graph dataset for outlier detection, specifically fake news detection. TAGFN enables rigorous evaluation of both traditional and LLM-based graph outlier detection methods. Furthermore, it facilitates the development of misinformation detection capabilities in LLMs through fine-tuning. We anticipate that TAGFN will be a valuable resource for the community, fostering progress in robust graph-based outlier detection and trustworthy AI. The dataset is publicly available at https://huggingface.co/datasets/kayzliu/TAGFN and our code is available at https://github.com/kayzliu/tagfn.

• The paper introduces TAGFN, a benchmark dataset that combines textual content and propagation structures to improve fake news detection.
• The study evaluates methods including zero-shot and one-shot LLM prompting, chain-of-thought reasoning, and supervised GNNs with notable gains in accuracy.
• Ablation studies confirm that both graph structure and user text are essential for enhanced detection performance, exposing LLM context length limitations.

TAGFN: A Text-Attributed Graph Dataset for Benchmarking Fake News Detection with LLMs

Motivation and Dataset Construction

Detecting fake news on social media requires modeling both the propagation dynamics and the textual signals inherent in the content and user behavior. Text-attributed graphs (TAGs) allow simultaneous encoding of structural (propagation) and semantic (textual) information, crucial for robust outlier—specifically fake news—detection. Prior benchmarks are inadequate either in scale, realism, or annotation fidelity, especially regarding graph structure and text attributes required for LLM-powered methods.

TAGFN addresses this gap by providing three large-scale, real-world resource subsets (Politifact, Gossipcop, and Fakeddit) tailored for outlier detection via fake news classification. Each graph corresponds to a news item (root node), child nodes represent propagating users, and all nodes are attributed with raw text content extending beyond simple embeddings. Unix timestamps further encode temporal data per node, although initial experiments suggest limited benefit from naïve inclusion.

Figure 1: A toy example illustrating TAGFN’s graph structure, with a news root and textual user nodes representing propagation.

TAGFN is available for academic use (https://huggingface.co/datasets/kayzliu/TAGFN), including accompanying code and splits faithful to previous benchmarks.

Experimental Protocol and Baseline Analysis

The experimental framework benchmarks multiple categories of detection approaches:

• Zero-shot LLM prompting: direct label prediction without supervision.
• Chain-of-thought reasoning: leveraging LLMs’ generalization and stepwise inference.
• Few-shot in-context learning (ICL): one-, two-, and three-shot examples to probe context-driven label assignment.
• Emb+GNN: supervised models using LLM-generated node embeddings for message-passing architectures (GraphSAGE).

On the Politifact subset, Qwen3-8B's accuracy increases sharply from 51.13% (zero-shot) to 78.28% (one-shot ICL). Chain-of-thought improves unsupervised accuracy to 69.68%. However, supervised Emb+GNN models outperform all prompting-based methods, achieving 84.16% accuracy, and comparable results are observed across other subsets and model families. Notably, increasing the number of ICL examples beyond one consistently degrades performance in larger graphs, attributed to prompt length exceeding effective LLM attention capacity.

LLM Family Comparison and Model Scaling Trends

A comprehensive sweep over LLM size and architecture confirms steep scaling laws in detection accuracy and F1 scores. On Politifact, zero-shot accuracy ranges from 51.13% (Qwen3-8B) to 85.07% (GPT-5), and all one-shot models exceed 78%, demonstrating transfer potential even with minimal context. GPT-4.1 surpasses the supervised baseline, indicating that scaling and prompt engineering can compensate for limited labeled data in some TAG domains.

Ablation Study: Role of Graph Structure and Text

To isolate the effects of different graph components, ablations remove either the graph structure or user text. Results show that omitting either sharply reduces performance; retaining only news content leads to pronounced drops in both accuracy and F1. Full TAGs (i.e., including propagation edges and user posts) yield the best results, underscoring the necessity of both structural and semantic evidence.

Figure 2: Ablation analysis quantifies the importance of graph structure and user posts in boosting one-shot LLM detection accuracy.

Practical and Theoretical Implications

TAGFN enables, for the first time, rigorous benchmarking of LLM- and graph-augmented models for fake news detection in real-world settings. Strong numerical results reveal that one-shot LLM prompting and chain-of-thought reasoning are highly competitive for smaller scales, but traditional GNNs with high-quality embeddings remain most effective for large, richly annotated graphs. The observed context length degradation in LLMs may stimulate future architectural modifications to address graph-data scaling.

The dataset’s holistic encoding of propagation, user history, and temporal attributes will support novel lines of research in trustworthy AI and graph foundation models. Moreover, the public release catalyzes reproducibility and the development of graph-aware LLM benchmarks, pushing the field toward multimodal outlier detection frameworks. Future work should augment TAGFN with dynamic graph encoding and extend comparisons to larger open-source models and diverse LLM families.

Conclusion

TAGFN constitutes the most comprehensive TAG benchmark for fake news detection to date, offering both structural fidelity and textual richness necessary for advanced LLM and graph learning research (2511.21624). Baseline experiments demonstrate the utility of in-context learning and graph-based approaches, with supervised GNNs currently offering the highest accuracy. The impact of context window limitations and the primacy of graph structure in detection accuracy suggest salient directions for future model architectures and dataset extensions.