LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection

Abstract: The rapid development of LLMs has transformed fake news detection and fact-checking tasks from simple classification to complex reasoning. However, evaluation frameworks have not kept pace. Current benchmarks are static, making them vulnerable to benchmark data contamination (BDC) and ineffective at assessing reasoning under temporal uncertainty. To address this, we introduce LiveFact a continuously updated benchmark that simulates the real-world "fog of war" in misinformation detection. LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information rather than on memorized knowledge. We propose a dual-mode evaluation: Classification Mode for final verification and Inference Mode for evidence-based reasoning, along with a component to monitor BDC explicitly. Tests with 22 LLMs show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems. More importantly, our analysis finds a significant "reasoning gap." Capable models exhibit epistemic humility by recognizing unverifiable claims in early data slices-an aspect traditional static benchmarks overlook. LiveFact sets a sustainable standard for evaluating robust, temporally aware AI verification.

• The paper introduces a dynamic, time-aware benchmark that evaluates LLM-driven fake news detection using continuously updated evidence and dual-mode scoring.
• The methodology leverages monthly updates, fine-grained temporal slicing, and a structured contamination audit to measure models’ reasoning and uncertainty under real-world conditions.
• Cost–performance experiments demonstrate that mid-scale sparse mixture-of-experts models achieve near state-of-the-art results at significantly lower costs compared to dense architectures.

LiveFact: A Transformational Paradigm for Dynamic, Temporal, and Contamination-Aware Fake News Detection Evaluation

Motivation and Benchmark Architecture

The LiveFact benchmark introduces a paradigm shift in LLM-centered fake news detection evaluation, directly addressing the fundamental limitations of static, contamination-prone datasets. Traditional benchmarks, ranging from LIAR [wang2017liar] to FEVER [thorne2018fever], offer temporally static, fixed-snapshot corpora that are fundamentally misaligned with the continuous evolution of news, LLM pre-training, and real-world misinformation ecology. The consequence is twofold: massive data contamination risk due to benchmark leakage into pre-training, and an inability to measure evidential reasoning or epistemic humility under temporal uncertainty.

LiveFact’s solution is three-pronged: (i) it continuously updates the benchmark monthly using newly emergent real-world events, (ii) it structures evidence along fine-grained temporal slices, and (iii) it rigorously controls and quantifies data contamination (“Benchmark Data Contamination,” BDC) using a dedicated perturbation and assessment subframework (SSA). The task is formalized as conditional claim verification under temporally indexed evidence sets $E^{(\delta)}, \delta \in \{-3, 0, +3\}$ days around the event. Dual-mode evaluation is core: Classification Mode assesses outright verdict against ground truth (mirroring classical datasets), while Inference Mode dynamically adjusts labels according to evidence available at the temporal slice, critically introducing “Ambiguous” when, for humans, a defensible verdict is not possible—a simulation of the real “fog of war” facing fact-checkers.

Figure 1: The LiveFact pipeline for real-time event scraping, temporal evidence set construction, LLM-driven claim/context generation, human refinement, and dual-mode evaluation with explicit contamination audit.

Evidence, Claim, and Grade Generation Pipeline

LiveFact’s construction pipeline integrates automated and human-in-the-loop methodology. Real-world events are excerpted daily (e.g., Nov. 2025 batch described in the paper: 737 events), with evidence sets strictly segmented by publication date relative to the event. Exemplary evidence set density allows for multi-hop, context-rich analysis. Claims (spanning “Real,” “Fake,” and “Ambiguous”) and background context are synthesized by high-quality LLMs (OpenAI o4-mini and gpt-4o-mini), followed by multi-round expert verification to ensure fidelity and class balance. Inference Mode labels are assigned post hoc based on evidence sufficiency at each $\delta$; this generates drastic label distributional shift (e.g., pre-event slices are $\sim85\%$ “Ambiguous”).

Dual-Mode Evaluation: Measuring True Reasoning and Epistemic Uncertainty

The benchmark establishes 12 axes of evaluation per model: accuracy and macro-F1 for both Classification and Inference Mode across all three temporal slices. This design isolates two confounding effects: (1) ‘Overconfidence’—the propensity of models to hallucinate decisive verdicts despite insufficient or absent evidence (quantified via a negative or minimal Reasoning Gap); (2) ‘Uncertainty Awareness’—the model’s capacity to predict “Ambiguous” in the face of information voids, which is essential for trustworthy deployment in real-time scenarios.

Empirical analysis demonstrates that strong models (e.g., Qwen3-235B-A22B-Instruct-2507, GPT-5.1, GPT-OSS-120b in verbose/“thinking” mode) both maximize final task accuracy and exhibit large positive Reasoning Gaps (Inference--Classification at $\delta=-3$), indicating robust detection of ambiguous context—a proxy for “epistemic humility.” In contrast, standard instruction-tuned models with low or negative gap typically overfit to classification without epistemic balance, and base models, lacking proper RLHF/SFT alignment, disintegrate from format non-compliance.

Figure 2: Models with large positive Reasoning Gaps (green) correctly predict ambiguity under evidence void, while negative/low gaps (red) reveal overconfidence or base format failure.

Temporal analysis substantiates this: at pre-event evidence slices, capable models recover accuracy in Inference Mode by leveraging "Ambiguous," but not in Classification Mode (forced verdict even without information). Diagnostic confusion matrix visualization affirms that only instruction-aligned and sufficiently scaled models meaningfully split “Ambiguous” at $\delta=-3$.

Figure 3: Performance evolution for several model families. In Classification Mode, all scores drop significantly under pre-event evidence; robust models in Inference Mode recover by leveraging “Ambiguous.”

Contamination Quantification and Control

Addressing evaluation inflation via BDC is central. LiveFact incorporates the SSA framework, introducing “entity shift” perturbations at all data entry fields (e.g., “Trump” $\rightarrow$ “Wannetta”), allowing direct computation of Overturn Rate (OTR) and performance drop $\Delta$ between original and shifted data. The SSA Factor ($\Delta \times OTR \times 100$) operationalizes contamination risk: a latent memorization or leakage is exposed by sharp accuracy/F1 degradation under entity-shift, unattainable by genuinely reasoned models.

Simulated contamination experiments (intentional fine-tuning on target data) demonstrate that contaminated models exhibit massive SSA Factor increases (e.g., $>6.0$), while clean models are $<0.2$. This empirically verifies that SSA is highly sensitive to semantic-level and label-level contamination, and that LiveFact maintains dynamic integrity for its intended high-stakes application.

Cost–Performance and Practical Deployability

A core experimental finding involves the cost–performance analysis. Mixture-of-Experts (MoE) architectures (Qwen3, DeepSeek, GPT-OSS) vastly outperform dense models (Llama series) in both average and peak macro-F1/accuracy, and dominate the efficiency frontier. For example, Qwen3-30B-A3B-Instruct-2507 achieves performance within 3pp of the state-of-the-art (SOTA) at 1/14th the cost of equivalently performing GPT-5.2 models. This demonstrates that well-tuned, mid-scale sparse MoEs are optimal for real-time, cost-sensitive applications, while “verbose,” reasoning-first models require nontrivial resource allocation but can match or exceed SOTA when evaluated with sufficiently extended output windows (1k+ tokens).

Figure 4: LiveFact cost-vs-performance: Qwen3-235B-A22B-Instruct-2507 maximizes raw performance; Qwen3-30B-A3B-Instruct-2507 defines optimal cost-efficiency, surpassing several proprietary GPT models at >10x lower cost.

Failure Modes and Model Archetypes

LiveFact exposes two principal model failure modes: (i) Overconfidence, typical of overfitted SFT models adhering to classification even under uncertainty; (ii) Format Non-Compliance, endemic in base pre-trained models lacking SFT/RLHF, unable to parse instructions or delimit output as required for automated grading. Ablation shows that instruction tuning is non-optional for robust, production-grade LLM-based verification agents; however, evidence suggests that pre-training with sufficient structured, logic-rich corpora (e.g. Qwen3-8B-Base) confers a partial capacity for uncertainty recognition, though SFT remains necessary to unlock full performance.

Theoretical and Practical Implications

LiveFact establishes new minimal standards for LLM verification evaluation: dynamic, temporally segmented, dual-mode scored, and contamination-audited. The benchmark’s architecture directly supports longitudinal monitoring of data drift, contamination, and emergent behavior—features unattainable for all prior static or pseudo-dynamic datasets in the verification literature (as documented in the extensive comparative table in the manuscript). Importantly, LiveFact’s entity shift paradigm for contamination detection is broadly applicable to other factuality- and reasoning-sensitive LLM tasks.

Practically, the performance gap between MoE and dense architectures, the cost–performance advantage of midscale open MoEs, and the demonstrable necessity of proper alignment protocols imply that operational deployment of LLM fake news detection agents (e.g., in social platforms, journalistic workflows, or governmental alerting systems) should prioritize open, cost-efficient, instruction-aligned, agent-like models continuously audited by a LiveFact-like dynamic standard.

Future Directions

The authors highlight three main extensions for LiveFact: (i) multilingual and cross-cultural coverage to manage regional misinformation and linguistic complexity; (ii) incorporation of non-textual (multimodal) evidence for alignment with current content formats; and (iii) semi-automated expert-in-the-loop expansion for greater scaling without ground-truth tradeoff. Each of these aligns with critical open frontiers in LLM reliability and robust digital fact-checking [ma2024ex, hu2025mage, shibu2025scarcity].

Conclusion

LiveFact operationalizes a sustainable and technically sound standard for evaluating LLM-driven factuality systems under real-world distributional shift, temporal ambiguity, and contamination risk (2604.04815). Its dynamic methodology, dual-mode scoring, and explicit contamination control address deep-rooted issues left unexamined by static benchmarks and provide a replicable template for high-integrity, cost-efficient, and trustworthy AI verification evaluation. As LLMs become further embedded in the information ecosystem, paradigms exemplified by LiveFact will be essential for the development and governance of temporally robust, epistemically credible verification agents.