The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality (2512.10791v1)

Abstract: We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of LLMs to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .

• The paper presents FACTS Leaderboard, a benchmark measuring LLM factuality across four facets: Multimodal, Parametric, Search, and Grounding.
• It combines human and automated evaluations to quantify metrics such as coverage, contradiction avoidance, and hedging with clear, reproducible scores.
• Current models achieve sub-70% aggregate accuracy, highlighting significant challenges and opportunities for improving LLM factual reliability.

The FACTS Leaderboard: A Comprehensive Benchmark for LLM Factuality

Overview of the Benchmark Suite

The FACTS Leaderboard provides an extensive evaluation protocol for measuring the factual accuracy of LLM generations. This protocol aggregates four specialized benchmarks, each probing a unique facet of model factuality: Multimodal, Parametric, Search, and Grounding. By reporting both granular scores per facet and a composite "FACTS Score," it offers a robust quantitative and qualitative assessment of factual reliability, spanning diverse modalities, knowledge sources, and response forms.

Benchmark Components

FACTS Multimodal

FACTS Multimodal evaluates factuality in image-grounded open-ended question answering. The assessment protocol employs detailed human-authored rubrics comprising Essential and Non-Essential facts for each query. Automated autoraters score model outputs on both coverage (essential fact recall) and contradiction avoidance. Only outputs covering all essential facts without contradiction attain a positive accuracy verdict. Gemini models display elevated coverage-oriented recall, while the GPT-family exhibits precision-oriented performance, with higher contradiction avoidance. Autorater-human reliability is substantial: macro F1 for coverage 72.3 and contradiction 78.2.

Figure 1: Sentence-level annotation interface for fine-grained factuality assessment; annotators label each sentence for contradiction and provide textual reasoning.

FACTS Parametric

The Parametric slice assesses the capacity of models to answer closed-book factoid queries solely from internal parametric knowledge. The benchmark construction pipeline performs adversarial sampling against open-source models to isolate challenging, user-driven queries; all gold answers are strictly supported by Wikipedia. Human annotation ensures compliance with query design criteria, unambiguous answerability, and factual support. The evaluation protocol leverages an auto-grader with explicit "unknown" and "not-attempted" options, mitigating ambiguity and incentivizing honest uncertainty calibration. Metrics reported include accuracy, F1, hedging (non-attempt), and attempted-accuracy.

Figure 2: Distribution of domain contexts and answer types in FACTS Parametric, reflecting real-world user query statistics.

Gemini 3 Pro leads with 76.4% accuracy, while models such as GPT-5 mini exhibit elevated hedging rates, highlighting significant calibration variance.

FACTS Search

FACTS Search measures factual response quality when models incorporate active search engine queries, using the Brave Search API. The test set, covering tail entities and multi-hop queries, is partitioned into subsets sourced from human raters and synthetic generation (Wiki Two-Hop, Wiki Multi-Doc, KG Hops). Human annotation enforces answer correctness, uniqueness, and temporal immutability. The Search slice is designed such that no evaluated question is answerable by closed-book LLMs, emphasizing tool reliance.

Metrics include overall and attempted accuracy, F1, hedging, and mean query count per answer. Gemini 3 Pro achieves the highest accuracy (83.8%), despite lower mean search invocations versus Grok and Claude, suggesting more efficient contextualization per query.

FACTS Grounding v2

Grounding v2 probes fidelity in long-form context-conditioned generation, utilizing queries paired with extensive documents across vertical domains. The principal metric is factuality conditioned on grounding, scored using two judge LLMs (Gemini 2.5 Flash and GPT-5), selected for reduced self-preference and calibrated according to macro F1 on held-out human-rated samples. Additional eligibility filters penalize responses that evade fulfilling user intent, thereby minimizing factuality "hacks" attributable to vacuous or nonresponsive outputs.

Figure 3: Distribution of domain contexts and user task types in FACTS Grounding, highlighting enterprise document diversity and prompt complexity.

Composite accuracy across all benchmarks ("FACTS Score") is currently capped at 69% (Gemini 3 Pro), leaving significant headroom for architectural innovation and algorithmic improvements.

Key Results and Strong Claims

• Headroom for Progress: No model exceeds 70% aggregate accuracy, even across distinct domains, modalities, and tasks.
• Prompt+Judge Diversity: Model outputs are assessed against auto-raters and judge LLMs selected to minimize bias and ensure reliability; dual-LM judging protocols outperform single-LM and self-evaluation.
• Calibration/Uncertainty: Explicit reporting of "hedging rate" and "attempted-accuracy" circumvents gaming by overconfident guessing and incentivizes appropriate uncertainty handling.
• Multimodal Recall-Precision Tradeoff: Gemini models favor recall (fact coverage), GPT models favor precision (contradiction avoidance), emphasizing a practical accuracy-completeness tradeoff.

Implications and Outlook

Practically, FACTS sets a new standard for comprehensive, reproducible factuality evaluation—especially critical as LLM adoption accelerates in domains subject to regulatory constraints (e.g., finance, medicine). The inclusion of tool-use (search), multimodal scenarios, and long-context conditioning provides granular diagnostics, enabling research into calibration, tail-entity recall, and tool integration efficacy.

Theoretically, the observed headroom underscores limitations in current architectures to generalize complex factual dependencies, integrate multimodal signals, and maintain calibration under adversarial inputs. As benchmarks like FACTS become standard, we expect research to:

• Refine Auto-Rating: Develop more granular rater LLMs and alignment verification techniques, mitigating self-preference and reference-free bias [NEURIPS2023_91f18a12, wataoka2024selfpreferencebiasllmasajudge, liu-etal-2024-LLMs-narcissistic].
• Advance Tail-Knowledge Acquisition: Challenge models to overcome long-tail entity representation failures [KandpalDRWR23].
• Drive Cross-Modal Grounded Generation: Improve fusion of visual and textual knowledge for reduced hallucination in multimodal responses.
• Design Benchmarks Beyond Text and Static Knowledge: Encompass video-based context, fast-changing world facts, and novel tool-use scenarios (browsing, programmatic reasoning).
• Experiment with Human-in-the-Loop Calibration: Incentivize LLMs to defer, hedge, or abstain in line with epistemic uncertainty.

Conclusion

The FACTS Leaderboard suite represents an advance in factuality benchmarking for LLMs, capturing multimodal, parametric, retrieval-based, and context-grounded dimensions. Current leader models attain sub-70% aggregate accuracy, indicating substantive unsolved challenges. The suite's detailed metrics and evaluation structure will catalyze future research into factual reliability, calibration, and robust grounding, with anticipated expansion into new modalities and dynamic information domains.

Figure 1: Sentence-level annotation interface illustrating fine-grained contradiction identification and annotation workflow.

Figure 2: Distribution of question domains and answer types in FACTS Parametric, reflecting coverage of real-world factual QA.

Figure 3: Distributions of context domains and user task types in FACTS Grounding, evidencing prompt diversity and challenge space.

• "The FACTS Leaderboard: A Comprehensive Benchmark for LLM Factuality" (2512.10791)
• "LLMs Struggle to Learn Long-Tail Knowledge" [KandpalDRWR23]
• "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" [NEURIPS2023_91f18a12]
• "Self-preference bias in LLM-as-a-judge" [wataoka2024selfpreferencebiasllmasajudge]
• "LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores" [liu-etal-2024-LLMs-narcissistic]
• "TruthfulQA: Measuring How Models Mimic Human Falsehoods" [lin2022truthfulqa]
• "BrowseComp: A simple yet challenging benchmark for browsing agents" [wei2025browsecomp]
• "Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities" (Comanici et al., 7 Jul 2025)