LongFact Benchmark: Evaluating Factuality

• LongFact is a comprehensive benchmark and framework for evaluating long-form factual responses in LLMs, combining systematic prompt design with advanced factuality metrics.
• It includes detailed sub-benchmarks targeting concept explanations and object-related fact checks across 2,280 curated prompts derived from diverse topics.
• The framework utilizes SAFE, an automated pipeline that decomposes responses into atomic facts, verifies them via search queries, and computes precision, recall, and F1@K metrics.

LongFact is a comprehensive benchmark, evaluation suite, and methodological framework for the measurement and analysis of long-form factuality in LLMs. Developed to probe and facilitate progress in the accurate generation and reliable assessment of open-domain, multi-paragraph factual responses, LongFact introduces a systematic testbed coupled with advanced automated evaluation methodologies and new performance metrics. Its impact extends across research domains in machine learning and natural language processing, providing rigorous tools to benchmark factual accuracy, depth, coverage, and reliability in long-form outputs.

1. Construction of the LongFact Benchmark

LongFact comprises a carefully curated prompt set designed to elicit multi-paragraph, open-ended factual answers from LLMs. The dataset encompasses two parallel sub-benchmarks: LongFact-Concepts (targeting concept explanations) and LongFact-Objects (targeting object/event/person descriptions), each spanning 38 manually defined topics derived from the MMLU categories and popular open-domain subjects. The construction pipeline consists of:

• Prompt Generation: Using GPT-4 (temperature 1.0, max length 128), with a sliding window of 10 in-context exemplars, to generate 60 candidate prompts per topic. For LongFact-Concepts, prompts focus on niche concepts (e.g., “Explain the concept of viral quasispecies and how it influences disease progression”); for LongFact-Objects, object-centric fact-seeking questions (e.g., “What happened during the Jonestown Massacre in 1978?”) are generated.
• Filtering: Manual deduplication and selection, yielding 30 prompts per topic after removing near-duplicates and ensuring prompts require answers rich in diverse factual content (dates, names, numbers, events).
• Scale: 1,140 prompts each for LongFact-Concepts and LongFact-Objects (total 2,280).
• Release: The full prompt set, with integrated canary strings to prevent data leakage, is publicly available.

This benchmark directly addresses the need for large-scale, topic-diverse evaluation of LLM long-form factuality beyond short/closed-form QA (Wei et al., 2024).

2. SAFE: Automated Search-Augmented Factuality Evaluation

The Search-Augmented Factuality Evaluator (SAFE) is a fully automated LLM agent system designed to provide fine-grained, scalable factuality assessments of long-form responses. SAFE operates via a four-stage pipeline:

1. Sentence-to-Atomic Fact Decomposition: Responses are split into sentences using NLTK, then each is further decomposed by GPT-3.5-Turbo (T=0, with chain-of-thought exemplars) into atomic facts (short, independent factual units).
2. Self-Containment Revision: Each atomic fact is rewritten for standalone clarity, removing referential ambiguity (e.g., “the company” → “Quoc Le teaches courses on deep learning”).
3. Relevance Filtering: Facts irrelevant to the prompt, as determined by the LLM (“Foo” Test), are discarded.
4. Automated Fact Verification: For each relevant fact, up to five Google Search queries are executed via the Serper API. Retrieved snippets are synthesized by the LLM in a multi-step reasoning chain, and each claim is labeled as [Supported] or [Not Supported].

SAFE yields for each response counts of supported (S), irrelevant (I), and unsupported (N) facts. It robustly outperforms crowdsourced annotators in both agreement and cost-efficiency (72% label agreement on FActScore, SAFE judged correct in 76% of sampled human/SAFE disagreements, and operates at $0.19/response vs.$4.00 for human labeling) (Wei et al., 2024).

3. F₁@K and Factuality Metrics for Long-Form Assessment

Traditional factuality metrics are insufficient for long-form, multi-claim outputs. LongFact introduces a principled aggregation metric:

• Precision: $\Pr(y) = \frac{S(y)}{S(y) + N(y)}$.
• Recall@K: $R_K(y) = \min\left(\frac{S(y)}{K},\,1\right)$, with $K$ as a hyperparameter for user-preferred fact count.
• F₁@K: Combines these as

$$F_{1}@K(y) = \begin{cases} \frac{2\,\Pr(y)\,R_{K}(y)}{\Pr(y) + R_{K}(y)}, & S(y)>0; \ 0 & S(y) = 0 \end{cases}$$

F₁@K generalizes factual assessment in long-form by balancing factual precision and coverage, regulated by user-determined information need ($K$). As $K$ increases, recall dominates; as $K=1$, precision only.

SAFE’s judgments, aggregated by F₁@K, enable robust cross-model and cross-topic comparison without the limitations of crowd or single-metric evaluations (Wei et al., 2024).

4. Benchmarking Results: Model Comparison and Scaling

Extensive benchmarking on LongFact-Objects (250 prompts) includes 13 models from Gemini, GPT, Claude, and PaLM-2 families:

• Leaderboard: GPT-4-Turbo achieves F₁@64 ≈ 95.0, F₁@178 ≈ 66.4; Gemini-Ultra and PaLM-2-L-IT-RLHF follow closely.
• Family Trends: Later/larger sub-releases within each family uniformly improve F₁@K, highlighting the scalability effect on long-form factuality.
• Parameter Tuning: Temperature is fixed at 0, response length is capped at 1024 tokens, and fixed model snapshots minimize temporal drift.

Empirically, iterative model improvements (particularly through scaling and RLHF) produce consistent factuality gains, with model size and training paradigm being primary predictors of performance (Wei et al., 2024).

5. Reproducibility, Implementation, and Community Impact

LongFact is fully reproducible, with all assets (prompts, SAFE pipeline code, experimental scripts) released open-source. Random seeds, exact prompt splits, and all prompt templates are published to ensure benchmark integrity across time and platform. Standard API endpoints (OpenAI, Google, Anthropic, Serper) are specified for consistent model evaluation, with key settings (temperature, batch sizes) controlled.

This benchmark suite forms a foundation for the systematic, community-driven study and improvement of long-form factuality in LLMs. It enables benchmarking advances, comparison of tuning/training methods, and tracking of the field’s progress in open-domain, high-fidelity factual QA (Wei et al., 2024).

6. Limitations and Future Directions

Limitations of LongFact and SAFE include dependence on external search APIs (introducing temporal drift and access variation), granularity constraints in atomic fact extraction, and the necessity of fixed model snapshots to maintain comparability over time. Furthermore, the F₁@K metric, while balanced, relies on the accurate decomposition and verification of individual facts, which itself depends on frontier LLM capabilities.

A plausible implication is that continued advances in automatic fact extraction, robust retrieval over diverse knowledge sources, and consensus-based aggregation metrics will further refine long-form factuality evaluation. Future community initiatives may expand LongFact coverage, diversify language and domain scope, and integrate more nuanced user-centric fact importance calibrations. The LongFact approach remains a reference standard for ongoing research in LLM factual reliability.