---
title: 'FastFact: Evaluating Long-Form Factuality'
url: https://www.emergentmind.com/topics/fastfact
type: topic
---

# FastFact: Evaluating Long-Form Factuality

FastFact, introduced as **FaStFACT**, is a long-form factuality evaluation framework for Large Language Model outputs. It addresses a recurrent weakness of earlier factuality pipelines: they typically decompose responses into claims, search for evidence, and verify claims, but they do so with sentence-level extraction, many expensive inference calls, and evidence limited to short search snippets. FaStFACT instead combines **chunk-level claim extraction**, **confidence-based pre-verification**, **document-level evidence collection from crawled webpages**, and **retrieval-augmented verification**, while aggregating claim judgments with a revised response-level metric \(F_1@K'\) intended to track human evaluation more closely [2510.12839].

## 1. Task definition and design rationale

FaStFACT is designed for the evaluation of **multi-sentence or multi-paragraph responses** in which a single answer may mix supported, false, irrelevant, ambiguous, redundant, and unverifiable material. The paper’s starting point is the now-standard decomposition paradigm: break a response into atomic factual claims, retrieve evidence, verify each claim, and aggregate the results. The paper argues that this paradigm is sound, but that previous implementations are bottlenecked by two problems: **inefficiency** and **ineffectiveness** [2510.12839].

The inefficiency claim is concrete. Sentence-level systems require approximately one extraction call per sentence, so a response with \(N\) sentences requires \(N\) extraction calls before verification even begins. Some pipelines then add post-hoc decontextualization, revision, and relevance-checking passes, further increasing the number of LLM calls. SAFE is singled out as especially expensive because it uses sentence-level extraction, revision, relevance checking, and iterative query generation with chain-of-thought prompting. This makes long responses costly to score at scale [2510.12839].

The ineffectiveness claim concerns the **quality of the extracted claim set** and the **quality of the evidence**. Sentence-level extraction can miss claims that depend on multiple neighboring sentences, can over-decompose a single idea into overlapping claims, and can create unverifiable or redundant claims. At the evidence stage, prior open-web systems often verify against search snippets that average about **23.75 words**, which the paper treats as insufficient for many detailed checks. The result is a systematic tendency toward incomplete or weakly grounded verification [2510.12839].

These design criticisms place FaStFACT within a broader fact-checking tradition that treats factuality as a sequence of structured subproblems rather than as a single scalar judgment. Earlier end-to-end systems such as FAKTA already organized claim verification into retrieval, stance detection, evidence extraction, and aggregation [1906.04164]. FaStFACT keeps the decomposition principle but changes the granularity and evidence model.

## 2. Chunk-level extraction and confidence-based pre-verification

FaStFACT’s first technical departure is **chunk-level claim extraction**. Instead of extracting claims sentence by sentence, it groups the generated response into chunks of \(w\) sentences and prompts an extractor to produce **verifiable atomic claims** directly. The chunk stride \(w\) is a tunable parameter, with \(w=1\) corresponding to ordinary sentence-level extraction and larger \(w\) providing more discourse context [2510.12839].

This changes both cost and claim quality. For a response with \(N\) sentences, the extractor-call count drops from approximately \(N\) to \(N/w\). More importantly, larger chunks give the extractor enough context to resolve cross-sentence references, reduce over-decomposition, and avoid producing multiple claims for the same proposition. The extractor prompt is written to output claims that are self-contained, specific, and independently meaningful, while excluding stories, hypotheticals, opinions, suggestions, and instructions. The paper treats this as a direct replacement for separate decontextualization and relevance-filtering sub-pipelines [2510.12839].

FaStFACT couples extraction with a **pre-verification** step. For each extracted claim, the model emits one of six labels:

- `IRRELEVANT`
- `SUPPORTED`
- `NON-SUPPORTED`
- `LIKELY SUPPORTED`
- `LIKELY NON-SUPPORTED`
- `UNSURE`

Only claims with a **definite** label—`SUPPORTED`, `NON-SUPPORTED`, or `IRRELEVANT`—and sufficiently high confidence are accepted without external evidence. The confidence score is defined as

\[
C = \exp(\text{target\_logprob}) = P_\theta(y \mid x),
\]

where \(x\) is the claim and \(y\) is the emitted label token [2510.12839].

This confidence gate is central to the framework’s efficiency. If the model is confident enough, the pipeline accepts the pre-verification result and skips search. If not, the claim is passed to the retrieval stage. The paper denotes by \(p\) the fraction of extracted claims that fail this threshold, so downstream search and evidence-based verification operate only on \(pM\) claims rather than all \(M\) claims [2510.12839].

## 3. Evidence collection, retrieval, and verification

For claims that require external checking, FaStFACT uses the claim itself as a web query, retrieves result URLs, and then accesses each linked webpage through the **Jina Reader API** to obtain **full document content** rather than relying on short search snippets. The paper reports an average of **7054.29 words per piece** for scraped webpages, compared with **23.75 words** for the snippets used in earlier pipelines [2510.12839].

The framework does not pass entire webpages directly to the verifier. Instead, it stores the crawled pages as a temporary document-level evidence base, splits them into chunks that preserve sentence integrity, and retrieves the most relevant pieces with a **BM25 retriever**. The verifier then receives the claim plus the retrieved evidence chunks and assigns one of five labels:

- `supported`
- `refuted`
- `conflicting evidence`
- `not enough evidence`
- `unverifiable`

The label semantics are explicit. A claim is `supported` if all of its parts are supported and none are refuted; `refuted` if some part is refuted and no evidence supports that same part; `conflicting evidence` if different evidence pieces both support and refute the same relevant part; `not enough evidence` if the evidence is insufficient to decide; and `unverifiable` if the claim itself is ambiguous, opinion-based, or not a concrete fact [2510.12839].

A plausible comparison is with **FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval**, which addresses a different long-context failure mode. That work argues that LLMs often lose track of multiple target facts during generation and proposes an iterative retrieve–rewrite–retrieve loop to recover missed facts [2410.21012]. FaStFACT instead treats **evidence insufficiency** and **claim-set quality** as the main bottlenecks, and addresses them through chunked extraction and document-level retrieval rather than through context rewriting [2510.12839].

## 4. Scoring, benchmark construction, and empirical results

FaStFACT aggregates claim-level judgments into a response-level factuality score. The starting precision term is inherited from FActScore:

\[
P(y) = \frac{S(y)}{S(y) + N(y)},
\]

where \(S(y)\) is the number of supported facts in response \(y\) and \(N(y)\) is the number of non-supported facts [2510.12839].

The paper then critiques SAFE’s \(F_1@K\) metric for having a verbosity blindspot and for depending on an arbitrary target \(K\). FaStFACT therefore defines a response-specific target \(K' = S'(y)+N'(y)\), where \(K'\) is the human-annotated number of valid verifiable claims for that response, and computes a revised \(F_1@K'\) that penalizes deviations between the number of supported claims and that annotated target. The stated purpose is to penalize both under-informative and over-verbose outputs [2510.12839].

The evaluation uses **FaStFact-Bench**, an aggregated and manually annotated benchmark constructed from five long-form factuality resources: **FActScore-Bio**, **Factcheck-Bench**, **ExpertQA**, **LongFact**, and **HelloBench**. The benchmark contains **100 questions total**, with **20** sampled from each source, and responses from **four models**—**DeepSeek-R1**, **DeepSeek-V3**, **GPT-4o**, and **Qwen-2.5-Instruct**—yielding **400 long-form QA pairs**. Average response length is about **465 words / 30.8 sentences**. Ten annotators corrected claim extraction outputs, added missing claims, removed redundant claims, and manually verified claims using evidence. The reported annotation effort exceeded **500 hours**, with inter-rater agreement of **93.6%** for claim-extraction operations and **95.0%** for verification labels [2510.12839].

The principal empirical result is that FaStFACT is closest to human judgment among the compared automatic pipelines. The benchmark’s human average is \(F_1@K' = 0.792\), while FaStFACT produces \(F_1@K' = 0.780\), for an absolute variance of **0.012**. It also attains **85.3%** exact agreement on claim verification and **92.9%** agreement on coarse scoring labels [2510.12839].

| Method | \(|\Delta K'|\) | \(|\Delta F_1@K'|\) | \(F_1@K'\) | Token cost |
|---|---:|---:|---:|---:|
| FaStFact | 3.35 | 0.012 | 0.780 | 5615 |
| ExpertQA | 21.26 | 0.127 | 0.919 | 7893 |
| FacTool | 9.09 | 0.195 | 0.987 | 4480 |
| VeriScore | 7.32 | 0.107 | 0.899 | 22848 |
| SAFE | 14.28 | 0.168 | 0.960 | 49622 |

The same evaluation also supports the paper’s efficiency claim. In token cost, FaStFACT is much cheaper than **VeriScore** and **SAFE**, while staying much closer to the human benchmark score. The paper summarizes its inference complexity as \(O(N/w + pM)\), with search and verification scaling as \(O(pkM + pM)\), where \(N\) is the number of sentences, \(w\) the chunk stride, \(M\) the number of extracted claims, \(p\) the uncertain-claim fraction, and \(k\) the number of search results per claim [2510.12839].

## 5. Position within the broader factuality and fact-checking landscape

A useful way to situate FastFact is as one instance of a broader movement toward **structured factuality pipelines** with explicit evidence artifacts, selective retrieval, and operational constraints.

| System or resource | Core artifact | Operational emphasis |
|---|---|---|
| FaStFact | Chunk-level claims, pre-verification, crawled document evidence | Long-form LLM factuality evaluation [2510.12839] |
| FACT | Iterative retrieve–rewrite–retrieve loop | Multi-fact retrieval completeness in long contexts [2410.21012] |
| FACTS | Offline SQL queries and Jinja2 templates | Reusable, privacy-compliant table summarization [2510.13920] |
| FAKTA | Retrieval, stance, evidence extraction, aggregation | End-to-end automatic claim verification [1906.04164] |
| aedFaCT | Keyword-guided expert opinions and paper retrieval | Scientific fact-checking assistance [2305.07796] |
| “Straight to the Facts” | Joint embedding retrieval of KB triples | Factual visual question answering [1809.01124] |
| FACTors | Ecosystem-level claim/report dataset with Lucene index | Cross-organization overlap and credibility analysis [2505.09414] |
| Fin-Fact | Finance-specific multimodal benchmark with justifications | Financial fact checking and explanation generation [2309.08793] |

The commonality across these systems is not a single model family, but a shared emphasis on **explicit intermediate objects**. In FaStFACT those objects are atomic claims, confidence labels, crawled documents, retrieved evidence chunks, and per-claim verdicts [2510.12839]. In FACTS they are schema-aware SQL queries and Jinja2 templates that can be reused across tables sharing a schema [2510.13920]. In FAKTA they are retrieved documents, stance labels, sentence-level rationales, and source-grouped evidence views [1906.04164]. In aedFaCT they are extracted keywords, expert-quote paragraphs, publication records, and ranked researchers [2305.07796].

The broader literature also shows that factuality systems vary in their **unit of retrieval**. FaStFACT retrieves evidence for claims from the open web at document scale [2510.12839]. “Straight to the Facts” retrieves RDF-style fact triples \((a,r,b)\) from a knowledge base for image-question pairs [1809.01124]. FACT targets multiple facts within long textual context through iterative context rewriting [2410.21012]. FACTors, by contrast, is not a verifier but a retrieval-ready fact-checking corpus containing **118,112 fact-checks**, **117,993 unique reports**, **113,762 unique claims**, and **7,327 fact-checks with overlapping claims**, enabling claim matching and cross-source aggregation [2505.09414].

Datasets and benchmarks play a similarly structuring role. Fin-Fact provides **3,369** finance-domain claims with **True**, **False**, and **NEI** labels, together with professional justifications and images, and shows that multimodal inputs do not uniformly help current models [2309.08793]. FACTors expands the scope from single benchmarks to the fact-checking ecosystem itself, covering **39 fact-checking organizations**, **1,953 authors**, and the period **1995–2025** [2505.09414]. FastFact’s own benchmark contribution, FaStFact-Bench, occupies a complementary niche: it evaluates not claim verifiers directly, but **factuality evaluators of long-form generations** [2510.12839].

## 6. Limitations and plausible research directions

FaStFACT’s scope is explicit. It depends on the quality and availability of web evidence, so performance can degrade when relevant information is sparse, paywalled, inaccessible, or buried in low-quality sources [2510.12839]. Its reported experiments focus on the included long-form benchmarks and are therefore primarily English and benchmark-specific. Its pre-verification stage also depends on model calibration: confidence filtering is meant to reduce overconfident internal judgments, but this still leaves the framework sensitive to the extractor’s parametric knowledge and token-level confidence estimates [2510.12839].

Several nearby lines of work suggest natural extensions. **FACTS** shows that factual summarization can be improved by compiling reusable executable artifacts offline while sending only table schemas to LLMs, which suggests a stronger privacy-preserving path for structured-domain factuality evaluation [2510.13920]. **aedFaCT** shows the value of expert-opinion discovery and peer-reviewed literature retrieval for human-centered fact-checking, suggesting that evaluator pipelines could incorporate expert-source priors rather than treating all web evidence equally [2305.07796]. **FACTors** shows that overlapping claims, organization metadata, and credibility-style scores can be maintained at ecosystem scale, which suggests claim-level evaluation systems could move from single-source evidence to cross-organization aggregation and disagreement-aware scoring [2505.09414].

The multimodal setting remains an open challenge. **Fin-Fact** reports that text-only GPT-4 outperformed the multimodal baselines it evaluated, and that multimodal inputs changed model behavior significantly without robustly positive gains, which suggests that extending FastFact-style evaluation beyond text will require more than simply attaching image inputs to a verifier [2309.08793]. At the same time, **FACT** on multi-fact retrieval argues that long-context systems can lose track of multiple target facts during generation, implying that document-level evidence sufficiency and multi-fact completeness are separate axes of robustness that could be combined in future evaluators [2410.21012].

In that broader context, FastFact is best understood not as a generic fact-checker, but as a specialized **automatic evaluator of long-form factuality**. Its distinctive contribution is to make the decomposition–retrieval–verification pattern operationally viable for long outputs by reducing extraction calls, pruning easy claims before search, and replacing snippet-level evidence with full-document retrieval [2510.12839].

Source: https://www.emergentmind.com/topics/fastfact