FastFact: Evaluating Long-Form Factuality

• FastFact is a long-form factuality evaluation framework that uses chunk-level extraction combined with confidence-based pre-verification to reduce inference calls.
• It improves evidence quality by retrieving full-document content from crawled webpages instead of limited search snippets.
• The framework aggregates claim judgments with a revised F1@K' metric, closely matching human evaluation on long-form responses.

FastFact, introduced as FaStFACT, is a long-form factuality evaluation framework for LLM outputs. It addresses a recurrent weakness of earlier factuality pipelines: they typically decompose responses into claims, search for evidence, and verify claims, but they do so with sentence-level extraction, many expensive inference calls, and evidence limited to short search snippets. FaStFACT instead combines chunk-level claim extraction, confidence-based pre-verification, document-level evidence collection from crawled webpages, and retrieval-augmented verification, while aggregating claim judgments with a revised response-level metric $F_1@K'$ intended to track human evaluation more closely (Wan et al., 13 Oct 2025).

1. Task definition and design rationale

FaStFACT is designed for the evaluation of multi-sentence or multi-paragraph responses in which a single answer may mix supported, false, irrelevant, ambiguous, redundant, and unverifiable material. The paper’s starting point is the now-standard decomposition paradigm: break a response into atomic factual claims, retrieve evidence, verify each claim, and aggregate the results. The paper argues that this paradigm is sound, but that previous implementations are bottlenecked by two problems: inefficiency and ineffectiveness (Wan et al., 13 Oct 2025).

The inefficiency claim is concrete. Sentence-level systems require approximately one extraction call per sentence, so a response with $N$ sentences requires $N$ extraction calls before verification even begins. Some pipelines then add post-hoc decontextualization, revision, and relevance-checking passes, further increasing the number of LLM calls. SAFE is singled out as especially expensive because it uses sentence-level extraction, revision, relevance checking, and iterative query generation with chain-of-thought prompting. This makes long responses costly to score at scale (Wan et al., 13 Oct 2025).

The ineffectiveness claim concerns the quality of the extracted claim set and the quality of the evidence. Sentence-level extraction can miss claims that depend on multiple neighboring sentences, can over-decompose a single idea into overlapping claims, and can create unverifiable or redundant claims. At the evidence stage, prior open-web systems often verify against search snippets that average about 23.75 words, which the paper treats as insufficient for many detailed checks. The result is a systematic tendency toward incomplete or weakly grounded verification (Wan et al., 13 Oct 2025).

These design criticisms place FaStFACT within a broader fact-checking tradition that treats factuality as a sequence of structured subproblems rather than as a single scalar judgment. Earlier end-to-end systems such as FAKTA already organized claim verification into retrieval, stance detection, evidence extraction, and aggregation (Nadeem et al., 2019). FaStFACT keeps the decomposition principle but changes the granularity and evidence model.

2. Chunk-level extraction and confidence-based pre-verification

FaStFACT’s first technical departure is chunk-level claim extraction. Instead of extracting claims sentence by sentence, it groups the generated response into chunks of $w$ sentences and prompts an extractor to produce verifiable atomic claims directly. The chunk stride $w$ is a tunable parameter, with $w=1$ corresponding to ordinary sentence-level extraction and larger $w$ providing more discourse context (Wan et al., 13 Oct 2025).

This changes both cost and claim quality. For a response with $N$ sentences, the extractor-call count drops from approximately $N$ to $N/w$. More importantly, larger chunks give the extractor enough context to resolve cross-sentence references, reduce over-decomposition, and avoid producing multiple claims for the same proposition. The extractor prompt is written to output claims that are self-contained, specific, and independently meaningful, while excluding stories, hypotheticals, opinions, suggestions, and instructions. The paper treats this as a direct replacement for separate decontextualization and relevance-filtering sub-pipelines (Wan et al., 13 Oct 2025).

FaStFACT couples extraction with a pre-verification step. For each extracted claim, the model emits one of six labels:

• IRRELEVANT
• SUPPORTED
• NON-SUPPORTED
• LIKELY SUPPORTED
• LIKELY NON-SUPPORTED
• UNSURE

Only claims with a definite label—SUPPORTED, NON-SUPPORTED, or IRRELEVANT—and sufficiently high confidence are accepted without external evidence. The confidence score is defined as

$N$0

where $N$1 is the claim and $N$2 is the emitted label token (Wan et al., 13 Oct 2025).

This confidence gate is central to the framework’s efficiency. If the model is confident enough, the pipeline accepts the pre-verification result and skips search. If not, the claim is passed to the retrieval stage. The paper denotes by $N$3 the fraction of extracted claims that fail this threshold, so downstream search and evidence-based verification operate only on $N$4 claims rather than all $N$5 claims (Wan et al., 13 Oct 2025).

3. Evidence collection, retrieval, and verification

For claims that require external checking, FaStFACT uses the claim itself as a web query, retrieves result URLs, and then accesses each linked webpage through the Jina Reader API to obtain full document content rather than relying on short search snippets. The paper reports an average of 7054.29 words per piece for scraped webpages, compared with 23.75 words for the snippets used in earlier pipelines (Wan et al., 13 Oct 2025).

The framework does not pass entire webpages directly to the verifier. Instead, it stores the crawled pages as a temporary document-level evidence base, splits them into chunks that preserve sentence integrity, and retrieves the most relevant pieces with a BM25 retriever. The verifier then receives the claim plus the retrieved evidence chunks and assigns one of five labels:

• supported
• refuted
• conflicting evidence
• not enough evidence
• unverifiable

The label semantics are explicit. A claim is supported if all of its parts are supported and none are refuted; refuted if some part is refuted and no evidence supports that same part; conflicting evidence if different evidence pieces both support and refute the same relevant part; not enough evidence if the evidence is insufficient to decide; and unverifiable if the claim itself is ambiguous, opinion-based, or not a concrete fact (Wan et al., 13 Oct 2025).

A plausible comparison is with FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval, which addresses a different long-context failure mode. That work argues that LLMs often lose track of multiple target facts during generation and proposes an iterative retrieve–rewrite–retrieve loop to recover missed facts (Wang et al., 2024). FaStFACT instead treats evidence insufficiency and claim-set quality as the main bottlenecks, and addresses them through chunked extraction and document-level retrieval rather than through context rewriting (Wan et al., 13 Oct 2025).

4. Scoring, benchmark construction, and empirical results

FaStFACT aggregates claim-level judgments into a response-level factuality score. The starting precision term is inherited from FActScore:

$N$6

where $N$7 is the number of supported facts in response $N$8 and $N$9 is the number of non-supported facts (Wan et al., 13 Oct 2025).

The paper then critiques SAFE’s $N$0 metric for having a verbosity blindspot and for depending on an arbitrary target $N$1. FaStFACT therefore defines a response-specific target $N$2, where $N$3 is the human-annotated number of valid verifiable claims for that response, and computes a revised $N$4 that penalizes deviations between the number of supported claims and that annotated target. The stated purpose is to penalize both under-informative and over-verbose outputs (Wan et al., 13 Oct 2025).

The evaluation uses FaStFact-Bench, an aggregated and manually annotated benchmark constructed from five long-form factuality resources: FActScore-Bio, Factcheck-Bench, ExpertQA, LongFact, and HelloBench. The benchmark contains 100 questions total, with 20 sampled from each source, and responses from four models—DeepSeek-R1, DeepSeek-V3, GPT-4o, and Qwen-2.5-Instruct—yielding 400 long-form QA pairs. Average response length is about 465 words / 30.8 sentences. Ten annotators corrected claim extraction outputs, added missing claims, removed redundant claims, and manually verified claims using evidence. The reported annotation effort exceeded 500 hours, with inter-rater agreement of 93.6% for claim-extraction operations and 95.0% for verification labels (Wan et al., 13 Oct 2025).

The principal empirical result is that FaStFACT is closest to human judgment among the compared automatic pipelines. The benchmark’s human average is $N$5, while FaStFACT produces $N$6, for an absolute variance of 0.012. It also attains 85.3% exact agreement on claim verification and 92.9% agreement on coarse scoring labels (Wan et al., 13 Oct 2025).

Method	$N$7	$N$8	$N$9	Token cost
FaStFact	3.35	0.012	0.780	5615
ExpertQA	21.26	0.127	0.919	7893
FacTool	9.09	0.195	0.987	4480
VeriScore	7.32	0.107	0.899	22848
SAFE	14.28	0.168	0.960	49622

The same evaluation also supports the paper’s efficiency claim. In token cost, FaStFACT is much cheaper than VeriScore and SAFE, while staying much closer to the human benchmark score. The paper summarizes its inference complexity as $w$0, with search and verification scaling as $w$1, where $w$2 is the number of sentences, $w$3 the chunk stride, $w$4 the number of extracted claims, $w$5 the uncertain-claim fraction, and $w$6 the number of search results per claim (Wan et al., 13 Oct 2025).

5. Position within the broader factuality and fact-checking landscape

A useful way to situate FastFact is as one instance of a broader movement toward structured factuality pipelines with explicit evidence artifacts, selective retrieval, and operational constraints.

System or resource	Core artifact	Operational emphasis
FaStFact	Chunk-level claims, pre-verification, crawled document evidence	Long-form LLM factuality evaluation (Wan et al., 13 Oct 2025)
FACT	Iterative retrieve–rewrite–retrieve loop	Multi-fact retrieval completeness in long contexts (Wang et al., 2024)
FACTS	Offline SQL queries and Jinja2 templates	Reusable, privacy-compliant table summarization (Yuan et al., 15 Oct 2025)
FAKTA	Retrieval, stance, evidence extraction, aggregation	End-to-end automatic claim verification (Nadeem et al., 2019)
aedFaCT	Keyword-guided expert opinions and paper retrieval	Scientific fact-checking assistance (Altuncu et al., 2023)
“Straight to the Facts”	Joint embedding retrieval of KB triples	Factual visual question answering (Narasimhan et al., 2018)
FACTors	Ecosystem-level claim/report dataset with Lucene index	Cross-organization overlap and credibility analysis (Altuncu et al., 14 May 2025)
Fin-Fact	Finance-specific multimodal benchmark with justifications	Financial fact checking and explanation generation (Rangapur et al., 2023)

The commonality across these systems is not a single model family, but a shared emphasis on explicit intermediate objects. In FaStFACT those objects are atomic claims, confidence labels, crawled documents, retrieved evidence chunks, and per-claim verdicts (Wan et al., 13 Oct 2025). In FACTS they are schema-aware SQL queries and Jinja2 templates that can be reused across tables sharing a schema (Yuan et al., 15 Oct 2025). In FAKTA they are retrieved documents, stance labels, sentence-level rationales, and source-grouped evidence views (Nadeem et al., 2019). In aedFaCT they are extracted keywords, expert-quote paragraphs, publication records, and ranked researchers (Altuncu et al., 2023).

The broader literature also shows that factuality systems vary in their unit of retrieval. FaStFACT retrieves evidence for claims from the open web at document scale (Wan et al., 13 Oct 2025). “Straight to the Facts” retrieves RDF-style fact triples $w$7 from a knowledge base for image-question pairs (Narasimhan et al., 2018). FACT targets multiple facts within long textual context through iterative context rewriting (Wang et al., 2024). FACTors, by contrast, is not a verifier but a retrieval-ready fact-checking corpus containing 118,112 fact-checks, 117,993 unique reports, 113,762 unique claims, and 7,327 fact-checks with overlapping claims, enabling claim matching and cross-source aggregation (Altuncu et al., 14 May 2025).

Datasets and benchmarks play a similarly structuring role. Fin-Fact provides 3,369 finance-domain claims with True, False, and NEI labels, together with professional justifications and images, and shows that multimodal inputs do not uniformly help current models (Rangapur et al., 2023). FACTors expands the scope from single benchmarks to the fact-checking ecosystem itself, covering 39 fact-checking organizations, 1,953 authors, and the period 1995–2025 (Altuncu et al., 14 May 2025). FastFact’s own benchmark contribution, FaStFact-Bench, occupies a complementary niche: it evaluates not claim verifiers directly, but factuality evaluators of long-form generations (Wan et al., 13 Oct 2025).

6. Limitations and plausible research directions

FaStFACT’s scope is explicit. It depends on the quality and availability of web evidence, so performance can degrade when relevant information is sparse, paywalled, inaccessible, or buried in low-quality sources (Wan et al., 13 Oct 2025). Its reported experiments focus on the included long-form benchmarks and are therefore primarily English and benchmark-specific. Its pre-verification stage also depends on model calibration: confidence filtering is meant to reduce overconfident internal judgments, but this still leaves the framework sensitive to the extractor’s parametric knowledge and token-level confidence estimates (Wan et al., 13 Oct 2025).

Several nearby lines of work suggest natural extensions. FACTS shows that factual summarization can be improved by compiling reusable executable artifacts offline while sending only table schemas to LLMs, which suggests a stronger privacy-preserving path for structured-domain factuality evaluation (Yuan et al., 15 Oct 2025). aedFaCT shows the value of expert-opinion discovery and peer-reviewed literature retrieval for human-centered fact-checking, suggesting that evaluator pipelines could incorporate expert-source priors rather than treating all web evidence equally (Altuncu et al., 2023). FACTors shows that overlapping claims, organization metadata, and credibility-style scores can be maintained at ecosystem scale, which suggests claim-level evaluation systems could move from single-source evidence to cross-organization aggregation and disagreement-aware scoring (Altuncu et al., 14 May 2025).

The multimodal setting remains an open challenge. Fin-Fact reports that text-only GPT-4 outperformed the multimodal baselines it evaluated, and that multimodal inputs changed model behavior significantly without robustly positive gains, which suggests that extending FastFact-style evaluation beyond text will require more than simply attaching image inputs to a verifier (Rangapur et al., 2023). At the same time, FACT on multi-fact retrieval argues that long-context systems can lose track of multiple target facts during generation, implying that document-level evidence sufficiency and multi-fact completeness are separate axes of robustness that could be combined in future evaluators (Wang et al., 2024).

In that broader context, FastFact is best understood not as a generic fact-checker, but as a specialized automatic evaluator of long-form factuality. Its distinctive contribution is to make the decomposition–retrieval–verification pattern operationally viable for long outputs by reducing extraction calls, pruning easy claims before search, and replacing snippet-level evidence with full-document retrieval (Wan et al., 13 Oct 2025).