FActScore-Bio: Evaluating Biographical Factuality

• FActScore-Bio is a method for breaking down biographical texts into atomic facts that can be independently verified against Wikipedia.
• It combines detailed human annotation and an automated estimator that achieves error rates below 2% with high human-model agreement.
• The approach fuses retrieval-based and nonparametric techniques to enable scalable benchmarking and targeted improvements in factual text generation.

FActScore-Bio is a fine-grained, human-and-model-in-the-loop methodology for evaluating the factual precision of long-form biographical text generated by large LMs. It decomposes model outputs into "atomic facts," evaluates the factual support of each proposition using English Wikipedia as the sole knowledge source, and aggregates these judgments into a quantitative score. FActScore-Bio provides both high-fidelity human annotation protocols and a cost-efficient automated estimator with error rates below 2%. The approach enables nuanced analysis of LMs' factual accuracy and supports scalable benchmarking and improvement of text generation systems (Min et al., 2023).

1. Atomic Fact Decomposition

An "atomic fact" within FActScore-Bio is defined as a minimal, self-contained proposition from a generated text that (a) can be judged true or false against a knowledge source (English Wikipedia), and (b) has no proper sub-statement which is itself verifiable. The decomposition pipeline is structured as follows:

1. Sentence Splitting: The biography is segmented into individual sentences.
2. Model-Assisted Fact Generation: Each sentence is sent to InstructGPT (text-davinci-003) with a prompt to "break down the following sentence into independent facts."
3. Expert Revision: Human annotators refine model output, merging fragments and splitting compound facts as necessary.
4. Labeling: The finalized atomic facts per sentence are provided to fact-checking annotators.

An example: the sentence "He made his acting debut in the film The Moon Is the Sun’s Dream (1992) and continued to appear in small and supporting roles throughout the 1990s," may produce atomic facts such as (1) "He made his acting debut in the film," (2) "He made his acting debut in The Moon Is the Sun’s Dream," (3) "The Moon Is the Sun’s Dream is a film," (4) "The Moon Is the Sun’s Dream was released in 1992."

2. Formal Definition of FActScore

Given a generated biography $G$ and the annotated set of atomic facts $A(G) = \{ f_1, \ldots, f_n \}$, each labeled Supported, Not-supported, or Irrelevant, let

• $$S = | \{ f \in A(G) : \mathrm{label}(f) = \mathrm{Supported} \} |$$
• $N = | \{ f \in A(G) : \mathrm{label}(f) = \mathrm{Not\mbox{-}supported} \} |$

Irrelevant facts are ignored for precision purposes. The FActScore for $G$ is defined as:

$\mathrm{FActScore}(G) = \frac{S}{S + N}$

At the dataset or system level, this fraction is averaged across all prompts or generations.

3. Human Annotation: Protocol and Agreement

Annotation is conducted by freelance "fact-checking experts" recruited via Upwork and compensated at \$15–\$25/hour. Qualification requires a two-hour session, including demonstration videos and feedback rounds.

Each human intelligence task (HIT) comprises three biographies (one each from InstructGPT, ChatGPT, PerplexityAI) for the same prompt. Annotators:

• Inspect the generation.
• Review each atomic fact alongside a Wikipedia page link.
• Assign Supported / Not-supported / Irrelevant.

The Irrelevant label applies to off-topic or extending facts. 10% of HITs are double-annotated blindly, yielding agreement rates of 96% (InstructGPT), 90% (ChatGPT), and 88% (PerplexityAI) on Supported vs. Not-supported vs. Irrelevant, with an overall Supported/Not-supported agreement near 91%. The sole oracle is English Wikipedia (snapshot April 1, 2023). Annotators must mark "Not-supported" for contradictory or unsupported claims. Disagreement categories include inference, subjectivity, and Wikipedia inconsistencies.

4. Automated Estimator: Architecture and Settings

The automated FActScore estimator aims to replicate human judgments with minimal annotation cost. The best-performing variant, Retrieve + NP, operates as follows:

4.1 Passage Retrieval

• Index: English Wikipedia (04/01/2023), split into passages ≤ 256 tokens.
• Retriever: GTR-Large (dense) retrieves $k=5$ passages per prompt’s Wikipedia page. Retrieval choice demonstrates minimal sensitivity between GTR-Large, BM25, and GTR-XL.

4.2 Language-Model Classifier (TF-Prompting)

• Prompt Template:

$A(G) = \{ f_1, \ldots, f_n \}$0

• Evaluation LMs: Inst-LLAMA 7B (open) or ChatGPT (commercial). Model logits for "True" and "False" are averaged across passages.

4.3 Nonparametric Masked LM (NPM)

• Method: Single-mask nonparametric MLM with BM25 retrieval.
• Threshold: Score < 0.3 classifies a fact as Not-supported.

4.4 Decision Fusion (Retrieve + NP)

• A fact is Not-supported iff both the retrieval-based LM and nonparametric LM predict Not-supported; otherwise, it is Supported.

5. Quantitative Results, Comparisons, and Correlation

The Inst-LLAMA 7B + Retrieve + NP model achieves a mean absolute error rate (ER) ≈ 1.9% versus human FActScore on people biographies. System-level Pearson correlation $ρ$ is ≈ 0.93, and segment-level $F1_{micro}$ for detecting Not-supported facts ranges from ≈ 83% (Inst-LLAMA 7B) to 87.5% (ChatGPT).

A sample of 183 Wikidata persons (across strata) and 500 prompts for ~13 LMs generated 6,500 long-form biographies. Average human FActScores:

Model	Human FActScore (avg)
GPT-4	≈ 75%
ChatGPT	≈ 58%
InstructGPT	≈ 64%
PerplexityAI(RAG)	≈ 44%
Vicuna	≈ 53%
Alpaca, LLaMA v., GPT-3.5 Turbo	45–55%

Automatic ranking and scores match human annotation within ±2%.

6. Error Analysis and Biography-Specific Failure Modes

Common sources of unsupported facts include:

• Date/place inconsistencies: Mismatches between Wikipedia sections.
• Over-inference: Asserting undated ongoing facts.
• Hallucinations: Minor, unverifiable assertions.

Estimator errors are largely due to:

• Passage coverage: 70% of automatic errors—no passage fully encapsulates the fact.
• Retriever distraction: 17% caused by irrelevant passages.
• Context dependence: 7% require context from prior statements.

Human annotation disagreements (~21%) often stem from vague or subjective facts.

7. Recommendations and Methodological Insights

Retrieval augmentation outperforms parametric-only evaluators by 10–20 F1_micro points and reduces ER by 3–5%. True/False (TF) prompting yields up to +13 F1_micro points over QA pipelines. Fusion of retrieval-based and nonparametric MLM evaluators further improves the error rate.

Automated editing ("atomic-fact repair")—rephrasing or removing unsupported facts—raises overall FActScore by 4–6 points with minimal manual intervention. A pilot on NLP paper summaries using the ACL Anthology yields ER ≈ 7.4%, indicating pipeline generalizability beyond biography generation.

By decomposing biographies into atomic facts and rigorously aggregating support via Wikipedia, FActScore-Bio delivers reliable, interpretable metrics for both benchmarking and optimizing factuality in long-form text generation (Min et al., 2023).