FActScore*: Multilingual Factual Accuracy Metric

• FActScore* is an automated metric that evaluates factual precision by decomposing long-form text into atomic claims and verifying them against reference sources.
• It employs multilingual pipelines with native, translated, and back-translated evaluations to robustly assess factual accuracy across diverse languages.
• Empirical findings reveal significant factuality gaps in low-resource languages, with improvements achieved through enhanced evidence integration.

FActScore* is an automated, fine-grained metric for evaluating factual precision in long-form text, particularly in multilingual and cross-domain settings. It extends the original FActScore framework by aggregating atomic claim decomposition and claim-level verification techniques, adapted to handle diverse languages and knowledge sources. The metric underpins large-scale measurement of hallucinated, unsupported, or entity-ambiguous content in LLM-generated outputs, and facilitates the study of factual performance across resource-rich and resource-poor linguistic contexts (Chataigner et al., 2024).

1. Formal Definition and Core Computation

FActScore* evaluates the factuality of a long-form model response $R$ by decomposing it into a set of atomic claims and verifying each claim against a reference knowledge source $C$ (such as Wikipedia). The procedure consists of the following steps:

1. Atomic Claim Decomposition: $A(R) = \{a_1, a_2, ..., a_n\}$, where each $a_i$ is a minimal, independent fact present in $R$.
2. Retrieval & Fact Checking: For each $a_i$, retrieve the most relevant passage(s) from $C$, and use an LLM-based evaluator to determine $\text{Supported}(a_i, C)$, which is $1$ if $a_i$ is judged true in context, and $C$0 otherwise.
3. (Optional) Length Penalty: For short outputs (less than threshold $C$1, default 10), a discount factor is applied: $C$2; otherwise $C$3.
4. Aggregation:

$C$4

where $C$5 is the factual precision (FActScore) for $C$6.

For multilingual FActScore*, each generated response $C$7 in language $C$8 is scored via:

• Native decomposition and fact-checking against $C$9 (Wikipedia-L)
• Translation of $A(R) = \{a_1, a_2, ..., a_n\}$0 to English and scoring against $A(R) = \{a_1, a_2, ..., a_n\}$1 (Wikipedia-EN)
• English "prompt-to-L" generation, back-translation, and scoring against $A(R) = \{a_1, a_2, ..., a_n\}$2

Final FActScore* per example is the mean over these pipelines (Chataigner et al., 2024).

2. Multilingual and Multisource Evaluation Pipelines

FActScore* systematically addresses language and knowledge source variability via parallel evaluation pipelines:

Pipeline Type	Prompt & Output	Knowledge Source	Verification Language
L$A(R) = \{a_1, a_2, ..., a_n\}$3L	Native in $A(R) = \{a_1, a_2, ..., a_n\}$4	Wikipedia-$A(R) = \{a_1, a_2, ..., a_n\}$5	$A(R) = \{a_1, a_2, ..., a_n\}$6
L$A(R) = \{a_1, a_2, ..., a_n\}$7EN	Native in $A(R) = \{a_1, a_2, ..., a_n\}$8	Wikipedia-English	English (after translation)
EN$A(R) = \{a_1, a_2, ..., a_n\}$9EN	English prompt	Wikipedia-English	English

Each pipeline applies the same core FActScore computation as above. Per-entity and per-model scores combine pipeline results by simple averaging. This multi-pipeline approach characterizes "multilingual hallucination gaps," revealing substantial factuality drops and increased variance in medium- and low-resource languages (Chataigner et al., 2024).

3. Recipe for FActScore* Calculation

A concrete computation workflow for any language $a_i$0 is:

1. Generation Step:
   • Elicit response $a_i$1 to a prompt in language $a_i$2.
2. Decomposition:
   • Extract atomic claims from $a_i$3 (natively or post-translation).
3. Fact-checking:
   • For each claim, retrieve relevant passage(s) from $a_i$4 (either Wikipedia in $a_i$5 or English).
   • Form fact-checking prompt: $a_i$6 + "Fact: $a_i$7. True or False?"
   • LLM evaluates truth of each $a_i$8 in context.
4. Aggregation:
   • Compute FActScore* for each pipeline.
   • Average results for the final metric.

The same process is applied to each prompt/model/language tuple. Worked examples in (Chataigner et al., 2024) demonstrate precise application and interpretation.

4. Cross-Linguistic Empirical Patterns

Empirical findings using FActScore* across 19 languages and multiple top-tier LLMs show:

• Factual precision declines systematically from high-resource (≈73%) to medium-resource (≈64%) and low-resource (≈57%) languages.
• Prompt and verification language matter: English-prompted outputs generally achieve higher factuality, with the advantage growing in lower-resource languages.
• The reliability of FActScore* itself degrades as resource-level drops, evidenced by rising variance in per-entity scores.
• Qwen-72B consistently achieves the highest FActScores among surveyed models; LLaMA variants perform more uniformly but at lower absolute levels.

A key insight is that neither scaling model size nor using English prompts closes factuality gaps in low-resource languages, highlighting the persistent challenge of multilingual evaluation and model robustness (Chataigner et al., 2024).

5. Knowledge Source Challenges and Mitigation

The limiting factor for FActScore* in many languages is knowledge source coverage. Research demonstrates:

• Wikipedia-L can be sparse in medium- and low-resource languages, leading to artificially low FActScores or unreliable support judgments.
• Machine translation errors in fact decomposition or evidence retrieval can propagate, but empirically result in negligible score drift when English-based verification is used (Shafayat et al., 2024, 2406.19415).
• Increasing the number of retrieved passages per claim, incorporating web/Google search results ("full Internet"), or using LLM-generated knowledge snippets significantly improves the estimation quality and accuracy of FActScore*, especially in under-resourced languages (2406.19415).

Mitigation strategies outlined in (2406.19415) include:

• Expanding retrieved passage sets ($a_i$9 up to 20)
• Using real-time web search per claim
• Context augmentation with LLM-generated Q&A pairs

Empirical results show that Internet-based evidence yields the largest boosts, increasing low-resource (e.g., Bengali) factuality accuracy from ≈53% to ≈88% (2406.19415).

6. Interpretation, Best Practices, and Limitations

FActScore* is a relative, atomic claim–based precision measure, with several critical properties:

• It provides a scalable, automated alternative to costly human evaluation for hallucination auditing.
• The metric is easy to game via claim minimization or uninformative outputs; thus, it must be interpreted alongside output length, claim counts, and qualitative inspection.
• FActScore* assumes the knowledge source is comprehensive and up-to-date, which is often violated outside English and high-profile topics.
• The methodology demands rigorous calibration, e.g., native-speaker annotation to interpret low-resource language scores and auditing of decomposition fidelity.
• No weighting is applied to claims' semantic informativeness; all atomic facts count equally, potentially obscuring high-level utility or informativeness.

Recommended reporting includes both FActScore* and the absolute number of correct atomic facts per example. The metric is best viewed as a diagnostic for model ranking and comparative analysis rather than an absolute measure of document truth (Chataigner et al., 2024, Shafayat et al., 2024). Ongoing work emphasizes further improvements to knowledge retrieval, finetuned fact extractors for non-English languages, and dynamic or domain-adaptive evidence integration.

7. Broader Implications and Ongoing Research

FActScore* enables robust, fine-grained audit of hallucinations and factuality gaps in multilingual, multi-domain NLG. Its adoption in recent benchmarking and error analysis pipelines exposes pronounced performance disparities and guides targeted mitigation. As research progresses, integrating broader evidence sources, calibrating cross-lingual fact-checkers, and quantifying claim informativeness are emerging avenues to increase both the accuracy and utility of FActScore* and its descendants. The required technical and infrastructural developments outline the broader research agenda for factuality measurement in the context of increasingly global and diverse LLM deployments (Chataigner et al., 2024, 2406.19415).