---
title: 'FActScore*: Multilingual Factual Accuracy Metric'
url: https://www.emergentmind.com/topics/factscore-41498db5-1b44-4331-b7ba-ed4d69603e54
type: topic
---

# FActScore*: Multilingual Factual Accuracy Metric

FActScore* is an automated, fine-grained metric for evaluating factual precision in long-form text, particularly in multilingual and cross-domain settings. It extends the original FActScore framework by aggregating atomic claim decomposition and claim-level verification techniques, adapted to handle diverse languages and knowledge sources. The metric underpins large-scale measurement of hallucinated, unsupported, or entity-ambiguous content in LLM-generated outputs, and facilitates the study of factual performance across resource-rich and resource-poor linguistic contexts [2410.18270].

## 1. Formal Definition and Core Computation

FActScore* evaluates the factuality of a long-form model response $R$ by decomposing it into a set of atomic claims and verifying each claim against a reference knowledge source $C$ (such as Wikipedia). The procedure consists of the following steps:

1. **Atomic Claim Decomposition**: $A(R) = \{a_1, a_2, ..., a_n\}$, where each $a_i$ is a minimal, independent fact present in $R$.
2. **Retrieval & Fact Checking**: For each $a_i$, retrieve the most relevant passage(s) from $C$, and use an LLM-based evaluator to determine $\text{Supported}(a_i, C)$, which is $1$ if $a_i$ is judged true in context, and $0$ otherwise.
3. **(Optional) Length Penalty**: For short outputs (less than threshold $\gamma$, default 10), a discount factor is applied: $p = \exp\left(\frac{1 - \gamma}{|A(R)|}\right)$; otherwise $p = 1$.
4. **Aggregation**:

\[
F(R, C) = \frac{p}{|A(R)|} \sum_{a \in A(R)} \text{Supported}(a, C)
\]
where $F(R, C)$ is the factual precision (FActScore) for $R$.

For multilingual FActScore*, each generated response $R_L$ in language $L$ is scored via:
- Native decomposition and fact-checking against $C_L$ (Wikipedia-L)
- Translation of $R_L$ to English and scoring against $C_{EN}$ (Wikipedia-EN)
- English "prompt-to-L" generation, back-translation, and scoring against $C_{EN}$

Final FActScore* per example is the mean over these pipelines [2410.18270].

## 2. Multilingual and Multisource Evaluation Pipelines

FActScore* systematically addresses language and knowledge source variability via parallel evaluation pipelines:

| Pipeline Type | Prompt & Output | Knowledge Source       | Verification Language    |
|---------------|----------------|-----------------------|-------------------------|
| L$\to$L       | Native in $L$   | Wikipedia-$L$         | $L$                     |
| L$\to$EN      | Native in $L$   | Wikipedia-English     | English (after translation) |
| EN$\to$EN     | English prompt  | Wikipedia-English     | English                 |

Each pipeline applies the same core FActScore computation as above. Per-entity and per-model scores combine pipeline results by simple averaging. This multi-pipeline approach characterizes "multilingual hallucination gaps," revealing substantial factuality drops and increased variance in medium- and low-resource languages [2410.18270].

## 3. Recipe for FActScore* Calculation

A concrete computation workflow for any language $L$ is:

1. **Generation Step**:
    - Elicit response $R_L$ to a prompt in language $L$.
2. **Decomposition**:
    - Extract atomic claims from $R_L$ (natively or post-translation).
3. **Fact-checking**:
    - For each claim, retrieve relevant passage(s) from $C$ (either Wikipedia in $L$ or English).
    - Form fact-checking prompt: $[{\text{evidence}}]$ + "Fact: $a$. True or False?"
    - LLM evaluates truth of each $a$ in context.
4. **Aggregation**:
    - Compute FActScore* for each pipeline.
    - Average results for the final metric.

The same process is applied to each prompt/model/language tuple. Worked examples in [2410.18270] demonstrate precise application and interpretation.

## 4. Cross-Linguistic Empirical Patterns

Empirical findings using FActScore* across 19 languages and multiple top-tier LLMs show:

- Factual precision declines systematically from high-resource (≈73%) to medium-resource (≈64%) and low-resource (≈57%) languages.
- Prompt and verification language matter: English-prompted outputs generally achieve higher factuality, with the advantage growing in lower-resource languages.
- The reliability of FActScore* itself degrades as resource-level drops, evidenced by rising variance in per-entity scores.
- Qwen-72B consistently achieves the highest FActScores among surveyed models; LLaMA variants perform more uniformly but at lower absolute levels.

A key insight is that neither scaling model size nor using English prompts closes factuality gaps in low-resource languages, highlighting the persistent challenge of multilingual evaluation and model robustness [2410.18270].

## 5. Knowledge Source Challenges and Mitigation

The limiting factor for FActScore* in many languages is knowledge source coverage. Research demonstrates:

- Wikipedia-L can be sparse in medium- and low-resource languages, leading to artificially low FActScores or unreliable support judgments.
- Machine translation errors in fact decomposition or evidence retrieval can propagate, but empirically result in negligible score drift when English-based verification is used [2402.18045, 2406.19415].
- Increasing the number of retrieved passages per claim, incorporating web/Google search results ("full Internet"), or using LLM-generated knowledge snippets significantly improves the estimation quality and accuracy of FActScore*, especially in under-resourced languages [2406.19415].

Mitigation strategies outlined in [2406.19415] include:
- Expanding retrieved passage sets ($k$ up to 20)
- Using real-time web search per claim
- Context augmentation with LLM-generated Q&A pairs

Empirical results show that Internet-based evidence yields the largest boosts, increasing low-resource (e.g., Bengali) factuality accuracy from ≈53% to ≈88% [2406.19415].

## 6. Interpretation, Best Practices, and Limitations

FActScore* is a relative, atomic claim–based precision measure, with several critical properties:

- It provides a scalable, automated alternative to costly human evaluation for hallucination auditing.
- The metric is easy to game via claim minimization or uninformative outputs; thus, it must be interpreted alongside output length, claim counts, and qualitative inspection.
- FActScore* assumes the knowledge source is comprehensive and up-to-date, which is often violated outside English and high-profile topics.
- The methodology demands rigorous calibration, e.g., native-speaker annotation to interpret low-resource language scores and auditing of decomposition fidelity.
- No weighting is applied to claims' semantic informativeness; all atomic facts count equally, potentially obscuring high-level utility or informativeness.

Recommended reporting includes both FActScore* and the absolute number of correct atomic facts per example. The metric is best viewed as a diagnostic for model ranking and comparative analysis rather than an absolute measure of document truth [2410.18270, 2402.18045]. Ongoing work emphasizes further improvements to knowledge retrieval, finetuned fact extractors for non-English languages, and dynamic or domain-adaptive evidence integration.

## 7. Broader Implications and Ongoing Research

FActScore* enables robust, fine-grained audit of hallucinations and factuality gaps in multilingual, multi-domain NLG. Its adoption in recent benchmarking and error analysis pipelines exposes pronounced performance disparities and guides targeted mitigation. As research progresses, integrating broader evidence sources, calibrating cross-lingual fact-checkers, and quantifying claim informativeness are emerging avenues to increase both the accuracy and utility of FActScore* and its descendants. The required technical and infrastructural developments outline the broader research agenda for factuality measurement in the context of increasingly global and diverse LLM deployments [2410.18270, 2406.19415].

Source: https://www.emergentmind.com/topics/factscore-41498db5-1b44-4331-b7ba-ed4d69603e54