AfrIFact: Multilingual African Fact-Checking Benchmark

• AfrIFact is a large-scale multilingual benchmark that structures fact-checking across document retrieval, evidence extraction, and verification pipelines.
• It comprises over 18,000 human-generated claims in healthcare and culture-news, covering 11 languages including 10 African languages and English.
• The benchmark exposes challenges in cross-lingual retrieval and low-resource evidence extraction, highlighting the benefits of language-adapted models over generic scaling.

Searching arXiv for AfrIFact and closely related African-language fact-checking / IR work to ground the article. AfrIFact is a large-scale, multilingual benchmark for automatic fact-checking in African languages. It was introduced to cover the full pipeline of information retrieval, evidence extraction, and fact checking in English and ten African languages, with claims drawn from two high-impact domains—healthcare and culture-news—and annotated with sentence-level evidence spans and three-way veracity labels: SUPPORTED, REFUTED, and NOT ENOUGH INFORMATION (Azime et al., 1 Apr 2026). Its central contribution is to make the bottlenecks of cross-lingual retrieval, multilingual evidence extraction, and low-resource fact verification directly measurable in a single dataset.

1. Scope and task formulation

AfrIFact comprises 18 256 human-generated claims in eleven languages: English, Amharic, Hausa, Igbo, Oromo, Shona, Swahili, Twi, Wolof, Yoruba, and isiZulu. The dataset is explicitly organized around the three stages of an automatic fact-checking pipeline. First, a system must retrieve relevant documents. Second, it must extract the minimal set of evidence sentences that support or refute the claim. Third, it must assign one of the three verdicts given the claim and the retrieved context (Azime et al., 1 Apr 2026).

The benchmark defines two retrieval regimes. The first is a cross-lingual multi-domain information-retrieval setting over a universal corpus that combines health documents, culture/news articles, and Wikipedia pages in all languages. The second is a monolingual, single-document setting in which each query retrieves evidence only from its source document. This dual design allows the benchmark to separate document-retrieval difficulty from sentence-selection difficulty and to expose the additional failure modes introduced by multilingual search.

The domain choice is consequential. Healthcare claims are drawn from WHO-derived material via AfriDocMT, while culture-news claims are drawn from Wikipedia and XL-Sum sources. This makes AfrIFact simultaneously a benchmark for multilingual factuality and a testbed for domain sensitivity, since the reported results show materially different behavior in health and culture-news conditions (Azime et al., 1 Apr 2026).

2. Dataset construction and annotation protocol

AfrIFact is divided between 9 328 healthcare claims and 8 928 culture-news claims. The healthcare portion is built from 93 WHO articles comprising 2 600 sentences translated into 11 languages; annotators generated 848 English claims and projected them into all languages, yielding 9 328 claim-language pairs. The culture-news portion uses 200 culturally focused Wikipedia/news articles per language, from which annotators generated 810–828 claims per language, yielding 8 928 pairs (Azime et al., 1 Apr 2026).

Component	Value	Notes
Total claims	18 256	Human-generated
Languages	11	English + 10 African languages
Healthcare claims	9 328	From WHO articles via AfriDocMT
Culture-news claims	8 928	From Wikipedia and XL-Sum
Train / Validation / Test	2 328 / 1 100 / 14 828	Summed over all languages

The split design is unusual by benchmark standards: the train set contains 2 328 examples, the validation set 1 100, and the test set 14 828. The authors state that the small train/validation allocation encourages fine-tuning research, while the large test set—about 80% of examples—supports robust evaluation (Azime et al., 1 Apr 2026).

Annotation quality control is a defining part of the resource. Each language uses three native-language annotators supervised by a language coordinator, and each annotator labels claims independently. Inter-annotator agreement is reported using Cohen’s $\kappa$, Fleiss’ $\kappa$, and Krippendorff’s $\alpha$, all above 0.7 across labels, indicating substantial agreement. Translation quality is checked with SSA-COMET, and any segment scoring below 0.6 is manually revised. These design choices make the dataset not merely multilingual in coverage but also explicitly curated for label reliability and translation fidelity.

3. Information retrieval and evidence extraction

AfrIFact benchmarks six embedding-based retrievers: mE5, mE5Instr, AfriE5, and Qwen-Embedding in 0.6B, 4B, and 8B variants. Retrieval is evaluated primarily with nDCG@10 and optionally Recall@100, with classic recall@k also reported as

$$\mathrm{Recall}@k \;=\;\frac{\bigl|\{\text{relevant docs}\}\;\cap\;\{\text{top-}k\text{ retrieved}\}\bigr|}{|\{\text{relevant docs}\}|}\,.$$

The principal empirical finding is that cross-lingual retrieval remains substantially weaker than monolingual retrieval, and that healthcare is harder than culture-news across the evaluated settings (Azime et al., 1 Apr 2026).

In monolingual retrieval, a same-language gap persists: on the health domain, mE5Instr reaches nDCG@10 of approximately 0.63 in English but only approximately 0.42 on average over African languages. In cross-lingual retrieval, performance drops by 50–70% relative to monolingual retrieval, which the paper describes as weak “strong-alignment” capabilities. Domain effects are also pronounced: culture-news retrieval reaches approximately 0.75 nDCG@10 in English and approximately 0.53 in African languages, whereas health retrieval reaches approximately 0.30 in English and approximately 0.22 in African languages. Model ranking is domain-sensitive: AfriE5 leads on health, while instruction-tuned mE5 leads on culture-news, indicating that adaptation to African-language data is beneficial.

Evidence extraction is evaluated after a document has been retrieved. The pipeline ranks sentences by embedding similarity and takes the top-3 as candidate evidence. Performance is measured primarily with nDCG@3 and Recall@3, and by $F_1$ when appropriate. Results show a pronounced English-versus-African-language gap in the health domain: peak English nDCG@3 reaches 0.79, while Swahili, the best-performing African language in this setting, reaches 0.52. In culture-news, the gap narrows and can reverse, with English at approximately 0.76 and Shona at approximately 0.80. The reported model-scaling result is equally important: larger generic embedding models yield only marginal gains, whereas language-adapted embeddings such as mE5 and AfriE5 outperform larger but non-adapted alternatives.

4. Fact-checking experiments and model behavior

AfrIFact evaluates fact verification under zero-shot prompting, few-shot prompting with three balanced in-context examples, and task-specific fine-tuning using LoRA, QLoRA, or full-model updates on Alpaca-formatted training data. The model set includes Gemma-3 12B and 27B, Qwen-14B, LLaMA-70B, Command-R, TinyAya, AfriQ-Gemma, AfriQueQwen, and GPT-5 (Azime et al., 1 Apr 2026).

Zero-shot results are weak. Accuracy on African languages is reported as hovering around random guess, approximately 33%, and the best open model, Gemma 3-27B, reaches only approximately 59% in zero-shot. Few-shot prompting changes the picture substantially. For AfriQueQwen-14B, accuracy improves by up to 43 points: from 33% to 77% on health and from 35% to 99% on culture. This establishes that in-context conditioning is a major factor for multilingual fact verification in the benchmark’s setting.

The effect of explicit evidence is more ambiguous than might be expected. The paper reports that simply providing the annotated evidence span offers inconsistent improvements, suggesting that models often rely on parametric knowledge rather than grounding decisively in the supplied evidence. That observation is technically important because it separates surface access to evidence from actual evidence-conditioned inference.

Fine-tuning results reinforce the value of parameter-efficient specialization. LoRA with rank 8 yields a 12–13% absolute gain on health and culture. QLoRA 8-bit yields a 26% gain on health, reaching approximately 65% average accuracy across African languages, and more than 20% on culture, reaching approximately 58%. Full fine-tuning shows marginal or no gains under the reported 3-epoch budget. Error analysis further indicates that fine-tuned models learn to use explicit evidence more effectively, reducing hallucinated SUPPORT labels and increasing correct NOT ENOUGH INFORMATION predictions. Persistent failure modes remain concentrated in low-resource languages such as Twi, Wolof, and Yoruba, and in healthcare topics where specialized terminology is sparse online.

5. Technical interpretation and common misconceptions

AfrIFact is best understood as a pipeline benchmark rather than a single-task dataset. Its structure reveals that failure can arise at multiple stages: cross-lingual retrieval can miss relevant documents, sentence ranking can fail to isolate the decisive evidence, and the verifier can misclassify even when evidence is present. A plausible implication is that end-to-end fact-checking accuracy in African languages is constrained less by any one model family than by compounded error across retrieval, extraction, and verification (Azime et al., 1 Apr 2026).

One common misconception is that model size alone is the dominant determinant of performance. AfrIFact does not support that view. In evidence extraction, model-size scaling yields marginal gains, while language-adapted embeddings outperform larger non-adapted models. In fact checking, PEFT methods outperform or match more expensive update regimes under the reported budget. The consistent pattern is that specialization to African-language data and tasks is more effective than generic scaling.

A second misconception is that multilingual zero-shot verification is already reliable if a sufficiently capable LLM is used. AfrIFact shows the opposite: zero-shot performance on African languages remains near random-guess levels in many settings. Few-shot prompting and PEFT materially improve accuracy, but the gains are not uniform across domains or languages, and health remains harder than culture-news. The benchmark therefore reframes multilingual fact checking as an adaptation problem rather than a pure inference problem.

The benchmark’s emphasis on language and culturally grounded documents also fits a broader African research context in which geography, history, culture, and language strongly shape knowledge networks rather than collapsing into a single uniform system (Adams et al., 2013). This suggests that AfrIFact’s multilingual design is not merely a translation exercise; it encodes heterogeneous linguistic and cultural retrieval conditions that affect downstream factuality assessment.

6. Applications, limitations, and future directions

AfrIFact is directly applicable to low-resource information retrieval, evidence retrieval, and multilingual fact checking. Its design is especially relevant for claims concerning healthcare and culture, which the paper identifies as areas with intensified real-world consequences when communities have limited access to information (Azime et al., 1 Apr 2026). In African public-health contexts, where limited health-care capacity and testing capability can coexist with severe consequences from delayed or ineffective responses, the availability of robust multilingual verification infrastructure is plausibly of practical importance (Schröder et al., 2020).

The resource also has clear limitations. Cross-lingual retrieval remains substantially weaker than monolingual retrieval, especially in healthcare. Evidence-conditioned verification is not yet robust, as shown by the inconsistent effect of supplying annotated evidence spans. Performance remains uneven across languages, with particular difficulty in Twi, Wolof, and Yoruba. These are not incidental shortcomings; they define the current research frontier that the benchmark is intended to expose.

The paper recommends four principal directions for subsequent work: extending the dataset to additional domains such as politics, finance, and local governance; exploring end-to-end neural pipelines that jointly retrieve, extract, and verify; developing specialized cross-lingual retrieval objectives to reduce the 50–70% performance gap; and investigating data augmentation and improved annotation strategies for low-resource coverage (Azime et al., 1 Apr 2026). Taken together, these recommendations position AfrIFact as an infrastructural benchmark for multilingual factuality research in Africa: a resource built not only to compare models, but to make visible where current systems still fail.