---
title: 'FactNet-Bench: Multilingual KG & Fact Checking'
url: https://www.emergentmind.com/topics/factnet-bench
type: topic
---

# FactNet-Bench: Multilingual KG & Fact Checking

FactNet-Bench is a unified evaluation suite derived from the FactNet resource, designed to facilitate rigorous and reproducible multilingual research in Knowledge Graph Completion, Multilingual Knowledge-Based Question Answering, and Closed-Context Fact Checking at scale. Built atop a billion-scale deterministic knowledge graph encompassing 1.7 billion atomic Wikidata statements and over 3 billion span-grounded evidence pointers across 316 Wikipedia editions, FactNet-Bench operationalizes three task-specific benchmarks with strict leakage controls and baseline implementations, establishing new standards in dataset auditability, provenance, and linguistic breadth [2602.03417].

## 1. Definition and Scope

FactNet-Bench serves as a comprehensive evaluation environment integrating three key tasks derived from the FactNet knowledge graph: Knowledge Graph Completion (FactNet-KGC), Multilingual Knowledge-Based Question Answering (FactNet-MKQA), and Multilingual Closed-Context Fact Checking (FactNet-MFC). Each task is instantiated with fixed dataset splits, rigorous leakage controls, and canonical baseline models. FactNet-Bench leverages FactNet’s uniquely deterministic, byte-precise evidence pointer system and broad multilingual coverage, enabling robust benchmarking across entity-centric, text-grounded, and evidence-intensive scenarios.

## 2. Dataset Construction and Key Statistics

FactNet-Bench is constructed using a fully deterministic pipeline, eschewing stochastic components to guarantee byte-level auditability and split reproducibility. The pipeline ingests two primary data sources: Wikidata JSON dumps (12.1K properties, 1.7B statements) and 316 Wikipedia XML editions with SQL sitelink tables (as of 2025-11-01). Canonical page views are extracted per Wikipedia page—Sentence (markup-stripped, segmented via Stanza or rule-based splitter), Template (AST-extracted infobox fields), and Table (cellular content). Statements are grouped via versioned normalization into FactSynsets, and fact-evidence alignments are performed using prioritized, datatype-aware matchers (infobox key-value, wikilink entity, lexical).

FactNet-Bench statistics after benchmark filtering are summarized below:

| Benchmark          | #Train    | #Dev   | #Test   | #Languages | Notes                                                            |
|--------------------|-----------|--------|---------|------------|------------------------------------------------------------------|
| KGC triples        | 4,180,000 | 520,000| 520,000 | —          | 248K entities, 320 relations, avg degree 33.7                    |
| MKQA questions     | 54,000    | 6,800  | 6,800   | 18         | 62% 1-hop/38% 2-hop, avg answer size 2.6                         |
| MFC claims         | 72,000    | 9,000  | 9,000   | 18         | S/R/NEI=.34/.33/.33, avg evidence 1.4, text length 210 chars      |

For MKQA and MFC, the supported languages are the 18 largest Wikipedias. Across FactNet, evidence units breakdown: Sentence 57.5%, Infobox 28.4%, Table 14.1%. Audited grounding precision is 0.921 (95% CI [0.913, 0.929]), with evidence-match type precisions: wikilink entity 0.973, infobox field 0.944, lexical value 0.889, lead weak 0.808.

## 3. Task Formalizations

### 3.1 Knowledge Graph Completion (FactNet-KGC)

The KG Completion task evaluates models on predicting missing entities for incomplete triples. Given $G_{train} = \{(s,p,o)\}$, input consists of either $(s,p,?)$ or $(?,p,o)$ and output is a ranked list of entity candidates $o \in E$. Metrics include filtered Mean Reciprocal Rank (MRR) and Hits@K:
$$
MRR = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{rank_i}, \qquad Hits@K = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \mathbb{1}(rank_i \leq K)
$$

### 3.2 Multilingual Knowledge-Based Question Answering (FactNet-MKQA)

FactNet-MKQA maps a natural-language question $q_\ell$ in language $\ell$ to an executable logical form $z$ over FactNet identifiers, conforming to the “hop1”, “hop2”, or “hop2c” grammar:
- hop1: $(hop1\ SUBJ\ PID)$
- hop2: $(hop2\ SUBJ\ PID\ PID)$
- hop2c: $(hop2c\ SUBJ\ PID\ PID\ CONSTRAINT)$

Output is $A = \varphi(z)$, the answer set executing $z$ on the frozen KG. Metrics are macro F1 (after normalization) and Valid% (fraction of outputs parsing and executing under the grammar).

### 3.3 Multilingual Closed-Context Fact Checking (FactNet-MFC)

Given a claim $c_\ell$, models classify it as Supported, Refuted, or NEI, and retrieve supporting FactSense evidence units, with optional token spans. Retrieval is performed over top-K evidence units. Metrics: label accuracy, macro F1, Evidence Recall@K (≥1 correct evidence in top K), and Span Evidence F1 (token-level).

## 4. Evaluation Protocol and Leakage Controls

Global dataset splits are generated by stable hashing of synset_id: $h = u32(SHA1(build\_id ∥ synset\_id)[0:4]) \bmod 100$, mapping Train $<$80, Dev 80–90, Test 90–100. Critical leakage controls are implemented:
- Training text drawn strictly from Train-aligned FactSenses.
- Predicate masking in text-aware KGC, obscuring property values in entity descriptions to block trivial extraction.
- RelationEdge restriction: only edges between Train synsets in KGC Graph Neural Networks.
- Distinct “Train-only”/“Full” retrieval indexes for MFC.
- Uniform negative sampling: 256 negatives for KGE, 128 for GNN baselines.
- Random seeds fixed at {13,21,42}; mean±std reported for learned models.

## 5. Baseline Models and Benchmark Results

### 5.1 Knowledge Graph Completion

Structural baselines: TransE, RotatE, CompGCN. Text-aware baselines: SimKGC (contrastive PLM), KG-S2S (seq2seq PLM). Predicate masking is verified crucial; without it, KG-S2S MRR increases from 0.298 to 0.351, validating that masking suppresses leakage.

| Model      | MRR   | Hits@10 |
|------------|-------|---------|
| TransE     | 0.243 | 0.410   |
| RotatE     | 0.261 | 0.453   |
| CompGCN    | 0.284 | 0.478   |
| SimKGC     | 0.293 | 0.486   |
| KG-S2S     | 0.298 | 0.492   |

### 5.2 Multilingual Question Answering

mT5 baseline: standard and grammar-guided decoding. LLMs: Qwen-2.5-72B, LLaMA-3.3-70B (5-shot, grammar-constrained).

| Model                 | Macro F1 | Valid % |
|-----------------------|----------|---------|
| mT5 (no grammar)      | 30.9     | 88.5    |
| mT5 (grammar guided)  | 34.1     | 95.2    |
| Qwen-2.5-72B (5-shot) | 41.4     | 93.8    |
| LLaMA-3.3-70B (5-shot)| 36.7     | 92.5    |

### 5.3 Fact Checking

Baselines: hypothesis-only classifier; retrieval + XLM-R NLI verifier (BM25, E5-large dense, translation-assisted retrieval). Top-5 retrieval aggregation improves accuracy and evidence F1.

| System                    | Acc   | Macro F1 | R@5  | Span F1 |
|---------------------------|-------|----------|------|---------|
| Hyp-only                  | 0.381 | 0.375    | —    | —       |
| BM25 + XLM-R              | 0.654 | 0.641    | 0.76 | 0.41    |
| E5-large + XLM-R          | 0.701 | 0.692    | 0.83 | 0.49    |
| E5-large + XLM-R (Top-5)  | 0.731 | 0.724    | 0.91 | 0.54    |

## 6. Analysis and Insights

Structural KGC baselines yield the expected performance hierarchy (TransE < RotatE < CompGCN), with text-aware models providing 0.01–0.02 improvement in MRR. GNNs gain further from train-only RelationEdges. Predicate masking is essential for suppressing trivial solutions in text-rich settings. In MKQA, grammar-guided decoding yields substantial validity (+6.7 points) and macro F1 (+3.2), with highest semantic accuracy from LLM few-shot prompting (Qwen-2.5-72B, 41.4 Macro F1), but best Valid% achieved by grammar-constrained mT5. Performance drops in low-resource languages; analysis identifies template mapping and alias gaps as bottlenecks in non-high-resource tiers.

In MFC, evidence retrieval dominates verifier performance: dense retrieval (E5) exceeds BM25 by 0.05–0.07 R@5, and top-5 aggregation further boosts F1 and accuracy. The hypothesis-only baseline attests to minimal annotation artifacts. While FactNet covers 316 languages, the top 5 comprise ~63% of senses; long-tail languages retain high grounding precision (Tier 3: 0.885). The primary bottlenecks are template extraction and alias resolution in non-majority languages.

## 7. Access, Environment, and Reproducibility

FactNet-Bench is available under permissive (CC0, Evidence-Text Pack CC BY-SA) licenses at HuggingFace (https://hf.co/collections/openbmb/factnet). The deterministic construction pipeline is hosted at https://github.com/yl-shen/factnet, supporting full dataset and split reconstruction. Data formats include JSONL and Parquet shards with explicit schema versioning; loading is supported by index scripts.

Dependencies: Python ≥3.8, Apache Parquet (pyarrow), mwparserfromhell, Stanza (with pinned checksums), and PyTorch/Transformers for baselines. Container manifests or Conda environments with exhaustive version control ensure execution reproducibility.

Best practices: employ released build manifests and language pack hashes, freeze random seeds (13, 21, 42), enforce split-aware filtering and predicate masking, and validate evidence pointers using the provided utility. Following the protocol enables full, byte-exact reproduction of dataset, splits, and baseline metrics for all FactNet-Bench tasks [2602.03417].

Source: https://www.emergentmind.com/topics/factnet-bench