Wiki20d: Distantly Supervised RE Corpus

• Wiki20d is a large-scale, fully distantly supervised relation extraction corpus that aligns English Wikipedia sentences with Wikidata’s knowledge graph.
• It comprises over 600,000 training instances and 755 relation types, reflecting realistic, noisy, multi-label annotation derived from KG supervision.
• The dataset leverages PLM embeddings and multi-label contrastive learning techniques to robustly evaluate models under long-tail and noisy conditions.

Wiki20d is a large-scale, sentence-level, fully distantly supervised (FDS) relation extraction (RE) corpus constructed by aligning English Wikipedia textual data with the Wikidata knowledge graph (KG). Each instance in Wiki20d corresponds to a sentence–entity-pair, annotated with all relations linking the head and tail entities in Wikidata, resulting in a multi-label classification setup that relies exclusively on KG-derived supervision. This methodology introduces inherent label noise, as the presence of a triple in Wikidata does not guarantee its semantic expression in a given sentence. Wiki20d provides a rigorous evaluation environment for developing relation extraction systems robust to noisy annotations and applicable in realistic KG-enrichment scenarios without recourse to manually annotated ground truth (Mariotti et al., 9 Jul 2025).

1. Construction and Annotation Protocol

Wiki20d is built upon pre-tokenized and sentence-split English Wikipedia articles, with each sentence $s_i$ processed to identify entity mention spans $\tau_{i,h}$ (head) and $\tau_{i,t}$ (tail). Entity recognition and linking are performed against Wikidata, yielding entity-pair candidates for relation annotation. The underlying KG is formalized as $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$ where $\mathcal{E}$ denotes the entity set, $\mathcal{R}$ the set of possible relation types, and $$\mathcal{T} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$$ the valid triples.

For each sentence containing mentions of entities $(e_h, e_t)$, distant supervision assigns as positive labels the relations $$\mathcal{R}_{i} = \{\, r\in\mathcal{R}\mid (e_h, r, e_t) \in\mathcal{T}\, \}$$, which are aggregated into a multi-hot relation vector $\mathbf{y}_i \in \{0,1\}^{|\mathcal{R}|}$ such that $\tau_{i,h}$0 if and only if $\tau_{i,h}$1. No manual filtering of noisy labels is conducted. The only annotation-centric preprocessing eliminates sentences labeled as “NA” (no-relation) and those that exceed the pre-trained LLM’s (PLM) context window length to prevent nonsensical or structurally invalid instances (Mariotti et al., 9 Jul 2025).

2. Dataset Statistics, Distribution, and Splits

After preprocessing, Wiki20d comprises 755 distinct relation types and over 600,000 sentence–entity pairs in the training set:

Split	Instances	Notes
Training	614,207	All supervision FDS only
Validation	56,187
Test	92,083

The empirical class-frequency distribution $\tau_{i,h}$2, where $\tau_{i,h}$3 is the number of training examples with relation $\tau_{i,h}$4 positive, follows a heavy long-tail—a small subset of relations dominates the label space, while many are rare. Approximately 20% of training instances in Wiki20d are multi-label ($\tau_{i,h}$5), compared to only about 1% in the test split. This imbalanced and noisy label regime is characteristic of real-world KG-derived distantly supervised corpora (Mariotti et al., 9 Jul 2025).

3. Features, Format, and Preprocessing

Each Wiki20d sample contains:

• Sentence text $\tau_{i,h}$6
• Head-mention span $\tau_{i,h}$7
• Tail-mention span $\tau_{i,h}$8
• Multi-hot relation vector $\tau_{i,h}$9

A common machine-readable representation is line-delimited JSON, with fields:

$\mathcal{E}$1

During preprocessing, each record is mapped to $\tau_{i,t}$0, where $\tau_{i,t}$1 is formed by concatenating the average PLM embeddings of the head and tail token spans. This facilitates both end-to-end model training and efficient inference (Mariotti et al., 9 Jul 2025).

4. Evaluation Protocols and Metrics

Wiki20d is evaluated using several complementary metrics tailored to the multi-label, long-tail, and noisy-label context:

• Micro- and Macro-averaged $\tau_{i,t}$2:
  • Micro: $\tau_{i,t}$3
  • Macro: $\tau_{i,t}$4
  • Micro $\tau_{i,t}$5 gauges performance over all decisions; macro $\tau_{i,t}$6 weights each class equally.
• Precision at R (P@R):

For test example $\tau_{i,t}$7 with $\tau_{i,t}$8 true relation labels, P@R is the fraction of correct labels in the model’s top $\tau_{i,t}$9 predictions:

$\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$0

• Correlation Structure Distance (CSD):

For label indicators $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$1, $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$2, let $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$3 denote their Pearson φ-coefficient. CSD measures the Frobenius norm between predicted and true correlation matrices: $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$4 CSD quantifies the model’s ability to preserve the true co-occurrence structure of relation labels.

Baseline performance metrics on Wiki20d (Table 3 of the reference):

Model	microF1	macroF1	CSD
PARE	67.2	19.6	5.30
SCoRE	66.9	23.9	2.31

SCoRE attains a slightly lower microF1 than PARE, but higher macroF1 and significantly reduced CSD, indicating improved handling of rare relations and better structural alignment to KG label co-occurrences (Mariotti et al., 9 Jul 2025).

5. Usage, Integration, and Best Practices

Wiki20d is distributed in JSON-lines or TSV formats under an open-source license permitting academic and commercial use. The recommended workflow is as follows:

• Encode each sentence once with any PLM (e.g., BERT-base), extract average vector embeddings for head/tail tokens to yield $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$5.
• Train a shallow multi-layer perceptron (MLP) using multi-label supervised contrastive loss to produce unit-norm embeddings $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$6.
• Inference proceeds by constructing a kNN datastore $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$7; for test sample $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$8, compute Bayesian kNN posteriors with an uninformative prior ($\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$9) and a universal threshold ($\mathcal{E}$0).

Noise handling is achieved via multi-label contrastive learning, which clusters samples sharing similar relation-label structure, and kNN-based posterior inference, which employs universal thresholding to boost the recall of rare long-tail relations. This regime implements only a single pass of PLM encoding at initial data construction, minimizing overfitting to noisy distantly supervised targets and avoiding full PLM finetuning (Mariotti et al., 9 Jul 2025).

6. Significance and Research Impact

Wiki20d’s fully distantly supervised, high-class-cardinality (755 relations), and large-scale (over 600,000 training pairs) configuration establishes it as a benchmark for developing noise-resilient and resource-efficient relation extraction systems. By reflecting the realities of knowledge graph enrichment without human labeling, it facilitates rigorous assessment of noise-robust learning and models’ ability to generalize to tail relations and complex co-occurrence structures. Its adoption is recommended for evaluating and benchmarking multi-label RE methodologies under realistic, real-world conditions (Mariotti et al., 9 Jul 2025).