---
title: 'Wiki20d: Distantly Supervised RE Corpus'
url: https://www.emergentmind.com/topics/wiki20d
type: topic
---

# Wiki20d: Distantly Supervised RE Corpus

Wiki20d is a large-scale, sentence-level, fully distantly supervised (FDS) relation extraction (RE) corpus constructed by aligning English Wikipedia textual data with the Wikidata knowledge graph (KG). Each instance in Wiki20d corresponds to a sentence–entity-pair, annotated with all relations linking the head and tail entities in Wikidata, resulting in a multi-label classification setup that relies exclusively on KG-derived supervision. This methodology introduces inherent label noise, as the presence of a triple in Wikidata does not guarantee its semantic expression in a given sentence. Wiki20d provides a rigorous evaluation environment for developing relation extraction systems robust to noisy annotations and applicable in realistic KG-enrichment scenarios without recourse to manually annotated ground truth [2507.06895].

## 1. Construction and Annotation Protocol

Wiki20d is built upon pre-tokenized and sentence-split English Wikipedia articles, with each sentence $s_i$ processed to identify entity mention spans $\tau_{i,h}$ (head) and $\tau_{i,t}$ (tail). Entity recognition and linking are performed against Wikidata, yielding entity-pair candidates for relation annotation. The underlying KG is formalized as $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T})$ where $\mathcal{E}$ denotes the entity set, $\mathcal{R}$ the set of possible relation types, and $\mathcal{T} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$ the valid triples.

For each sentence containing mentions of entities $(e_h, e_t)$, distant supervision assigns as positive labels the relations $\mathcal{R}_{i} = \{\, r\in\mathcal{R}\mid (e_h, r, e_t) \in\mathcal{T}\, \}$, which are aggregated into a multi-hot relation vector $\mathbf{y}_i \in \{0,1\}^{|\mathcal{R}|}$ such that $\mathbf{y}_i^k=1$ if and only if $r_k \in \mathcal{R}_i$. No manual filtering of noisy labels is conducted. The only annotation-centric preprocessing eliminates sentences labeled as “NA” (no-relation) and those that exceed the pre-trained language model’s (PLM) context window length to prevent nonsensical or structurally invalid instances [2507.06895].

## 2. Dataset Statistics, Distribution, and Splits

After preprocessing, Wiki20d comprises 755 distinct relation types and over 600,000 sentence–entity pairs in the training set:

| Split         | Instances       | Notes                      |
|---------------|----------------|----------------------------|
| Training      | 614,207        | All supervision FDS only   |
| Validation    | 56,187         |                           |
| Test          | 92,083         |                           |

The empirical class-frequency distribution $p_r = \frac{N_r}{\sum_{k=1}^R N_k}$, where $N_r$ is the number of training examples with relation $r$ positive, follows a heavy long-tail—a small subset of relations dominates the label space, while many are rare. Approximately 20% of training instances in Wiki20d are multi-label ($\|\mathbf{y}_i\|_1 > 1$), compared to only about 1% in the test split. This imbalanced and noisy label regime is characteristic of real-world KG-derived distantly supervised corpora [2507.06895].

## 3. Features, Format, and Preprocessing

Each Wiki20d sample contains:

- Sentence text $s_i$
- Head-mention span $\tau_{i,h} = (\text{start},\text{end})$
- Tail-mention span $\tau_{i,t} = (\text{start},\text{end})$
- Multi-hot relation vector $\mathbf{y}_i \in \{0,1\}^{755}$

A common machine-readable representation is line-delimited JSON, with fields:

```
{
  "sentence": ...,
  "head_span": [start, end],
  "tail_span": [start, end],
  "relations": [r_1, r_2, ...]
}
```

During preprocessing, each record is mapped to $(\mathbf{x}_i, \mathbf{y}_i)$, where $\mathbf{x}_i \in \mathbb{R}^{2h}$ is formed by concatenating the average PLM embeddings of the head and tail token spans. This facilitates both end-to-end model training and efficient inference [2507.06895].

## 4. Evaluation Protocols and Metrics

Wiki20d is evaluated using several complementary metrics tailored to the multi-label, long-tail, and noisy-label context:

- **Micro- and Macro-averaged $F_1$:**  
  - Micro: $\displaystyle \text{microF1} = \frac{TP}{TP + 0.5\,(FN + FP)}$  
  - Macro: $\displaystyle \text{macroF1} = \frac{1}{R}\sum_{k=1}^R \frac{TP_k}{TP_k + 0.5\,(FN_k + FP_k)}$  
  Micro $F_1$ gauges performance over all decisions; macro $F_1$ weights each class equally.

- **Precision at R (P@R):**  
  For test example $j$ with $R_j = \|\tilde{\mathbf{y}}_j\|_1$ true relation labels, P@R is the fraction of correct labels in the model’s top $R_j$ predictions:  
  $$
  \text{P@R} = \frac{1}{|\mathcal{D}_{\mathrm{test}}|} \sum_{j=1}^{|\mathcal{D}_{\mathrm{test}}|} \frac{\left|\hat{\mathbf{y}}_{R_j}(\tilde{\mathbf{x}}_j)\,\cap\,\tilde{\mathbf{y}}_j\right|}{R_j}
  $$

- **Correlation Structure Distance (CSD):**  
  For label indicators $r_h$, $r_p$, let $\phi(\mathbf{Y}, r_h, r_p)$ denote their Pearson φ-coefficient. CSD measures the Frobenius norm between predicted and true correlation matrices:  
  $$
  \text{CSD} = \left\|\Phi(\hat{\mathbf{Y}}) - \Phi(\tilde{\mathbf{Y}})\right\|_F = \sqrt{\sum_{h,p} [\phi(\hat{\mathbf{Y}}, r_h, r_p) - \phi(\tilde{\mathbf{Y}}, r_h, r_p)]^2}
  $$  
  CSD quantifies the model’s ability to preserve the true co-occurrence structure of relation labels.

Baseline performance metrics on Wiki20d (Table 3 of the reference):

| Model | microF1 | macroF1 | CSD  |
|-------|---------|---------|------|
| PARE  | 67.2    | 19.6    | 5.30 |
| SCoRE | 66.9    | 23.9    | 2.31 |

SCoRE attains a slightly lower microF1 than PARE, but higher macroF1 and significantly reduced CSD, indicating improved handling of rare relations and better structural alignment to KG label co-occurrences [2507.06895].

## 5. Usage, Integration, and Best Practices

Wiki20d is distributed in JSON-lines or TSV formats under an open-source license permitting academic and commercial use. The recommended workflow is as follows:

- Encode each sentence once with any PLM (e.g., BERT-base), extract average vector embeddings for head/tail tokens to yield $\mathbf{x}_i$.
- Train a shallow multi-layer perceptron (MLP) using multi-label supervised contrastive loss to produce unit-norm embeddings $\mathbf{z}_i$.
- Inference proceeds by constructing a kNN datastore $\{ \mathbf{z}_i, \mathbf{y}_i \}$; for test sample $\tilde{\mathbf{z}}_j$, compute Bayesian kNN posteriors with an uninformative prior ($P(r_h)=\frac{1}{2}$) and a universal threshold ($c=0.7$).

Noise handling is achieved via multi-label contrastive learning, which clusters samples sharing similar relation-label structure, and kNN-based posterior inference, which employs universal thresholding to boost the recall of rare long-tail relations. This regime implements only a single pass of PLM encoding at initial data construction, minimizing overfitting to noisy distantly supervised targets and avoiding full PLM finetuning [2507.06895]. 

## 6. Significance and Research Impact

Wiki20d’s fully distantly supervised, high-class-cardinality (755 relations), and large-scale (over 600,000 training pairs) configuration establishes it as a benchmark for developing noise-resilient and resource-efficient relation extraction systems. By reflecting the realities of knowledge graph enrichment without human labeling, it facilitates rigorous assessment of noise-robust learning and models’ ability to generalize to tail relations and complex co-occurrence structures. Its adoption is recommended for evaluating and benchmarking multi-label RE methodologies under realistic, real-world conditions [2507.06895].

Source: https://www.emergentmind.com/topics/wiki20d