---
title: PolyTruth Disinfo Corpus
url: https://www.emergentmind.com/topics/polytruth-disinfo-corpus
type: topic
---

# PolyTruth Disinfo Corpus

Searching arXiv for the specified paper and closely related context.
arxiv_search.search(query="PolyTruth Multilingual Disinformation Detection using Transformer-Based Language Models", max_results=5)
I found the relevant arXiv entry for the topic-specific paper and use it as the primary source for the article: "PolyTruth: Multilingual Disinformation Detection using Transformer-Based Language Models" [2509.10737].
PolyTruth Disinfo Corpus is a multilingual disinformation-detection benchmark introduced alongside a systematic comparison of five multilingual transformer models—mBERT, XLM, XLM-RoBERTa, RemBERT, and mT5—on a common fake-vs-true machine learning classification task. It consists of $|D| = 60{,}486$ statement pairs, with 30,243 debunked false-claim statements sourced from MindBugs Discovery and 30,243 corresponding true or factual-correction statements created as PolyTruth augmentations. The corpus spans over twenty five languages, covers five language families, and includes the domains politics, health, climate, finance, and conspiracy. It was introduced to address the gap between multilingual disinformation spread and the continued predominance of English-only benchmarking in AI systems for disinformation detection [2509.10737].

## 1. Corpus definition and representational design

The corpus is organized around paired statements rather than isolated texts. Each record contains a false statement and a corresponding true or corrective statement, with binary labels assigned as `"false"` for the disinformation claim and `"true"` for the factual correction or neutral true statement. This pairing makes the resource explicitly suitable for binary classification under a fake-vs-true formulation, while also preserving a direct semantic relation between the disinformation item and its correction [2509.10737].

The data record schema is JSON-style and includes an identifier, language code, paired texts, topical domain, debunking date, source, and the two labels. All text fields are Unicode strings, dates follow ISO 8601, and languages use two-letter ISO 639-1 codes. The schema is as follows:

| Field | Description |
|---|---|
| `id` | unique identifier for the pair |
| `lang` | ISO 639-1 code, e.g., `en`, `ru` |
| `text_false` | original false statement text |
| `text_true` | paired true/correction text |
| `domain` | one of `politics`, `health`, `climate`, `finance`, `conspiracy` |
| `date_debunked` | date the false claim was fact-checked, `YYYY-MM-DD` |
| `source` | fact-check organization or news outlet |
| `label_false` | `false` |
| `label_true` | `true` |

This record design places factual correction adjacent to the disinformation claim at the instance level. A plausible implication is that the corpus can support not only standard supervised classification but also analyses of corrective language, although only binary classification is explicitly described.

## 2. Linguistic coverage and resource stratification

PolyTruth covers 25+ languages. The language distribution is explicitly stratified into high-resource, mid-resource, and low-resource groupings. The top five high-resource languages, each with more than 3,000 pairs, are Russian, Portuguese, German, Czech, and Dutch. Russian is the largest single language with approximately 4,488 pairs. Mid-resource languages with 1,000–3,000 pairs are English, Spanish, French, Polish, Arabic, Hungarian, and Romanian. Low-resource languages with fewer than 1,000 pairs include Bulgarian, Greek, Slovak, Estonian, Latvian, and Azerbaijani [2509.10737].

The corpus spans five language families with approximate counts given for several members. Indo-European languages dominate the inventory and include Germanic, Romance, Slavic, and Hellenic branches; Afro-Asiatic is represented by Arabic; Uralic by Hungarian and Estonian; Turkic by Azerbaijani; and Sino-Tibetan is listed with Chinese marked as absent. The documented excerpt gives the following language counts: English (2,500), German (3,200), Dutch (1,800), Spanish (2,700), Portuguese (3,100), French (2,200), Romanian (1,100), Russian (4,488), Polish (1,200), Czech (3,300), Bulgarian (450), Slovak (380), Greek (520), Arabic (1,050), Hungarian (1,100), Estonian (480), and Azerbaijani (310).

The benchmark further defines a high- vs. low-resource split: high-resource comprises 10 languages with at least 1,000 pairs, from Russian through Hungarian, and low-resource comprises the remaining languages, from Romanian down to Azerbaijani. This resource-sensitive partition is central to the reported experimental findings, because performance differences across multilingual models are presented partly through their degradation on low-resource languages.

## 3. Topical composition and temporal profile

PolyTruth is topically organized into five domains: politics, health, climate, finance, and conspiracy. The approximate statement-pair counts are balanced between false and true items within each domain and are distributed as follows: politics $\sim 12{,}000$, health $\sim 11{,}000$, climate $\sim 9{,}000$, finance $\sim 10{,}000$, and conspiracy $\sim 8{,}486$ [2509.10737].

The domain distribution is not temporally uniform. Politics disinformation peaks around US elections in 2016 and 2020 and the Russo-Ukrainian war in 2022. Health disinformation surges in 2020 during COVID-19. Finance and climate exhibit more gradual yearly growth. Conspiracy claims are characterized as often short, sarcastic, or coded in local idioms. These observations are significant because they indicate that the benchmark mixes event-driven surges with slower longitudinal accumulation, producing a corpus whose class instances are shaped both by global crises and by domain-specific discourse patterns.

The topical range matters methodologically because multilingual disinformation detection can otherwise be confounded by narrow-domain regularities. The inclusion of politics, health, climate, finance, and conspiracy reduces the likelihood that benchmark performance reflects only a single discourse regime. This suggests that cross-lingual generalization in PolyTruth is tested against both linguistic variation and topical heterogeneity.

## 4. Source construction, correction generation, and quality control

The false claims are drawn from MindBugs Discovery: 30,243 statements from 2009–2024, each verified by established fact-checkers such as PolitiFact and AFP Fact Check. The corresponding true or correction statements were created through two routes. The primary route used an OpenAI API prompt to “state the corrected fact” in the same language. The fallback route extracted a neutral true statement from reputable news in the same language and topic. Annotation guidelines and prompt design are available in the GitHub repository [2509.10737].

Quality control was designed to constrain synthetic error propagation. Each synthetic true statement underwent human review to eliminate hallucinations. Pairs were filtered out when the true text introduced new false claims. Explicit fact-check verdict tags such as “(False)” and “(Hoax)” were removed. These steps define the true side of the corpus not as unchecked synthetic text but as reviewed corrective content aligned to the language and topic of the source claim.

The preprocessing pipeline comprises four stages: lowercasing and Unicode normalization; stripping URLs and user mentions; tokenization via each model’s subword tokenizer; and a stratified 80/10/10 train/val/test split with no leak of statements from the same fact-check across splits. The no-leak constraint is especially important because related statements from a single fact-check could otherwise artificially inflate test performance through near-duplicate content or source-level lexical signatures.

## 5. Evaluation protocol and reported metrics

The benchmark evaluates five models fine-tuned under identical settings: mBERT, XLM, XLM-R (base), RemBERT, and mT5 (base). The task is binary classification over the `"false"` and `"true"` classes using the 80/10/10 split. The reported metrics include accuracy, macro-F1, and $F1$ for the `"false"` class. Cross-lingual average accuracy is defined as
$$
Acc_{avg} \;=\; \frac{1}{L}\sum_{i=1}^{L} Acc_i
$$
where $L$ is the number of languages and $Acc_i$ is the accuracy on language $i$. Macro-F1 is defined as
$$
F1_{macro} = \frac{1}{C}\sum_{c=1}^{C} F1_c
$$
for $C=2$ classes, namely false and true [2509.10737].

The principal test results are:

| Model | Accuracy | Macro F1 |
|---|---:|---:|
| mBERT | 79.3% | 0.793 |
| XLM | 81.0% | 0.810 |
| XLM-R (base) | 85.4% | 0.854 |
| RemBERT | 87.1% | 0.871 |
| mT5 (base) | 84.6% | 0.846 |

The benchmark also reports average $F1$ on the `"false"` class for high-resource and low-resource language groups. RemBERT achieves 0.91 in high-resource and 0.78 in low-resource settings; XLM-R (base) 0.89 and 0.74; mT5 (base) 0.88 and 0.72; XLM 0.85 and 0.67; and mBERT 0.83 and 0.61. Because the resource-group comparison is framed specifically on the false class, it directly tracks the models’ ability to identify disinformation rather than aggregate balanced performance alone.

## 6. Empirical patterns, limitations, and deployment implications

The reported findings emphasize performance variation across model families and resource conditions. RemBERT achieves the highest overall accuracy at 87.1% and the best low-resource consistency, with only an approximately 0.13 drop from high-resource to low-resource average $F1$ on the false class. XLM-R is described as offering an excellent accuracy/efficiency trade-off, with 85.4% accuracy and 0.74 low-resource $F1$. mBERT exhibits substantial limitations when training data is scarce, reaching only 0.61 low-resource $F1$. mT5, despite being a seq2seq model, is competitive with encoder-only models, which the paper interprets as indicating that generative approaches are viable for classification [2509.10737].

These results bear directly on a common misconception in multilingual NLP: that English-centric success with transformer models straightforwardly transfers to multilingual disinformation detection. The benchmark explicitly frames that assumption as unsettled, stating that transformer-based language models have demonstrated notable success in English while their effectiveness in multilingual contexts remains up for debate. The low-resource results support that caution. Performance remains materially dependent on model choice and language-resource conditions rather than on the mere use of a multilingual transformer.

Another potential misconception is that the corpus is simply a translation benchmark. The documented construction does not describe translated duplicates across languages; it describes false claims sourced from MindBugs Discovery and paired true statements produced in the same language or extracted from reputable news in the same language and topic. This suggests a multilingual fact-checking-aligned corpus rather than a parallel translation dataset.

For real-world deployment, the reported pattern implies that model selection should account for low-resource robustness and not only aggregate accuracy. The paper’s discussion of performance patterns and implications for deployment is grounded in the observation that languages with fewer training instances expose the limitations of some widely used multilingual encoders more clearly than high-resource languages do.

## 7. Access, reuse, and research utility

The corpus is publicly available at `https://github.com/gouliev/polytruth`, with license CC-BY 4.0 per the repository. The data format is JSON with one record per line, accompanied by train/val/test TSV splits. Researchers can directly download and load the JSON lines via the GitHub URL, filter by language or domain, and reproduce fine-tuning experiments using the provided train/val/test splits and sample scripts in the repository [2509.10737].

The recommended citation given in the repository is: Gouliev Z, Wang C, Waters J (2024), “PolyTruth: Multilingual Disinformation Detection using Transformer-Based Language Models.” The resource is therefore positioned not only as a benchmark for model comparison but also as a reusable corpus for controlled experiments across languages, domains, and resource regimes.

In the broader research landscape, PolyTruth is best understood as a multilingual benchmark centered on verified disinformation claims and paired factual corrections, with explicit attention to low-resource evaluation. Its main contribution lies in combining broad linguistic coverage, domain diversity, paired false/true structure, fact-check-grounded sourcing, and a uniform transformer comparison within a single experimental framework.

Source: https://www.emergentmind.com/topics/polytruth-disinfo-corpus