Multilingual Datasets

• Multilingual datasets are structured corpora comprising text, speech, and image data in multiple languages, supporting cross-lingual transfer and detailed evaluation.
• They are constructed using methods like web-scale crawls, machine translation, and native-speaker annotation, ensuring high-quality data through rigorous filtering and validation.
• These datasets drive progress in NLP, speech technologies, and multimodal AI by providing benchmarks, reducing language biases, and enabling research on low-resource languages.

Multilingual datasets are structured corpora encompassing multiple languages, providing a foundational resource for training, evaluation, and benchmarking of models across diverse language technologies. Their construction, annotation methodologies, challenges, and influence on model development are central to NLP, speech technologies, and multimodal AI. Recent efforts have produced large-scale and fine-grained multilingual datasets that cover a broad spectrum of tasks and typologies, from fundamental pre-training corpora to specialized benchmarks.

1. Definition, Scope, and Research Motivations

A multilingual dataset consists of data instances (text, speech, image-text, etc.) sampled or generated in multiple languages, potentially including parallel (aligned) and non-parallel components. These datasets serve to:

• Enable cross-lingual transfer learning, supporting both resource-rich and low-resource languages.
• Benchmark model performance under multilingual and code-switched conditions.
• Facilitate sociolinguistic and cross-cultural analysis of language phenomena and technology impact.

Large, high-quality multilingual datasets are essential for both the pre-training of foundation models (e.g., LLMs, multimodal transformers) and for targeted downstream evaluation. As shown in the analysis of 156 NLP datasets, more than two-thirds of world languages still lack manually annotated data, despite recent expansions in automated corpus induction (Yu et al., 2022).

2. Construction Methods, Data Pipelines, and Quality Assurance

Modern multilingual dataset construction employs a diversity of methodologies:

• Web-Scale Crawls and Filtering: Repositories such as HPLT v2/v3 (Oepen et al., 2 Nov 2025, Burchell et al., 13 Mar 2025) aggregate multilingual content from the Internet Archive, Common Crawl, and other sources, applying language identification (e.g., OpenLID-v2), deduplication (MinHash and Jaccard filtering), boilerplate removal (e.g., Trafilatura), and content-based scoring (Web Docs Scorer, News Report classifier).
• Machine Translation Pipelines: Many task-specific multilingual datasets (e.g., answer sentence selection (Gabburo et al., 2024), relation extraction (Bassignana et al., 2023), aspect-based sentiment analysis (Wu et al., 17 Feb 2025)) are generated via supervised MT from English to target languages, often accompanied by semantic filtering (cosine similarity thresholds), artifact removal, and manual quality control.
• Multilingual Annotation and Curation: Datasets such as DimStance (Becker et al., 29 Jan 2026), Multi3WOZ (Hu et al., 2023), and SwitchLingua (Xie et al., 30 May 2025) employ native-speaker annotation for complex phenomena (e.g., stance valence-arousal, culturally adapted dialog, code-switching). Human validation is essential for marking entity spans, switch-points, and cross-cultural features.
• Benchmark-Oriented Multi-Agent Generation: SwitchLingua employs a multi-agent LLM framework, where specialized agents generate, evaluate, and refine code-switched text. Quality assurance is iterative and constrained by explicit linguistic and social factors.
• Sample-Quality Auditing: Quality control methods, such as Preference Proportion Test (PPT) (Samir et al., 2024), use statistical hypothesis testing to systematically identify low-quality language subsets within large collections, maximizing downstream utility and generalization.

3. Linguistic, Domain, and Script Coverage

Multilingual datasets now span a wide typological, domain, and script landscape:

• Scale and Diversity: Datasets like HPLT v3.0 include monolingual and parallel data for nearly 200 language–script combinations, with token volumes reaching 30 trillion across domains (news, government, web forums, Wikipedia, legal, etc.) (Oepen et al., 2 Nov 2025).
• Low-Resource and Code-Switching Support: Efforts such as CUTE (Chinese-Uyghur-Tibetan-English) deliver 25 GB-scale corpora for notably low-resource languages, achieving human-validated translation quality sufficient for high-quality pretraining and cross-lingual transfer (Zhuang et al., 21 Sep 2025). SwitchLingua and CS-FLEURS curate the largest open code-switched resources, supporting up to 113 code-switched pairs and 52 languages (Xie et al., 30 May 2025, Yan et al., 17 Sep 2025).
• Task Domains:
  • NLP Tasks: Answer sentence selection (Gabburo et al., 2024), argument mining (Toledo-Ronen et al., 2020), relation extraction (Bassignana et al., 2023), sentiment analysis (Wu et al., 17 Feb 2025), question answering, and summarization (Hewapathirana et al., 2024).
  • Vision-Language: DataComp's multimodal pool and translation-augmented image-text curation methods improve both monolingual and cross-lingual vision-language representation learning (Nguyen et al., 2024).
  • Conversational and Dialog Systems: Multi3WOZ (Hu et al., 2023) establishes scale, parallelism, and cultural fidelity for multi-domain, multi-lingual task-oriented dialog.
  • Paraphrase/Summarization: IndicNLG (Kumar et al., 2022) and M2DS (Hewapathirana et al., 2024) fill regional and summarization gaps at scale in Indic, South Asian, and East Asian languages.

4. Annotation Paradigms, Manual vs. Automatic Induction, and Translation Issues

A prominent trend is the combination of manual annotation, automated distant supervision, and machine translation:

Creation Method	Proportion (Typical in Survey)	Tasks Benefited
Manual annotation	17–48% (varies by language)	NER, sentiment, argument mining
Auto-induced (heuristics)	33–84%	POS, segmentation, MT failures
Human or auto-translation	5–20%	Parallel corpora, classification

Manual annotation yields the highest utility in low-resource settings and complex labeling (stance, code-switching, semantic roles), but is rarely scalable. Automatic induction and MT pipelines enable wide coverage but risk translationese, label misalignment, and typological mismatches, as measured by annotation agreement (RMSE, Cohen's $\kappa$, F1), and semantic preservation (Pearson $r$, cos similarity) (Yu et al., 2022, Toledo-Ronen et al., 2020, Wu et al., 17 Feb 2025).

Label preservation varies by task: stance and evidence transfer better than argument quality or category (Toledo-Ronen et al., 2020); entity span preservation can degrade (notably for compound languages or typologically distant pairs) (Bassignana et al., 2023).

5. Applications, Benchmarks, and Evaluation Metrics

Multilingual datasets are central to a range of benchmarks and deployment scenarios:

• Pretraining and NLU: HPLT v2/v3 models, trained on filtered corpora, consistently outperform prior masked LLMs on POS, NER, and parsing across 52+ languages. Monolingual T5 models trained on HPLT v3.0 data outperform mT5 on WikiAnn NER and MultiBLIMP-style linguistic competence (Oepen et al., 2 Nov 2025, Burchell et al., 13 Mar 2025).
• Zero-Shot and Cross-Lingual Transfer: Transfer learning using multilingual datasets closes most of the performance gap between English and L2 systems in answer sentence selection (MAP, P@1 improvements of 8–11%) (Gabburo et al., 2024); code-switching ASR remains challenging (SAER 0.15–0.25 for best models) (Xie et al., 30 May 2025, Yan et al., 17 Sep 2025).
• Evaluation Metrics: Traditional WER/CER are insufficient for code-switching; metrics such as Semantic-Aware Error Rate (SAER), combining form error and multilingual embedding-based semantic similarity, correct for script and orthographic equivalence (Xie et al., 30 May 2025).
• Dimensional Analysis: DimStance introduces continuous valence-arousal annotation, supporting regression objectives beyond categorical stance tasks, and provides RMSE as a cross-lingual metric (Becker et al., 29 Jan 2026).
• Resource Surveys: Dataset availability, by language and task, correlates strongly with researcher and crowd-worker demographics, with coverage, manual-vs-automatic index, and translation reliance serving as disparity metrics (Yu et al., 2022).

6. Challenges, Limitations, and Best Practices

Persistent challenges in multilingual dataset construction include:

• Low-Resource Data Gaps: 68% of world languages lack manually labeled data; most benchmarks bias toward Indo-European and high-resource scripts (Yu et al., 2022). Targeted initiatives like CUTE and IndicNLG attempt to ameliorate this gap (Zhuang et al., 21 Sep 2025, Kumar et al., 2022).
• Quality Control and Bias: Noisy language identification and translation artifacts risk systematic errors, especially in under-resourced settings. Methods such as the Preference Proportion Test provide lightweight audit protocols to flag unreliable subsets with small annotated samples (Samir et al., 2024).
• Data Curation and Filtering: Filtering after translation and deduplication is essential to surface high-quality non-English samples; otherwise, models overfit to English-centric content (Nguyen et al., 2024, Burchell et al., 13 Mar 2025).
• Ethics and Licensing: Modern large-scale projects (e.g., Sri Lanka Document Datasets) align with FAIR principles, and permissive licensing (CC-BY-4.0, MIT) is emerging as a standard (Senaratna, 5 Oct 2025). However, research-only restrictions and explicit misuse prohibitions (e.g., in speech datasets for voice-cloning) remain common (Xie et al., 30 May 2025).

Recommended Practices:

• Compose datasets via scrupulous multi-stage filtering, language identification, and native-speaker quality control.
• Union original and translated pools with separate re-filtering, retaining multiple aligned captions when possible (Nguyen et al., 2024).
• Use power analyses to justify annotation budget for statistical quality assurance (Samir et al., 2024).
• Publish all dataset creation, filtering, and benchmarking code under open-source licenses for reproducibility (Oepen et al., 2 Nov 2025, Burchell et al., 13 Mar 2025).
• Develop explicit resource disparity measures and document provenance, annotation, and translation pipelines with standardized data statements (Yu et al., 2022).

7. Impact and Future Directions

Multilingual datasets are now the bedrock of advances in cross-lingual LLMs, transfer learning, and robust AI systems for global contexts. Their inclusion demonstrably benefits even English-centric tasks through diversification of concepts and representations (Nguyen et al., 2024). Future work is anticipated to focus on:

• Expanding fine-grained, culturally adapted resources (e.g., Multi3WOZ, SwitchLingua) beyond translationese paradigms (Xie et al., 30 May 2025, Hu et al., 2023).
• Quantitative measurement of annotation and translation reliability across all typological spectra.
• Longitudinal and sociocultural analysis of multilingual model biases as dataset diversity increases.
• Semi-automated, crowd-based rapid curation workflows for emergent or rare languages, tuned for label quality via qualification tasks and compensation models (Yu et al., 2022).

The scaling, annotation rigor, and methodological sophistication of recent multilingual datasets mark a transition to mature, reproducible, and globally inclusive language technology research.