Multilingual Web Pretraining Data

• Multilingual web pretraining data is a large-scale, diverse corpus collected from web crawls and meticulously processed to support cutting-edge LLM training across multiple languages.
• Advanced pipelines employ language identification, deduplication, and model-based filtering to ensure high quality and balanced representation across high- and low-resource languages.
• Data augmentation through synthetic translation and dynamic sampling strategies drives significant performance gains, enhancing cross-domain generalization and benchmark accuracy.

Multilingual web pretraining data denotes large-scale, high-diversity, web-crawled corpora assembled and curated to support the training of LLMs in many languages, including both high- and low-resource settings. Such datasets underpin the development of state-of-the-art multilingual LLMs and downstream benchmarks. They are distinguished by their scale (often tens of trillions of tokens), systematic pipeline construction (including extraction, language identification, deduplication, quality filtering, and annotation), and a strong emphasis on the representation of hundreds of language–script pairs, including typologically diverse and low-resource languages (Oepen et al., 2 Nov 2025).

1. Corpus Composition, Scale, and Language Diversity

Modern multilingual web pretraining corpora—such as HPLT 3.0, FineWeb2, OSCAR 22.01, and others—are primarily constructed from large-scale web crawls (notably Common Crawl and Internet Archive), often exceeding several petabytes of raw input.

• Language–script coverage: HPLT 3.0 covers ≈200 language–script pairs; FineWeb2 contains 1,868 pairs, OSCAR 202 include over 200, and WanJuanSiLu covers selected low-resource languages. Top datasets document per-language distributions, e.g., HPLT 3.0 contains 16T English tokens (55% of total), but 45% (13T) is non-English, with French, Spanish, and a range of minoritized languages explicitly tabulated (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Abadji et al., 2022).
• Token volume: State-of-the-art resources reach approximately 30 trillion tokens (HPLT 3.0), 3.3 trillion (FineWeb2, word-level), or hundreds of billions for language-specific efforts (WanJuanSiLu: 57.2B Arabic, 87B Korean, etc.) (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Yu et al., 24 Jan 2025).
• Genre and thematic diversity: Inclusion of science, news, government/legal, Wikipedia, educational, code, and parallel corpora, as well as open-domain web text, is optimized to improve cross-domain generalization and knowledge transfer.

2. Data Processing Pipelines and Filtering Strategies

The construction of multilingual web pretraining data hinges on sophisticated, multi-stage pipelines that handle language identification, deduplication, noise removal, and document-level metadata annotation.

• Language identification (LID): Tools such as OpenLID-v2, GlotLID V3, and FastText classifiers, with per-language or per-family confidence thresholds, are standard. These allow pipelines to discern noisy or code-mixed documents, and to assign appropriate language labels even in the presence of substantial linguistic noise (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Abadji et al., 2022).
• Deduplication: MinHash-based shingling identifies both exact and near-duplicate documents at petabyte scale. For example, MinHash with 5-gram windows and 14 hash bands (FineWeb2), or k-grams with multiple MinHash sketches (HPLT 3.0), typically using a Jaccard similarity threshold (e.g., τ=0.8) to cluster near-duplicates, which are then pruned (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Kumar et al., 2024).
• Heuristic and model-based filtering: Heuristics include document/line-level length, repetition, symbol proportion, stopword presence, and perplexity (e.g., PPL filtering with KenLM). More recently, model-based filters—such as lightweight regression heads trained from LLM judgements (JQL (Ali et al., 28 May 2025)), or classifier-based pipelines (FineWeb2-HQ (Messmer et al., 14 Feb 2025), MuRating (Chen et al., 2 Jul 2025))—quantify document quality. These approaches replace or complement rule-based filters, yielding better cross-lingual performance and higher retention of high-quality documents.
• Manual and semi-automatic annotations: Pipelines may annotate register (e.g., News, Blog, Encyclopedia disambiguation via Turku Register Classifier), text quality metrics (segment length, unique segment ratios), PII detection, and topic/thematic labels—with increasing frequency especially for low-resource or sensitive languages (Oepen et al., 2 Nov 2025, Yu et al., 24 Jan 2025).

3. Data Selection, Balancing, and Language Mixtures

Oversampling and under-sampling strategies are critical to avoid language dominance and ensure adequate training for tail languages.

• Sampling techniques: Temperature-based sampling $$\pi_l^{\text{temp}, \tau} = \omega_l^{1/\tau} / \sum_{l'} \omega_{l'}^{1/\tau}$$, with τ ≈ 3.3 achieving a moderate reweighting that favors low-resource languages without excessive noise amplification (Foroutan et al., 29 Oct 2025, Penedo et al., 26 Jun 2025).
• Duplication-aware rehydration: After deduplication, some pipelines use “rehydration” schemes that resample cluster representatives proportional to their cluster size and quality (e.g., $w_i$ as a monotonic function of local filtering rates, up to a maximum of 10) in FineWeb2 (Penedo et al., 26 Jun 2025).
• Pivot languages and mixture ratios: Empirical results find no inherent “curse of multilinguality” up to at least 400 languages; as long as each language is budgeted ≥1–2B tokens, there is no meaningful loss in per-language performance. English serves as a universal pivot, occasionally supplemented with family-specific pivots for extremely low-resource subgroups (Foroutan et al., 29 Oct 2025).
• Quality over size: Model-based filtering (e.g., JQL, MuRating) can match or outperform previous pipelines using a fraction of the data: e.g., matching MMLU performance with as little as 15% of tokens for major languages, or by selecting only the top 10–20% per language (Ali et al., 28 May 2025, Messmer et al., 14 Feb 2025, Chen et al., 2 Jul 2025). This demonstrates that token volume is not a sole predictor; quality and information density dominate.

4. Synthetic and Translated Data in Multilingual Pretraining

For many low-resource or medium-resource languages, high-quality native corpora are insufficient. Synthetic data generation, especially via NMT, is thus heavily leveraged.

• Large-scale translation pipelines: TransWeb-Edu and TransWebEdu showcase workflows where 100 B to 1.7 T tokens of high-quality English (FineWeb-Edu) are machine-translated into 3–9 target languages (e.g., French, German, Spanish, Indonesian, Welsh), using models such as NLLB-200 1.3B or Mistral-7B-Instruct (Wang et al., 2024, Wang et al., 18 Feb 2025).
• Balanced mixes: Entirely synthetic multilingual corpora match or exceed closed-data LLMs (e.g., Llama 3.2, Gemma) on reasoning and understanding benchmarks, even with an order-of-magnitude less pretraining data, when continued pretraining uses small language-specific or domain-specific slices (<5%) (Wang et al., 2024, Wang et al., 18 Feb 2025).
• Thematic and genre biases: Pipeline design influences domain/genre representation. Model-based filters often preferentially select “knowledge-intensive” web documents (e.g., science, society, health) while under-selecting narrative or story-rich content, which can be offset through targeted sampling or uniform mixing (Chen et al., 2 Jul 2025).

5. Quality Evaluation, Benchmarks, and Empirical Results

Rigorous evaluation protocols are integral when constructing and validating multilingual web pretraining datasets.

• Manual quality probes: Multi-language audits (e.g., 23 languages, 50–1000 docs each in HPLT 3.0) record artifact rates, unnatural text, language errors, and inappropriate content with explicit 95% confidence intervals (Oepen et al., 2 Nov 2025, Yu et al., 24 Jan 2025).
• Contrastive and statistical metrics: Unique-segment ratios, domain-diversity indices (Shannon entropy), and overrepresentation analyses (e.g., Wikipedia share per language) allow quantification of de-duplication, content diversity, and corpus “cleanliness” (Oepen et al., 2 Nov 2025).
• LLM pretraining sweeps: Direct training of 1–3 B parameter models on sampled slices (e.g., 100 B tokens per corpus; matched English/multilingual mixes) establishes relative merit. Empirical findings consistently demonstrate that cleaner, model-filtered, or rehydrated datasets yield faster convergence and greater accuracy on multilingual benchmarks (MMLU, HellaSwag, ARC, Flores, etc.) than volume-matched rule-based baselines (Oepen et al., 2 Nov 2025, Ali et al., 28 May 2025, Chen et al., 2 Jul 2025, Messmer et al., 14 Feb 2025).
• Prompting and normalization in evaluation: Advanced multilingual benchmarks (HPLT 3.0-E) use 3–7 tailored prompts per task, baseline-normalized and aggregated over tasks, to enhance ranking stability and reduce language/model prompt-sensitivity (Oepen et al., 2 Nov 2025).
• Downstream impact: On end-to-end benchmarks, model-based filtered or synthetic-augmented corpora can drive 4–7% absolute improvements in token-normalized accuracy, and outperform or match LLMs trained on closed data across a wide variety of tasks and languages (Ali et al., 28 May 2025, Chen et al., 2 Jul 2025, Wang et al., 18 Feb 2025).

6. Dataset Release, Reproducibility, and Open Practices

Recent efforts prioritize open distribution, reproducibility, extensibility, and ethical considerations.

• Open pipelines and licensing: HPLT 3.0, FineWeb2, JQL, WanJuanSiLu, Lucie-7B, and others publish full code, metadata, and data via platforms such as Hugging Face, with open or ODC-By licenses and rich aquisition and filtering logs (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Ali et al., 28 May 2025, Yu et al., 24 Jan 2025, Gouvert et al., 15 Mar 2025).
• Metadata provision: Released datasets contain not just text but URL/crawl provenance, annotation tags (text quality, language distribution, register, PII flags), statistical metrics (unique/total segment ratio, domain entropy), and thematic labels for granular downstream sampling and auditing (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Yu et al., 24 Jan 2025).
• Extensibility and transparency: Most pipelines permit per-language customization, modular filter swapping, and the rapid addition of new languages or new automatic quality criteria. Some, such as JQL, offer hooks for new code/toxicity/factuality judges by simply swapping annotation prompts (Ali et al., 28 May 2025).

7. Best Practices and Open Problems

• Document-level orientation: Operating at the document level (rather than lines/fragments) is best practice to preserve context and optimize for language modeling objectives (Abadji et al., 2022).
• Dynamic selection: Temperature-based and duplication-aware sampling, as well as per-language automatic threshold tuning, are necessary to avoid language/genre drift in very large, highly multilingual mixtures (Foroutan et al., 29 Oct 2025).
• Model-based and embedding-based filtering: State-of-the-art pipelines such as MuRating, JQL, and FineWeb2-HQ demonstrate that compact, cross-lingual embedding backbones and lightweight heads yield scalable and transferable quality assessment across scripts and resource conditions (Chen et al., 2 Jul 2025, Ali et al., 28 May 2025, Messmer et al., 14 Feb 2025).
• Synthetic translation at scale: When high-quality native data is impractical for many languages, translation-based synthetic data is empirically effective. Translation fidelity and domain/genre bias require careful monitoring, but BLEU and cross-lingual benchmarks confirm strong alignment for most high- and mid-resource languages (Wang et al., 2024, Wang et al., 18 Feb 2025, Seto et al., 2024).
• Open evaluation and "canary tasks": Early-signal multilingual benchmarks are critical for pipeline tuning, especially in the absence of large, standardized evaluation suites for many non-English languages (Penedo et al., 26 Jun 2025).

Multilingual web pretraining data now forms the technical backbone of broad-coverage LLMs, supporting typologically and geographically diverse research and applications. State-of-the-art pipelines integrate scalable extraction, advanced filtering, dynamic upsampling, synthetic/translated augmentation, and rigorous evaluation, marking a transition toward open, high-quality, and deeply annotated resources for the next generation of LLMs (Oepen et al., 2 Nov 2025, Penedo et al., 26 Jun 2025, Ali et al., 28 May 2025, Chen et al., 2 Jul 2025, Wang et al., 2024, Wang et al., 18 Feb 2025, Abadji et al., 2022, Gouvert et al., 15 Mar 2025, Kumar et al., 2024, Yu et al., 24 Jan 2025).