Synthetic Multilingual Pretraining Data

Updated 20 November 2025

Synthetic multilingual pretraining data is algorithmically generated text designed to augment language models with diverse multilingual content, especially for low-resource languages.
It employs techniques like machine translation, LLM-based generation, and code-switching to construct large-scale, parallel and monolingual corpora.
Robust quality control measures such as perplexity filtering, deduplication, and balanced data mixing are critical for ensuring effective model pretraining.

Synthetic multilingual pretraining data refers to textual resources for LLM pretraining that are algorithmically generated, rather than directly collected from natural human communication. These resources are structurally constructed to provide multilingual coverage, often aiming to overcome the inherent scarcity or imbalance of high-quality monolingual or parallel corpora, especially for low-resource languages. Synthetic data can be created through a variety of techniques, such as machine translation (MT), LLM generation, code-switching augmentation, rephrasing pipelines, or even fully artificial language constructions. Their integration has become central to scaling LLMs and addressing global linguistic coverage gaps.

1. Methodologies for Generating Synthetic Multilingual Data

Synthetic multilingual pretraining data is produced using several principal methodologies, each with structural variants and distinctive quality-control pipelines:

Machine Translation: High-resource language corpora are translated into target languages using neural MT models, either at document or sentence granularity. The HPLT 3.0 initiative generates synthetic parallel corpora from 128B English tokens, using Marian NMT (OPUS-MT) with beam search decoding, and produces ~200M sentence pairs across 36 language variants (Oepen et al., 2 Nov 2025). Similar approaches translate FineWeb-Edu into multiple target languages, leveraging NLLB-200 or Mistral-7B-Instruct for up to 1.7T tokens in TransWebEdu (Wang et al., 18 Feb 2025, Wang et al., 31 Oct 2024).
LLM-based Generation: Large multilingual LLMs are prompted to produce monolingual or parallel texts natively in the target language, conditioned on structured prompts, personas, or topic retrievals. For Indic languages, BhashaKritika synthesizes 540B tokens with strategies such as document-grounded, persona-based, and math/reasoning grounding, using Krutrim-2, Gemma-3, Llama-3.3, and Sarvam-Translate as the generative backbone (Manoj et al., 13 Nov 2025).
Synthetic Code-Switching: Code-mixing is algorithmically injected at sentence or token level, using translation and distilled LLMs, as in SynCS, which applies four types of annotation and replacement for inter/intra-sentential switches (Wang et al., 2 Apr 2025). PreAlign further introduces input-only code-switching to reinforce early lexical alignment (Li et al., 23 Jul 2024).
Rephrasing Pipelines: Existing corpora are passed through paraphrasing LLMs (e.g., Mistral 7B, Qwen2-Instruct), using specially designed prompt templates for QA, narrative, or Wikipedia styles. The output is synthetically rephrased monolingual or multilingual text, often mixed with the original corpus at 1:1 or tuned ratios (Pieler et al., 28 Oct 2024).
Synthetic Parallelism without Human Language: Fully artificial languages (obfuscated, phrase-concatenated, randomly permuted) are devised, testing the degree to which lexical or structural scaffolding alone aids cross-lingual transfer in NMT settings (He et al., 2022).
Prompt-Space Optimization: Translated task prompts are transformed before sampling completions, using teacher LLMs to introduce naturalness, cultural adaptation, and difficulty enhancements, reshaping the training distribution for improved multilingual robustness (Mora et al., 22 Oct 2025).
LLM-Generated Retrieval Datasets: In IR, synthetic passage-query pairs are generated across 33 languages via summarize-then-ask prompting, using few-shot LLM instruction for content-grounded, language-specific question generation (SWIM-IR) (Thakur et al., 2023).

2. Design, Quality Control, and Filtering Techniques

Quality control in synthetic resource construction is multifaceted, combining model-based, heuristic, and metadata-based filtering to ensure pretraining corpora maximize downstream value while minimizing artifacts.

Perplexity-Driven Filtering: TinyLMs or KenLM n-gram models, trained on a small real language seed corpus, assign PPL scores to synthetic outputs; high PPL documents are filtered out. This recovers >50% of the performance gap between noisy synthetic and clean data (Doshi et al., 20 Mar 2024, Manoj et al., 13 Nov 2025).
Deduplication: Global MinHash + locality-sensitive hashing (LSH) is deployed to remove near-duplicate documents at scale, important both before and after synthesis (Oepen et al., 2 Nov 2025). Fuzzy deduplication is also applied to translationese corpora using NeMo-Curator (Joshi et al., 18 Oct 2024).
Script and Language ID Consistency: Language identification ensembles ensure samples are in the intended language and script, essential for Indic, Cyrillic, and similar contexts (Manoj et al., 13 Nov 2025).
Heuristic and Content Filters: These include document length, repetition (n-gram ratio), NSFW or AI word checks, and stop-word thresholds (Manoj et al., 13 Nov 2025).
Human or LLM Evaluations: Some pipelines employ LLMs for quality classification, or limited human assessments for translation/faithfulness (Manoj et al., 13 Nov 2025).
Bias Detection: Corpus-level bias statistics (e.g., WEAT for gender/religion) are computed and mitigated by targeted data augmentation (Manoj et al., 13 Nov 2025).
Prompt Transformation and Selection: For LLM-generated synthetic tasks, prompt naturalization (fixing translationese), cultural adaptation, and difficulty tuning precede completion sampling, with post-hoc language ID filtering for prompt drift (Mora et al., 22 Oct 2025).

Pipeline/Filter	Core Mechanism	Papers
MinHash Deduplication	LSH on document shingles	(Oepen et al., 2 Nov 2025)
TinyLM PPL Filtering	LM-based sequence scoring	(Doshi et al., 20 Mar 2024)
KenLM N-gram Filtering	Per-language fluency check	(Manoj et al., 13 Nov 2025)
Prompt Optimization	LLM-based prompt rewriting	(Mora et al., 22 Oct 2025)
Language ID Filtering	Ensemble LID classifiers	(Manoj et al., 13 Nov 2025)

Quality filtering is highly impactful: for instance, applying a TinyLM PPL filter to translationese in Hindi improves NLU/NLG metrics from –3.56% to –1.54% vs clean data, and an additional 10% continued training on real data closes the gap almost entirely (Doshi et al., 20 Mar 2024).

3. Strategic Data Mixing and Curriculum Integration

Blending synthetic data with real corpora or other synthetic resource types is accomplished through carefully designed sampling and curriculum strategies:

Token Budget Partitioning: Frequently, tokens from real and synthetic sources are mixed at preset ratios, e.g. 5% synthetic parallel, 95% monolingual, as a curriculum ramp in HPLT 3.0 (Oepen et al., 2 Nov 2025), or 80:20 synthetic:real for Hindi in Nemotron-Mini-Hindi-4B (Joshi et al., 18 Oct 2024).
Batch-level Sampling: Equal proportions or batch-level oversampling of under-represented real language data mitigates synthetic noise (Joshi et al., 18 Oct 2024).
Curriculum Scheduling: The share of synthetic data may be linearly increased across training epochs, encouraging sustained cross-lingual alignment (Oepen et al., 2 Nov 2025).
Domain/Style Diversification: Synthetic datasets are built using arrays of prompts or model architectures to maximize stylistic and factual coverage (e.g., persona, document-grounded, math/reasoning, retrieval-augmented) (Manoj et al., 13 Nov 2025).

Quantitative ablations consistently indicate that optimal performance in low-resource settings is achieved by maintaining a nonzero fraction (often ≥20%) of real language data in the mix, while high synthetic ratios are effective in expanding language coverage rapidly when little real data exists (Joshi et al., 18 Oct 2024, Doshi et al., 20 Mar 2024).

4. Empirical Impact and Evaluation

Comprehensive evaluation across LLM pretraining and MT/NLU/NLG downstream tasks confirms the utility of synthetic multilingual data:

Zero-shot and Few-shot Task Performance: Benchmarks such as MMLU, XNLI, IndicXTREME, HellaSwag, and Global-MMLU are used to measure gains. Nemotron-Mini-Hindi-4B demonstrates +12% F1 on IndicSentiment and +11.7% on IndicCopa (base model, pretrain+synthetic), consistently outperforming synthetic-only and real-only regimes (Joshi et al., 18 Oct 2024). In multilingual LLMs pretrained only on synthetic MT corpora (CuatroLLM, TransWebLLM), models match or exceed the accuracy of Llama3.2 and Gemma2, despite using a fraction (6–25%) of the training data (Wang et al., 31 Oct 2024, Wang et al., 18 Feb 2025).
Translation and NLU Metrics: Cross-lingual BLEU (e.g. 25.7 on WMT14 EN↔FR, 27.6 on WMT16 EN↔DE with synthetic-only data, no explicit parallel training) and ROUGE-L F1 for generative tasks are boosted by synthetic augmentation (Wang et al., 31 Oct 2024, Doshi et al., 20 Mar 2024).
Data Efficiency: As little as 100M synthetic code-switch tokens can yield the transfer benefits of 2B monolingual tokens (20× efficiency), notably in code-switching augmentation (Wang et al., 2 Apr 2025).
Structure and Diversity Effects: Rephrasing noisy web corpora and mixing 1:1 with original data produces large performance gains for low-quality and non-English text subsets (~+3.7 pp on CulturaX-G, +1.5–2.5 pp CX-all), but offers diminishing returns as base quality increases (Pieler et al., 28 Oct 2024).
Bias and Fluency Trade-offs: Document/prompt grounding, robust deduplication, fluency scoring, and bias mitigation reliably improve the human- and automatic-assessed output quality, critical for language-specific and multicultural benchmarks (Manoj et al., 13 Nov 2025).
Instruction Tuning and Retrieval: Synthetic datasets constructed for dense retrieval (SWIM-IR) or instruction fine-tuning (e.g., using optimized prompt strategies) can outperform human-annotated benchmarks in cross-lingual IR (e.g., +7.1 R@5kt over supervised baseline on XOR-Retrieve) (Thakur et al., 2023, Mora et al., 22 Oct 2025).

Setting	Result	Reference
Syn. only LLM (CuatroLLM, French)	38.5% (vs. 39.9% SOTA)	(Wang et al., 31 Oct 2024)
80:20 synth:real Hindi LLM (IndicSent.)	F1 84.31 (+12%)	(Joshi et al., 18 Oct 2024)
Code-switch SynCS, Chinese HellaSwag/ARC	39.99% (Sent-Repl., S=300M)	(Wang et al., 2 Apr 2025)
Persona+doc ground. Indic (BhashaKritika)	540B-token, parity+ on XNLI	(Manoj et al., 13 Nov 2025)
Title-rephrased Oscar (German)	+3.7 pp (over baseline)	(Pieler et al., 28 Oct 2024)

5. Limitations, Challenges, and Best Practices

While synthetic multilingual pretraining data has catalyzed rapid improvements across languages and tasks, several challenges remain:

Translationese and Surface Artifacts: Synthetic translations often embed non-native structures (“translationese”), limiting the model’s naturalness and cultural alignment. Prompt-space optimization (e.g. T_N, T_C, T_D transformations) demonstrably mitigates these effects, improving accuracy by up to 4.7 pp on G-MMLU (Mora et al., 22 Oct 2025).
Quality Bottlenecks in Low-resource MT: MT-based synthesis inherits the errors and deficiencies of the underlying NMT models, especially for truly low-resource languages. Reliance on real data seeds for filtering and continued pretraining is essential (Doshi et al., 20 Mar 2024).
Domain and Style Narrowness: Synthetic corpora built from narrow or monolithic source domains (e.g., educational-only FineWeb-Edu) risk domain overfitting. Diversification through prompt/grounding variety and short “cooldown” phases with targeted data are recommended (Wang et al., 31 Oct 2024, Manoj et al., 13 Nov 2025).
Resource and Compute Costs: Rephrasing, translation, or LLM-driven generation pipelines are computationally and economically intensive, though less so than manual annotation at comparable scale (Thakur et al., 2023).
Tokenization and Vocabulary: Empirical analyses show that subword vocabulary coverage for synthetic corpora often exceeds 98% when leveraging large, open-base BPE models (e.g., Llama 2’s 32k vocab), minimizing the need for per-language vocabularization (Wang et al., 31 Oct 2024, Wang et al., 18 Feb 2025).
Mixing Ratios: Over-reliance on pure synthetic data can degrade perplexity and fluency; optimal performance typically requires strategic mixing (e.g., 80:20 synthetic:real for low/resource, 1:1 original:rephrased for noisy data) (Joshi et al., 18 Oct 2024, Pieler et al., 28 Oct 2024).
Evaluation Limitations: Benchmarking must cover a wide, multilingual task suite; some gains remain inconclusive, especially for high-resource languages or fine-tuned SFT results (Pieler et al., 28 Oct 2024).

Consensus best practices include:

Always ground generation in real documents/personas/topics and select prompts in a language- and domain-matched fashion (Manoj et al., 13 Nov 2025).
Deduplicate across all data sources globally, especially when blending monolingual, synthetic, and web-mined corpora (Oepen et al., 2 Nov 2025).
Filter noisy outputs by PPL and language ID, and enforce target-script and content heuristics, particularly for synthetic data in low-resource languages (Manoj et al., 13 Nov 2025).
Mix original and synthetic data at empirically tuned ratios, avoiding overcommitment to synthetic-only regimens (Joshi et al., 18 Oct 2024).

6. Extensions: Fully Synthetic, Obfuscated, and Unsupervised Data Approaches

Alternative regimes for constructing synthetic multilingual data decouple structural signals, privacy requirements, or resource constraints:

Obfuscated and Artificial Corpora: Pseudorandom vocabularies or phrase concatenations preserve only structural and alignment cues, susceptible to high privacy and low toxicity settings, often with only a minor BLEU penalty (1–4 points at R=75% obfuscation) (He et al., 2022).
Unsupervised Parallelism: Back-translation, denoising autoencoding, and margin-based mining of synthetic parallel pairs enable unsupervised alignment for languages lacking any parallel resource. Fine-tuning XLMs on such data closes 14–22 F1 points of the gap to supervised performance (Kvapilıkova et al., 2021).
LLM-based Synthetic Retrieval: Modular LLM prompting (summarize-then-ask) creates competitive retrieval datasets in dozens of languages, matching or exceeding supervised dense retrieval (Thakur et al., 2023).

These strategies demonstrate that much of the transferable scaffolding necessary for LLM alignment, translation, or cross-lingual NLU can be synthesized without large real-world corpora, especially when coupled with strong alignment or filtering protocols.

In summary, synthetic multilingual pretraining data now constitutes a foundational strategy for large-scale language modelling, especially for under-represented languages. MT, LLM generation, code-switching, rephrasing, obfuscation, and advanced prompt-oriented recipes are all deployed at web and trillions-of-token scale. Their effective use hinges on robust quality filtering, deduplication, balanced mixing, and continuous benchmarking, providing a path toward globally robust, culturally adaptive, and resource-efficient LLMs (Oepen et al., 2 Nov 2025, Joshi et al., 18 Oct 2024, Wang et al., 31 Oct 2024, Wang et al., 18 Feb 2025, Manoj et al., 13 Nov 2025, Wang et al., 2 Apr 2025, Doshi et al., 20 Mar 2024, Pieler et al., 28 Oct 2024, He et al., 2022, Mora et al., 22 Oct 2025, Thakur et al., 2023, Kvapilıkova et al., 2021, Li et al., 23 Jul 2024).