Scaling Multilingual Curation and Its Interaction with General-Purpose Mixtures

Determine how multilingual data curation strategies scale to larger token budgets and model sizes, and characterize how multilingual data curation interacts with general-purpose pretraining mixtures in large-scale language model training.

Background

Earlier sections present smaller-scale controlled bilingual experiments (60B tokens, 3B-parameter models) demonstrating that targeted data curation improves cross-lingual transfer and multilingual performance.

The authors note that understanding how these multilingual curation strategies extend to larger token budgets and larger model sizes, and how multilingual curation interacts with a broader general-purpose mixture (including English, code, STEM, and reasoning data), is a critical question for practical large-scale pretraining. They subsequently describe a 20T-token curated corpus and 1T-token training runs as an initial step toward addressing this question.

References

However, an open question is how such curation strategies can scale to larger token budgets and models, and how multilingual curation interacts with general purpose curation.

— ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset  (2602.15210 - DatologyAI et al., 16 Feb 2026) in Subsection 'Integrating multilingual curation into a general pretraining mix' (Section 'Main Findings'; label: section:big_boi_exps)

Although the discovered recipes transfer across model scales within this setting, their generalization to other corpora, domains, and larger model scales remains an open question.

— AutoData: Agentic Search for Pre-training Data Selection  (2609.19754 - Meng et al., 17 Sep 2026) in Section Limitations

It remains to be replicated on web data and at other data sizes.

— The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype  (2609.39001 - Serval, 30 Sep 2026) in Section 5, paragraph “Controlled experiment”

Our study is limited to only three languages (English, Dutch, and Chinese) within one domain (Wikipedia), thus it is uncertain whether our findings will apply to other languages or domains.

— Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training  (2610.20102 - Hwang et al., 14 Oct 2026) in Limitations