ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality (2510.22037v1)

Abstract: Scaling laws research has focused overwhelmingly on English -- yet the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 multilingual training experiments, spanning 10M-8B model parameters, 400+ training languages and 48 evaluation languages. We introduce the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual pretraining, which outperforms existing scaling laws' out-of-sample generalization often by more than 0.3 R^2. Our analyses of the experiments shed light on multilingual learning dynamics, transfer properties between languages, and the curse of multilinguality. First, we derive a cross-lingual transfer matrix, empirically measuring mutual benefit scores between 38 x 38=1444 language pairs. Second, we derive a language-agnostic scaling law that reveals how to optimally scale model size and data when adding languages without sacrificing performance. Third, we identify the computational crossover points for when to pretrain from scratch versus finetune from multilingual checkpoints. We hope these findings provide the scientific foundation for democratizing scaling laws across languages, and enable practitioners to efficiently scale models -- beyond English-first AI.

• The paper introduces the Adaptive Transfer Scaling Law (ATLAS) to model multilingual pretraining by accounting for data repetition and cross-lingual transfer.
• It empirically validates scaling dynamics across 400+ languages with high R² values while quantifying capacity constraints responsible for the curse of multilinguality.
• The study offers actionable tools for choosing between pretraining from scratch and finetuning through detailed analyses of transfer matrices and compute budgets.

ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

Introduction and Motivation

The ATLAS framework addresses critical gaps in scaling law research for multilingual LLMs (MLLMs). While previous scaling laws have focused primarily on English, ATLAS systematically investigates scaling dynamics across 400+ languages, leveraging 774 training experiments and 48 evaluation languages. The work introduces the Adaptive Transfer Scaling Law (ATLAS), which models both monolingual and multilingual pretraining, explicitly accounting for cross-lingual transfer and the curse of multilinguality—where adding languages can degrade per-language performance due to limited model capacity.

Adaptive Transfer Scaling Law: Formulation and Empirical Fit

ATLAS generalizes prior scaling laws by introducing a repetition-aware, transfer-adaptive formulation. The loss for a target language $t$ is modeled as:

$$\mathcal{L}(N, \mathcal{D}_{\mathrm{eff}}) = E + \frac{A}{N^\alpha} + \frac{B}{\mathcal{D}_{\mathrm{eff}}^\beta}$$

where $\mathcal{D}_{\mathrm{eff}}$ is a weighted sum of target language data, transfer language data, and other language data, each passed through a saturation function to account for data repetition. Transfer coefficients $\tau_i$ are initialized from empirically measured cross-lingual transfer scores.

Empirical results demonstrate that ATLAS achieves superior generalization across held-out axes—model size ($N$), token count ($D$), compute ($C$), and unseen language mixtures ($M$)—with $R^2$ values frequently exceeding 0.9 in multilingual settings. Notably, only by including explicit transfer terms does ATLAS outperform prior multilingual scaling laws in mixture generalization.

Figure 1: Optimal scaling trajectories for six languages under different vocabulary and training regimes, revealing compute efficiency tax and data repetition effects.

Cross-Lingual Transfer: Measurement and Implications

ATLAS provides the most comprehensive empirical resource to date for cross-lingual transfer, measuring mutual benefit scores between 38 languages and constructing a $38 \times 38$ transfer matrix. The Bilingual Transfer Score (BTS) quantifies the relative training efficiency of bilingual models compared to monolingual baselines. Positive scores indicate beneficial transfer, while negative scores indicate interference.

Figure 2: Cross-lingual transfer matrix for 30 languages, highlighting top-5 source languages for each target and the predictive role of language similarity.

Analysis reveals that language pairs sharing both family and script exhibit strong, symmetric transfer, while greater linguistic distance correlates with increased asymmetry and reduced positive transfer.

Figure 3: Left: Transfer score symmetry scatter plot; Right: Impact of linguistic similarity on transfer scores, with statistically significant differences ($p < .001$).

Script similarity is a stronger predictor of positive transfer than family alone, suggesting that shared subword representations are a primary mechanism for transfer. Transfer scores are not generally reciprocal except within closely related pairs, corroborating findings that altruistic languages do not always yield mutual benefits.

The Curse of Multilinguality: Capacity Constraints and Scaling

ATLAS empirically quantifies the curse of multilinguality by measuring the degradation in target language loss as the number of training languages increases. The loss penalty is most pronounced for small models and declines with increased capacity.

Figure 4: Relative loss degradation as a function of model size and number of training languages, showing mitigation with larger models and more data.

The scaling law for multilingual capacity is:

$$L(K, N, D_t) = L_\infty + A \frac{K^\phi}{N^\alpha} + B \frac{K^\psi}{D_t^\beta}$$

where $K$ is the number of languages, $\phi$ captures capacity-driven loss increase, and $\psi < 0$ indicates positive transfer (sublinear data needs per language as $K$ increases). Empirical fits yield $\phi = 0.11$ and $\psi = -0.04$, indicating a mild curse tempered by positive transfer.

Practitioners can compute iso-loss frontiers and derive closed-form scaling equations for model size and data when expanding language coverage:

$$\frac{N^*(rK)}{N^*(K)} = r^{\phi/\alpha}, \quad \frac{D_t^*(rK)}{D_t^*(K)} = r^{\psi/\beta}, \quad \frac{D_{\mathrm{tot}}^*(rK)}{D_{\mathrm{tot}}^*(K)} = r^{1+\psi/\beta}$$

Pretraining vs. Finetuning: Compute-Efficiency Tradeoffs

ATLAS resolves a practical question: when is it more efficient to pretrain from scratch versus finetune from a multilingual checkpoint? Loss curves for 2B parameter models show that finetuning from a multilingual base is preferable for compute budgets below 144B tokens, while pretraining from scratch surpasses finetuning for budgets above 283B tokens.

Figure 5: Loss curves for pretraining from scratch vs. finetuning from Unimax base, with annotated crossover points for eight languages.

The compute budget required for pretraining to outperform finetuning scales as $\log(C) = 1{,}113{,}708 \times N^{1.65}$, providing a practical heuristic for model developers.

Extended Analysis: Transfer Dynamics at Scale

Further analysis demonstrates that cross-lingual transfer dynamics are established early in training and remain stable with more data, but larger models are more effective at mitigating negative transfer for challenging pairs.

Figure 6: Left: Transfer score as a function of training tokens; Right: Transfer score as a function of model size.

The full $38 \times 38$ transfer matrix further substantiates the predictive role of language similarity and provides actionable guidance for mixture selection.

Figure 7: Full language transfer heatmap, with top-5 source languages bolded for each target.

Conclusion

ATLAS advances the state of scaling law research for MLLMs by introducing a repetition-aware, transfer-adaptive law that robustly models multilingual pretraining and finetuning. The empirical transfer matrix and capacity scaling law provide actionable tools for practitioners to optimize model scaling, mixture selection, and compute allocation. The findings highlight the importance of explicit modeling of cross-lingual transfer and capacity constraints, and suggest that future MLLM development should leverage these insights to democratize high-quality language modeling across diverse linguistic communities. The methodology and resources established by ATLAS will inform both theoretical research and practical deployment of multilingual foundation models.