--- title: Proportion-Dependent Multilingual Scaling Law url: https://www.emergentmind.com/topics/proportion-dependent-multilingual-scaling-law type: topic --- # Proportion-Dependent Multilingual Scaling Law A Proportion-Dependent Multilingual Scaling Law (PDMSL) characterizes how the performance of large models trained on data spanning multiple languages (or language families) systematically varies as a function of the allocation proportions, model/data scale, and, in advanced formulations, the transfer/synergy between languages. These laws enable precise prediction of per-language or aggregate performance at arbitrary scales and corpus compositions, provide guidance for optimal data mixture design, and illuminate the limits and emergent properties of cross-lingual generalization. ## 1. Formal Definitions and Universal Formulations Proportion-dependent scaling laws extend classic power-law scaling (e.g., Chinchilla) by introducing explicit dependence on the sampling proportions allocated to each language or group. A generic law for test loss $L_f$ of language (or family) $f$, for non-embedding model size $N$, total data $D$, and $p_f$ = proportion of tokens from $f$, is given by: \[ L_f(N, D, p_f) = \left(E_f + \frac{A_f}{N^{\alpha_f}} + \frac{B_f}{D^{\beta_f}}\right) p_f^{-\gamma_f} \] where $E_f$, $A_f$, $B_f$, $\alpha_f$, $\beta_f$, and $\gamma_f$ are empirically fitted per family or language. The exponent $\gamma_f > 0$ controls the sensitivity of loss to the allocation proportion $p_f$; $p_f$ is estimated as the fraction of tokens sampled for family $f$ out of the total $D$. Multiple studies (multilingual language modeling [2410.12883], code LLMs [2512.13472], NMT [2302.09650], reasoning models [2510.02272]) empirically validate this structure, with variations in details reflecting differing degrees of cross-lingual transfer and model domain. ## 2. Family- and Language-Wise Laws and Optimal Allocation A key empirical observation is that, when languages are coherently grouped (e.g., by linguistic family), the loss for each group often depends only on its own proportion, not that of unrelated groups. Under this minimal transfer regime, each $L_f$ decouples, allowing for straightforward prediction and optimization: \[ L_\text{tot} = \sum_{f=1}^n w_f L_f(N,D,p_f) \] The optimal allocation problem (given weights $w_f$ for group importance) is: \[ \min_{p \in \Delta_n}\; L_\text{tot} = \sum_{f=1}^{n} w_f L_f^\star(N,D)\,p_f^{-\gamma_f} \quad \text{s.t.} \sum_{f} p_f = 1 \] For small $\gamma_f$, the weights that minimize the loss under resource constraints are: \[ p_f^* \approx \frac{w_f L_f^\star(N,D) \gamma_f}{\sum_{i} w_i L_i^\star(N,D) \gamma_i} \] This formula is robust across model scales, enabling the use of optimal sampling ratios derived from small models in large-scale pretraining [2410.12883]. If $w_f = 1/L_f^\star$, then $p_f^* \propto \gamma_f$ and is scale-invariant. ## 3. Empirical Results and Implications ### Table: Empirical Exponents and Impact Across Domains | Domain | Law Structure | Key Exponents/Synergy | Notable Findings | |-----------------------------------|----------------------------------------------------------|-----------------------|---------------------------------------------------------------------------| | Multilingual LMs [2410.12883] | $L_f = L_f^\star \, p_f^{-\gamma_f}$ | $\gamma_f\sim0.05-0.1$| Proportion-independent exponents, optimal $p_f^*$ from small models | | Multilingual Code LLMs [2512.13472]| $\mathcal{L}(N,D;p) = A N^{-\alpha_N(p)} + B D_x^{-\alpha_D(p)} + L_\infty(p)$ | $\alpha_N^k$, $\alpha_D^k$, $\tau_{ij}$ (synergy) | Token allocation guided by per-language exponents and pairwise synergy | | NMT [2302.09650] | $L_i(N;w_i) = C_i(w_i) N^{-\alpha_i} + L_\infty^{(i)}$ | $C_i(w_i)\propto w_i^{-\gamma_i}$ | Same $\alpha_i$ for all mixes, $f_i(w_i)$ gives effective capacity split | | Reasoning LRM [2510.02272] | $P(X) = \alpha X^{\beta}$ | $\beta=0.29$ (MTI) | Marked "First-Parallel Leap" with diminishing marginal returns | | Speech (ASR/ST) [2502.10373] | $Y(N,D,C) = c + A_N N^{-\alpha} + A_D D^{-\beta} + A_C C^{-\gamma}$ | $\alpha\sim0.19$ | Scaling aids low-resource languages; no explicit $p$-dependence modeled | Empirical studies confirm that allocating a higher $p_f$ enhances loss reduction for that family, but with strongly sublinear effects due to small $\gamma_f$. Crucially, allocating tokens using the computed $p_f^*$ (rather than uniform or raw-data-proportional) yields measurable aggregate gains and fairness, especially as family-level $\gamma_f$ is easy to estimate in small-scale pilots [2410.12883]. In code LLMs, cross-lingual synergy is explicitly modeled by augmenting the data budget $D_x$: \[ D_{x} = D_{\rm all}\,\left(1 + \gamma\sum_{i\neq j} p_i p_j \tau_{ij} \right) \] Positive $\tau_{ij}$ (e.g., Java/C# or JavaScript/TypeScript) amplifies the benefit of proportionally increasing both languages. Such second-order corrections are validated by direct experiment [2512.13472] and outperform uniform allocations under fixed compute. In NMT [2302.09650], mixture effects manifest almost purely in the coefficient, with the same power-law exponent $\alpha_i$ regardless of $w_i$. Effective capacity splits are accurately predicted via $f_i(w_i) = N_\text{eff}^{(i)} / N$. Directional effects persist, with significant positive synergy for many-to-English setups and neutrality for English-to-many. ## 4. The "First-Parallel Leap" and Monolingual Generalization Gap In LRM cross-lingual reasoning [2510.02272], the transition from monolingual to even a single additional parallel language produces a disproportionate gain ("First-Parallel Leap"). For example, transferability (MTI) jumps from 1.16 (X=1) to 2.50 (X=2), greatly exceeding the incremental benefits per added language thereafter. Simultaneously, the monolingual generalization gap is defined as the shortfall between the observed monolingual metric and its power-law extrapolation from the multilingual regime: \[ \text{Gap}_t = P_t(1)_{\mathrm{predicted}} - \mathrm{MTI}_{\mathrm{actual}} \] This gap quantifies suboptimal transfer from monolingual-only training and persists across both accuracy and generalization metrics. The phenomenon is robust to task and model, indicating a structural limitation of monolingual pretraining with respect to cross-lingual reasoning. ## 5. Incorporating Cross-Lingual Transfer and Synergy Advanced scaling laws such as ATLAS [2510.22037] and the code LLM law [2512.13472] explicitly inject language-pair transfer through empirical transfer matrices or synergy coefficients. In ATLAS, the effective data budget is constructed as a sum over target, top-k transfer languages, and the remaining languages, weighted by learned $\tau_\ell$ and saturated for repeated tokens: \[ \mathcal{D}_{\rm eff} = \sum_{\ell} \tau_\ell \mathcal{S}_\lambda(D_\ell; U_\ell) \] Transfer matrices—empirically measured, e.g., via BTS (Bilingual Transfer Score) in ATLAS—guide both initial $\tau_\ell$ assignment and optimal selection of $\mathcal{K}_t$ (high-benefit transfer groups). These refinements yield greatly improved and robust held-out $R^2$ (e.g., $R^2=0.89$ for model size, $R^2=0.96$ for data, $R^2=0.82$ for unseen mixtures) far surpassing classic Chinchilla-style monolingual or uniform-multilingual laws [2510.22037]. ## 6. Practical Guidelines and Limitations Practical design of pretraining mixtures involves: - Estimating per-family or per-language $\gamma_f$ (or Chinchilla exponents for code); - Computing $p_f^*$ for target resource and model scales; - Adjusting raw allocations to encourage synergy between high-transfer pairs; - Validating performance under constraints (e.g., held-out loss, fairness objectives). Scaling laws delivered from small models generalize with high fidelity to much larger scales [2410.12883], allowing rapid prototyping and efficient resource investment. However, limitations include: - Assumptions of negligible cross-family transfer (violated for linguistically incoherent groupings) [2410.12883]; - Static corpus composition—adaptive, curriculum, or staged sampling is not addressed; - Synergy and transfer coefficients [2512.13472], [2510.22037], are corpus- and architecture-dependent and may not extrapolate to domain-specific or low-resource languages without recalibration; - For vision-language or speech settings, results demonstrate classic scaling but lack finely resolved $p$-dependent experiments [2503.09443], [2502.10373]. ## 7. Broader Impact, Generalization, and Frontier Directions Proportion-dependent multilingual scaling laws establish a robust theoretical and empirical foundation for principled multilingual model construction. They unify the optimization of model/data scale with data mixing, span domains (text, code, speech, and reasoning), and imbue the model design process with predictable trade-offs: - Raising representation for high-utility or underrepresented groups while avoiding over-investment in redundancy (fast-saturating languages); - Harnessing and quantifying cross-lingual transfer for maximum resource efficiency; - Quantitatively illuminating the phenomenon and magnitude of transfer asymmetries, fairness gains for low-resource contexts, and emerging capabilities at scale. A plausible implication is that future extensions will integrate dynamic or task-adaptive data allocation, further refine transfer/synergy modeling, and generalize to multi-modal or multi-task settings with proportionally resolved scaling, catalyzing continued advances in equitable and efficient multilingual AI. --- **Key References:** [2510.02272] (Parallel Scaling Law for LRMs), [2410.12883] (Multilingual LM Scaling), [2512.13472] (Multilingual Code LLMs), [2302.09650] (Multilingual NMT Scaling), [2510.22037] (ATLAS cross-lingual scaling), [2502.10373] (Multilingual Speech Scaling). Source: https://www.emergentmind.com/topics/proportion-dependent-multilingual-scaling-law