Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Abstract: For most languages of the world, LLM pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new knowledge. Both offer a principled way to improve training in a data-constrained domain. We compare these levers systematically across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Three findings emerge. First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, we quantify how much mixing helps: it boosts performance by an amount equivalent to 2--3$\times$ the unique target data on validation loss and 2--13$\times$ on downstream task accuracy, with the gain scaling steeply with model size. Third, this divergence reveals that target-language validation loss systematically underestimates mixing's value. Mixing regularizes by diversifying the training signal and contributes knowledge the repeated target corpus cannot supply; validation loss captures only the first effect. Our practical recommendations are: mix in a high-resource language, prioritize the mixing ratio over hyperparameter tuning, and transfer hyperparameters from a small proxy model via $μ$P.

• The paper finds that bilingual pre-training using mixed high- and low-resource data yields significant improvements in validation loss and task accuracy compared to hyperparameter tuning.
• With larger models, the optimal mixing ratio shifts toward more high-resource English data, achieving up to a 13.4x data equivalence in downstream benchmarks.
• While Arabic validation loss reliably selects near-peak checkpoints, downstream evaluations reveal that mixing provides additional transfer benefits beyond mere regularization.

Bilingual Pre-Training Versus Hyperparameter Search in Data-Constrained LLM Regimes

Motivation and Background

Modern pre-training practices for LMs increasingly encounter data-constrained settings, particularly for low-resource languages where compute budgets outpace the available high-quality corpus. Model repetition on limited data induces overfitting, deteriorating generalization and shifting representational capacity toward memorization. Practitioners typically counteract this degradation by aggressive HP tuning (notably elevated weight decay) or by training with a mixture of target (low-resource, LR) and auxiliary (high-resource, HR) language data, the latter diversifying the input distribution and regularizing learning.

Previous studies, e.g., [muennighoff2025scalingdataconstrainedlanguagemodels], [kim2025pretraininginfinitecompute], [seto2025trainingbilinguallmsdata], suggested that regularization and cross-lingual data mixing could independently ameliorate repetitious data harms, but no systematic comparison across model scales and detailed quantification had been established. This paper addresses these gaps by exhaustively traversing the axes of data mixture, HP regime, and model scale (150M–1.43B), using Arabic as LR and English as HR, and analyzing both intermediate (validation loss) and downstream (ARC Easy) metrics.

Experimental Design

The study encompasses four paradigms defined by the cross-product of monolingual/bilingual training and basic/tuned HP regime. The corpus comprises 200M unique Arabic tokens and an effectively unlimited English source. Bilingual models sample each batch with $\alpha$ probability for Arabic and $1-\alpha$ for English; $\alpha$ serves as the principal mixing hyperparameter, swept alongside repetition count ($R_{\max}$), weight decay ($\lambda$), and learning rate ($\eta$). HP selection employs $\mu$P transfer [yang2022tensorprograms] from the 150M proxy unless explicitly tuned per scale. The evaluation benchmarks include held-out Arabic validation loss and 5-shot ARC Easy (translated to Arabic).

Empirical Findings

Data Mixing Dominates Regularization

Data mixing outperforms HP tuning on both validation loss and task accuracy, and the advantage intensifies with model size. At 380M, mixing yields a 5.7% reduction in Arabic validation loss, compared to 1.2% for HP tuning; at 1.43B, the respective improvements are 12.9% (mixing) and 3.1% (HP tuning). With increasing scale, the optimum mixing ratio shifts sharply toward English ($\alpha^* = 0.33 \rightarrow 0.10$ from 380M to 1.43B).

Figure 1: Left panel shows best Arabic validation loss versus model size, revealing pronounced separation for bilingual models as scale increases. Right panel demonstrates the optimum mixing fraction moving toward English and the validation basin flattening.

Benchmark Performance: Mixing Scales Steeply, Regularization Plateaus

Monolingual models show negligible scaling effect: Arabic ARC Easy accuracy increases from 29.7% to 32.3% across a $10\times$ parameter sweep, with English accuracy remaining near random. In contrast, bilingual models scale steeply: at 1.43B, Arabic accuracy reaches 41.4% under tuned HPs and English accuracy 57.7%. Mixing thus confers substantial transfer and regularization, far exceeding gains from HP tuning alone.

Figure 2: ARC Easy accuracy (5-shot) versus model size, demonstrating plateaued monolingual accuracy and steep bilingual scaling for both languages.

Validation Loss as Checkpoint Proxy

Within a single run, Arabic validation loss reliably selects near-peak checkpoints for both target and auxiliary language accuracy. Across runs, however, validation loss systematically underestimates the value of mixing, as benchmark multiplier quantifications reveal.

Figure 3: High correlation between checkpoint selection by Arabic validation loss and peak ARC Easy accuracy, confirming VL's effectiveness as per-run proxy.

Quantifying Mixing: Data Multiplier and the Validation-Benchmark Divergence

The data multiplier $N$ is defined as the equivalent corpus size a monolingual model would need to match the performance of a bilingual model trained on only 200M Arabic tokens. On validation loss, mixing equates to $1-\alpha$0–$1-\alpha$1 more unique target data. On downstream accuracy, the multiplier escalates from $1-\alpha$2 at 380M to $1-\alpha$3 at 1.43B, reflecting substantial benchmark gains undetected by validation loss.

Figure 4: Data multiplier versus model size—mixing achieves up to $1-\alpha$4 data equivalence in benchmark accuracy, substantially higher than the validation loss multiplier.

Mixing thus serves dual roles: diversifying training signal (regularization) and introducing new knowledge (transfer). Validation loss captures only the former, neglecting the latter's contribution—explaining why mixing's true efficacy is revealed only by task-based evaluation.

Figure 5: Peak ARC Easy accuracy versus Arabic data budget, showing bilingual stars for 200M tokens nearly matching monolingual performance at 2B tokens (14× more data), especially pronounced at larger scales.

Hyperparameter Sensitivity and Practitioner's Guidance

Variance decomposition (ANOVA) analysis confirms that mixing ratio selection ($1-\alpha$5 or $1-\alpha$6) dominates the impact on Arabic validation loss, especially at larger scales and within the operational neighborhood (e.g., near $1-\alpha$7P points). HP tuning provides marginal gains—at 1.43B, $1-\alpha$8P transfer from small proxy remains within $1-\alpha$9 of the tuned optimum for bilingual VL, rendering wide grid search unnecessary except for highly aggressive practitioners.

Figure 6: Contiguous neighborhood re-centering—variance shares indicate that $\alpha$0 drives performance in the vicinity of $\alpha$1P or optimum HPs, especially at operational scales.

Implications and Future Directions

These findings carry significant implications:

• Low-resource LM Pre-Training: Prioritize inclusion of HR auxiliary language over exhaustive HP tuning; mixing not only regularizes but enriches with external knowledge, markedly enhancing downstream capabilities.
• Practical Training Protocol: Use $\alpha$2P HP transfer from a small proxy for large-scale runs; hyperparameter search is best allocated to mixing ratio $\alpha$3, not weight decay or learning rate.
• Benchmark-Evaluation: Downstream task metrics should guide model selection, as validation loss does not fully reflect the gains enabled by cross-lingual mixing.

For theory, the observed divergence between VL and downstream accuracy challenges the standard reliance on in-distribution loss as a proxy for model generalization, suggesting that auxiliary data induces transfer effects beyond regularization.

Speculation on Future Developments

Potential future advances include dynamic mixing schedules, joint optimization of mixture weights across multiple auxiliary languages, and studies on typologically diverse language pairs. Extension to more difficult or natively-authored evaluation sets is necessary to calibrate mixing's impact accurately in practical deployment. Further, scaling laws and variance analysis techniques will likely be integrated into automated tuning systems for data-constrained multilingual models.

Conclusion

This work decisively demonstrates that bilingual pre-training with auxiliary HR data outperforms the traditional strategy of deep HP tuning under data constraints. The mixing ratio is the primary driver of model quality, and practitioners should default to $\alpha$4P HP transfer, reserving tuning resources for mixture optimization. Downstream evaluation is vital, as validation loss systematically underestimates both the regularization and transfer benefits of mixing. These results promote a new default recipe for low-resource LM construction: mix, don't tune.