Reliable hyperparameter transfer across scales

Develop reliable methods to transfer optimal training hyperparameters such as learning rate and initialization from small-scale proxy models to large-scale Large Language Models while guaranteeing stable training dynamics.

Background

Given the prohibitive cost of hyperparameter search at LLM scale, the paper discusses frameworks like maximal update parametrization (μP) and emerging scaling laws intended to enable zero-shot transfer of hyperparameters from small proxies. It notes promising empirical validations but also architectural sensitivities and limitations.

The authors frame the central unresolved issue as ensuring that such transfers are reliable across architectures, data regimes, and optimization dynamics, motivating principled methods and theoretical guarantees for cross-scale hyperparameter selection.

References

Consequently, a critical open question is how to reliably transfer optimal hyperparameters (e.g., learning rate, initialization) found on small-scale proxy models to large-scale target models.

— Beyond the Black Box: Theory and Mechanism of Large Language Models  (2601.02907 - Gan et al., 6 Jan 2026) in Subsubsection Hyperparameter Transfer, Section 4: Training Stage (Advanced Topics and Open Questions)

Nevertheless, sparsity follows distinct scaling behaviors \citep{clark2022unified,tian2025towards,team2025kimi}, and it remains unclear whether existing hyperparameter transfer frameworks derived for width scaling generalize to the sparsity dimension. This critical gap remains largely unaddressed, leaving optimal hyperparameter configuration prediction for large-scale MoE training an open problem.

— Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts  (2608.20061 - Kim et al., 20 Aug 2026) in Section 1, Introduction

We apply Vankadara's initialization formulas and (for the non-approximated variant) their LR exponents anchored at our proxy width $N_u{\text{ref}{=}512$; whether the same exponents are theoretically optimal under simplified ZOH is not established. Implementing true-ZOH Mamba and re-evaluating block-wise $P$ on its own derived discretization, ideally also under proportional $N_u/N_x$ scaling and at production-scale $N_x$, is left to future work.

— Learning Rate Transfer for Hybrid Transformer-SSM Architectures  (2610.01172 - Seo et al., 1 Oct 2026) in Appendix, Section “Applying P-SSM Beyond the Pure-SSM SGD Setting,” paragraph “Regime gaps from the original derivation”; Conclusion, Limitations paragraph

Mamba-3~\citep{gu2026mamba3} is left to future work: its complex-valued state lies outside the real-valued $P$ rules, and we know of no complex-state $P$ formulation; a hybrid built on a real-valued non-Mamba recurrent mixer, Gated Linear Attention (GLA)~\citep{yang2024gla}, does transfer (Appendix~\ref{sec:appendix:gla}).

— Learning Rate Transfer for Hybrid Transformer-SSM Architectures  (2610.01172 - Seo et al., 1 Oct 2026) in Conclusion, Limitations paragraph