Maximal-Update Parameterization (μP)

• Maximal-Update Parameterization (μP) is a neural network framework that defines scaling rules for initialization, learning rates, and output weights to ensure O(1) feature updates as network width increases.
• It guarantees width-invariant optimization by converging to a nonzero optimal learning rate, enabling zero-shot hyperparameter transfer from small proxy models to full-scale networks.
• Empirical benchmarks show μP outperforms standard and NTK parameterizations by maintaining stable, nontrivial feature learning across deep neural networks.

Maximal-Update Parameterization (μP) is a principled neural network parameterization framework engineered to ensure stable and nontrivial feature learning at arbitrary width. μP uniquely prescribes how layerwise initialization, scaling of output weights, and learning rates are chosen, such that the optimal learning rate converges to a nonzero constant as network width increases, enabling zero-shot hyperparameter transfer from small proxy models to full-scale networks and ensuring robust, width-invariant optimization. μP stands in contrast to standard and Neural Tangent Parameterizations, which respectively suffer from vanishing or exploding learning rates and feature updates in the infinite-width limit (Hayou, 3 Nov 2025).

1. Formal Definition and Scaling Rules

The Maximal-Update Parameterization is specified for deep multi-layer perceptrons as follows. Consider an $L$-layer MLP,

$f(x) = V^T W_L \cdots W_1 W_0 x$

where each hidden layer $W_\ell \in \mathbb{R}^{n \times n}$ has width $n$, $W_0 \in \mathbb{R}^{n \times d}$ is the input layer, and $V \in \mathbb{R}^n$ is the output readout. In μP, one chooses scaling exponents $\alpha_0, \ldots, \alpha_L, \alpha_V, \alpha_\eta$ such that:

• Initializations:
  • $W_0^{ij} \sim \mathcal{N}(0, d^{-\alpha_0})$, $\alpha_0 = 1$
  • $W_\ell^{ij} \sim \mathcal{N}(0, n^{-\alpha_\ell})$, $\alpha_\ell = 1$ for $\ell=1,\ldots,L$
  • $V^i \sim \mathcal{N}(0, n^{-\alpha_V})$, $\alpha_V = 2$
• Learning rate:
  • For full-batch gradient descent: $\eta = O(1)$ ($\alpha_\eta = 0$)
  • For Adam: $\eta n^{-1}$ ($\alpha_\eta = 1$)

Equivalently, one can write for $\ell \ge 1$:

• $W_\ell \sim n^{-1/2} \widetilde W_\ell$, $V \sim n^{-1} \widetilde V$, $\eta \sim n^0$ (for GD)

A critical μP prescription is the “mean-field” scaling of the readout: one power smaller than standard ($\alpha_V = 2$ rather than $1$) (Hayou, 3 Nov 2025).

2. Theoretical Guarantees for Learning Rate Transfer

μP is characterized by rigorous theorems establishing the convergence of the optimal learning rate $\eta_n^{(t)}$ with width $n$:

Theorem 2.1: One-Step Transfer

With input Gram matrix $K$ and target vector $y$ such that $Ky \neq 0$, the unique minimizer of the infinite-width one-step loss

$$L_\infty^{(1)}(\eta)=\frac{1}{2m}\| -y+\eta (L/m)K y \|^2$$

is given by

$$\eta_\infty^{(1)} = \frac{m}{L}\frac{y^T K y}{\| K y \|^2} > 0$$

and for any compact interval containing $\eta_\infty^{(1)}$,

$\eta_n^{(1)} - \eta_\infty^{(1)} = O_P(n^{-1/2})$

Theorem 2.2: $t$-Step Transfer

For any fixed number of steps $t$, the empirical loss curve $L_n^{(t)}(\eta)$ uniformly converges to a deterministic limit $L_\infty^{(t)}(\eta)$, whose minimizer $\eta_\infty^{(t)} > 0$ is the limit of the finite-width optimizers.

Standard Parametrization and Neural Tangent Parameterization fail this transfer property: under SP, the optimal learning rate shrinks toward zero as $n$ grows; under NTK scaling, the optimum diverges (Hayou, 3 Nov 2025).

3. Mechanistic Rationale: Feature Updates and Infinite-Width Dynamics

In μP, every pre- and post-activation change $\Delta z$ per gradient step remains $\Theta(1)$ as $n \to \infty$. Contrastingly, SP forces $\Delta z \to 0$ (lazy/NTK regime) or causes updates to explode (unstable regime). The core mechanistic insight is that μP “maximizes” feature updates—a fully nonlinear mean-field-type dynamics in which features adapt and evolve with width, maintaining nontrivial feature learning (Hayou, 3 Nov 2025). This is achieved by scaling the output-layer variance and learning rate to guarantee that each gradient update induces an $O(1)$ change to activations throughout training as width increases.

4. Empirical Evidence and Benchmarks

Extensive experiments using synthetic datasets (dimension $d=100$, $N=1000$ samples), over MLPs with depths $L \in \{3, 9, 27\}$ and widths $n \in \{128, \ldots, 8192\}$, demonstrate:

• Under μP, loss-vs-learning-rate curves for different widths overlap tightly, and the optimal learning rate converges rapidly to a nonzero value, with error decaying at $O(n^{-1/2})$.
• Standard Parametrization yields optima that shrink toward zero with increasing width and misaligned loss curves.
• μP with Adam exhibits stable width-invariance of optimal learning rates even in very deep nets ($L=27$).
• Extended training widens the basin of near-optimal $\eta$, but the location of the optimum remains invariant (Hayou, 3 Nov 2025).

5. Practical Implementation: Guidelines for Hyperparameter Tuning

μP provides a robust workflow for practitioners:

• Tune the learning rate $\eta$ on a proxy model with moderate width ($n_0 = 256\!-\!1024$) via logarithmic grid sweep to locate the optimal basin.
• For full width $n \gg n_0$, reuse the same $\eta$ with negligible retuning—performance changes are typically under $20\%$ even at $10\times$ scale.
• For Adam, scale the learning rate by $n$ as $\eta_\text{actual}=\eta_\text{grid}/n$.
• When increasing depth $L$, theory and empirics suggest tuning $\eta$ near $O(1/L)$, as the infinite-width optimum is proportional to $1/L$.
• The prescription ensures width-robust transferability and zero-shot deployment for hyperparameters as model size increases (Hayou, 3 Nov 2025).

6. Broader Implications: Feature Learning and Parameterization Selection

The maximal-update property of μP fundamentally preserves nontrivial feature adaptation, enabling mean-field, nonlinear learning dynamics and edge-of-stability phenomena as observed in wide deep nets. Unlike SP and NTP, which respectively drive networks into lazy/NTK or unstable regimes, μP is the unique parameterization (within the abc-framework and linear MLP analysis) that yields both stable infinite-width learning rates and maximal feature updates. This regime is theorized to capture essential behaviors of practical deep networks, while freeing practitioners from costly, width-specific hyperparameter retuning (Hayou, 3 Nov 2025).