On the Optimizer Dependence of Neural Scaling Laws

Abstract: The scaling exponent $α$ in neural scaling laws $L(N) \propto N^{-α}$ is commonly treated as a fixed constant set by architecture and data. We present evidence that $α$ depends systematically on the optimizer. In controlled random-feature regression experiments -- the canonical theoretical framework for neural scaling -- we measure $α$ across five optimizer variants and six spectral conditions. Preconditioned optimizers consistently yield steeper scaling (larger $α$), with the $α$-shift increasing across most of the tested spectral range, peaking near $s = 1.5$, and remaining large at $s = 2.0$. At $s \approx 1.0$ (characteristic of natural language), the full natural gradient achieves $α\approx 0.31$ versus $α\approx 0.12$ for gradient descent -- a $2.6\times$ larger fitted exponent that, within the random-feature model, compounds with each model-size doubling. Whether and how this exponent shift transfers to large-scale LLM training -- where recent evidence suggests the advantage may attenuate with scale -- remains an important open question. Our results imply that scaling-law forecasts should account for optimizer choice, and we provide a spectral diagnostic predicting when advanced optimizers will pay off.

• The paper shows that preconditioned optimizers systematically shift scaling exponents, with full natural gradient yielding up to 2.6 times higher exponent than gradient descent in steep spectra.
• The methodology uses a random-feature regression model to isolate top-layer learning, ensuring that differences in scaling exponents are solely due to optimizer choice.
• The study highlights that spectral steepness modulates the impact of preconditioning, equalizing per-mode learning rates and leading to exponentially greater returns with increased model size.

Optimizer-Dependent Neural Scaling Laws: A Formal Summary

Motivation and Context

Neural scaling laws of the form $L(N) \propto N^{-\alpha}$ are fundamental to understanding the efficiency of increasing model size $N$ in large-scale neural networks. The exponent $\alpha$ directly determines returns to scale, yet it is typically treated as intrinsic to the data and model architecture. This paper demonstrates through precisely controlled random-feature regression experiments that $\alpha$ is not invariant, but systematically depends on the optimizer. Particularly, preconditioned optimizers induce significantly higher scaling exponents, an effect that is amplified with increasing spectral decay of the data.

The investigation addresses open questions suggested by inconsistencies in empirical literature, such as the scale-dependent efficacy of recent second-order optimizers (e.g., Muon, SOAP, Shampoo). Recent works reported both strong, apparently compounding compute advantages and a diminishing effect at higher parameter counts. This study provides principled, mechanistically interpretable evidence for optimizer-induced exponent shifts and their dependence on spectral statistics.

Experimental Framework

The authors employ a random-feature regression model, which isolates top-layer learning and enables precise manipulation of the data spectrum, preconditioner type, and model capacity. The input covariance has power-law spectrum $\lambda_i \sim i^{-(1+s)}$ parameterized by exponent $s$, encompassing regimes typical of natural language data ($s \approx 1$). The student uses fixed random features; training optimizes only top-layer coefficients with different optimizers implemented as preconditioners. This structure ensures that all differences in scaling exponents can be attributed to the optimizer’s action on the spectrum.

The study evaluates five optimizers:

• Gradient Descent (GD)
• Diagonal Preconditioner (AdamW proxy)
• Full Natural Gradient (K-FAC/Shampoo proxy)
• Matrix-Sign (Muon proxy)
• Sign-GD (negative control)

Scaling exponents are fitted by OLS regression of log test loss against log model size for each optimizer and spectrum, ensuring power-law behavior in the tested regime for all preconditioned variants.

Core Empirical Findings

The primary result is that preconditioned optimizers yield larger exponents $\alpha$ than GD across all spectral conditions, with the gap, $\Delta\alpha$, increasing with spectral steepness $s$ and saturating for the steepest tested spectra.

At $N$0, characteristic of language data, the natural gradient achieves $N$1 versus $N$2 for GD, a $N$3 multiplier in exponent. This exponent shift compounds with every doubling of $N$4, implying exponentially greater returns to scale for preconditioned optimization.

Figure 1: Scaling curves $N$5 for five optimizers at two spectral exponents; preconditioned methods exhibit visibly steeper slopes, especially at $N$6.

Figure 2: Scaling exponent $N$7 versus spectral exponent $N$8; Full NG and Matrix-Sign maintain high $N$9, while GD degrades precipitously with increasing $\alpha$0.

For all tested optimizers, the ordering is consistent: Full NG $\alpha$1 Matrix-Sign $\alpha$2 Diagonal $\alpha$3 Sign-GD $\alpha$4 GD. The effect size is negligible for flat spectra $\alpha$5, confirming that preconditioning is superfluous where the spectrum is already uniform.

The translate of the $\alpha$6 shift into an effective compute multiplier shows that, in the random-feature regime, strong preconditioning can yield parameter-equivalent advantages exceeding those inferred from recent frontier LLM benchmarks.

Figure 3: Effective compute multiplier—the factor by which GD must scale $\alpha$7 to match preconditioned test loss—rises sharply with $\alpha$8 for strong preconditioners.

Spectral Mechanism and Theoretical Implications

The central mechanistic insight is that GD allocates capacity inefficiently: high-variance (dominant) modes are overfit while “hard” (low-variance) modes are underfit at any finite $\alpha$9. Preconditioning equalizes per-mode learning rates, ensuring that all modes benefit from increased capacity. Theoretical analysis of the per-mode convergence factors under each preconditioner supports:

• Preconditioning universally increases $\alpha$0 versus GD for any spectrum with $\alpha$1.
• The $\alpha$2-gap grows monotonically with the spectrum’s steepness, saturating only where finite-size (finite $\alpha$3) effects dominate.

This identifies spectral steepness as a diagnostic for when optimizer choice impacts scaling law exponents, not merely convergence rate.

Robustness and Limitations

Results are robust to substantial changes in input dimension and target spectrum (teacher coefficient decay), with all main patterns and relative optimizer ordering preserved.

Figure 4: Robustness checks—qualitative scaling patterns and $\alpha$4 ordering are invariant with increased input dimension and steeper target coefficient decay.

Convergence diagnostics show that, for strong preconditioners, iterative and direct-solve implementations yield virtually identical exponents, affirming that results reflect genuine differences in scaling behavior, not just convergence speed under finite gradient step budgets.

Sign-GD serves as a negative control, displaying poor scaling and noisy exponent estimates, confirming that adaptivity alone does not improve scaling laws unless spectral reweighting is achieved.

Relationship to Prior Work and Practical Implications

Directional alignment is found with LLM studies reporting substantial (though strongly scale-dependent) compute advantages for second-order optimizers over AdamW. The key extension here is the interpretation of optimizer effects not as fixed constant-factor improvements, but as exponent shifts—with theoretically compounding returns in the regime where spectral imbalance is strong and feature learning is absent or limited.

However, evidence from large-scale models indicates that feature learning may reduce spectral imbalances at scale, thus dampening the exponent-shifting effect. The theoretical prediction arising from this work, directly testable in LLMs, is that optimizer-dependent scaling advantages should vanish for spectrally flattened data, and survive in steep-spectrum settings.

Limitations and Directions for Future Work

The analysis is restricted to the random-feature (lazy) regime: results may overestimate exponent shifts compared to real deep networks in the feature-learning (rich) regime. Examination of depth scaling, interaction with momentum and normalization, and the computational cost-benefit tradeoff for preconditioned optimizers in practice are deferred.

A real-model validation protocol is proposed to directly test for optimizer-dependent exponent shifts in GPT-scale transformers on natural and spectrally-equalized datasets.

Conclusion

This paper establishes that the optimizer fundamentally impacts neural scaling laws by shifting the power-law exponent $\alpha$5 in a spectrum-dependent manner. Preconditioned optimizers, by equalizing per-mode training rates, yield more favorable scaling—especially as the spectrum steepens—while adaptive but non-spectrally aware methods provide no benefit. These findings challenge the standard practice of extrapolating future model performance under a fixed scaling paradigm and demonstrate that optimizer selection must be accounted for in compute budgets and scaling forecasts. The precise transfer of these exponent-shifting phenomena to large-scale feature-learning tasks remains an open and critical theoretical and empirical question.