Fantastic Pretraining Optimizers Overview

• Fantastic pretraining optimizers are advanced algorithmic tools that enhance pretraining efficiency through structured adaptations and meticulous hyperparameter tuning.
• They integrate matrix preconditioners, scalar adaptive methods, and sign-based techniques, offering modest speedups over standard optimizers like AdamW, particularly in smaller models.
• Rigorous benchmarking and tailored evaluation protocols are critical to accurately assess performance gains across model scales and ensure practical applicability in various deep learning domains.

Fantastic pretraining optimizers are advanced algorithmic tools and methodologies designed to accelerate and stabilize the optimization phase of large-scale neural network pretraining. Their development is driven by the scaling of deep learning models, the high cost of pretraining, and the limitations of traditional hand-designed optimizers such as AdamW. Recent years have seen the emergence of new families of optimizers that exploit structural properties of model parameters, leverage meta-learning, or adopt more careful benchmarking and hyperparameter tuning methodologies to close the gap between theory and practice. These techniques are evaluated on challenging setups involving LLMs, computer vision architectures, and multimodal networks, with rigorous empirical validation protocols.

1. Systematic Comparison of Optimizers for Pretraining

A major challenge in the field is the fair evaluation of novel optimizers against robust baselines. Many earlier studies reported speedups of 1.4× to 2× over AdamW; however, rigorous benchmarking under a common experimental protocol—hyperparameter tuning, end-of-training comparisons, and multiple model/data scales—shows that these claims are generally overstated. When properly evaluated, the speedup of advanced optimizers is lower and decreases with model scale, e.g., matrix-based optimizers such as Muon and Soap provide a 1.4× advantage over AdamW for 0.1B scale models but only 1.1× at 1.2B parameters (Wen et al., 2 Sep 2025).

Optimizers are compared across scalar (per-parameter) adaptation schemes (AdamW and variants), matrix-based preconditioners (e.g., Muon, Soap, Kron, Scion), sign-based methods (Lion, Signum), schedule-free techniques (Prodigy, SF-AdamW), and methods incorporating second-order information (Sophia, D-Muon). Performance is quantified in terms of validation loss or perplexity, as well as the number of tokens required to reach target loss curves. Rankings between optimizers can also be reversed depending on when during training they are evaluated: performance at intermediate checkpoints is not predictive of final efficiency due to learning rate decay and scale effects.

Optimizer Class	Mechanism	Speedup Over AdamW (0.1B–1.2B)
Matrix preconditioners	Muon, Soap	1.4× → 1.1×
Scalar adaptive methods	AdamW, AdEMAMix, Lion	baseline
Sign-based	Signum, Lion	competitive at large batch
Schedule-free	Prodigy, SF-AdamW	variable
Second-order/curvature	Sophia, D-Muon	unstable/stable, per tuning

2. Hyperparameter Tuning and Evaluation Protocols

The efficacy of any optimizer is strongly dependent on thorough and independent hyperparameter tuning (Wen et al., 2 Sep 2025). Blindly transferring hyperparameters between optimizers, even among very similar Adam-like methods, produces misleading comparisons and can exaggerate claimed improvements. Grid search or coordinate descent over learning rate $\eta$, weight decay $\lambda$, warmup duration, and other optimizer-specific parameters is necessary for every optimizer–model–data regime pairing. For instance, the AdamW update

$$w_{t+1} = w_t - \eta\, \frac{m_t}{\sqrt{v_t} + \epsilon} - \eta\, \lambda\, w_t$$

must be tuned for $\eta$, $\lambda$, and warmup to ensure competitive performance.

Evaluations must compare optimizers at matched end-of-training checkpoints (i.e., after the full training budget) and across several model sizes and data/model ratios to ensure generalizable conclusions. Intermediate checkpoints frequently misrepresent final performance: optimizer rankings can flip as adaptive learning rate decays are triggered.

3. Matrix-Preconditioned Optimizers

Matrix-based preconditioners apply structured adaptation by manipulating gradient matrices rather than treating every parameter as an independent scalar. In Muon, Soap, Kron, and related methods, a preconditioning matrix $P_t$ derived from the local curvature or historical gradients rescales and rotates updates:

$w_{t+1} = w_t - \eta\, P_t\, g_t$

such that $P_t$ can be approximated by iterative methods like Newton–Schulz or via QR-based factorization (Wen et al., 2 Sep 2025). These optimizers "whiten" parameter updates and accelerate convergence in directions of high anisotropy, which is especially beneficial in early training for smaller models where curvature is not yet uniform.

However, as the model size increases, the relative benefit of precise structural adaptation decreases, with speedup benefits saturating. Matrix methods add computational overhead due to matrix operations, necessitating careful engineering and, sometimes, adjusting for specific architectural components (e.g., "fat" layers).

4. Scalar Adaptive and Sign-Based Methods

Scalar adaptive methods (AdamW, AdEMAMix) compute per-parameter adaptive updates using running averages of the gradient and squared gradient, but do not exploit any parameter-tensor structure. These methods are highly robust and remain the standard for very large models. Their update is

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

with bias correction.

Sign-based methods such as Lion and Signum use only the sign of the aggregated gradient or momentum,

$x_{t+1} = x_t - \gamma\, \mathrm{sign}(m_t)$

reducing sensitivity to exact scaling but requiring larger batch sizes to overcome increased gradient noise. Their performance is highly dependent on batch size and model/optimizer-specific hyperparameter schedules.

5. Practical Implications and Recommendations

Practitioners must avoid default reliance on any optimizer and instead:

• Tune hyperparameters independently for each scenario and optimizer, especially learning rates, weight decay, and warmup schedules.
• Prefer matrix-preconditioned optimizers (Muon, Soap) for smaller or moderately sized models and in regimes where speedup is tangibly beneficial, but expect diminishing returns for massive models.
• Sign-based methods (Lion) are competitive in large batch settings and may offer compute efficiency if stability is not an overriding concern.
• Carefully choose evaluation setups: compare all optimizers at the end of training for identical training budgets and across data/model scales, as pointwise comparison can be misleading.

6. Methodological Contributions and Future Research Directions

The paper calls for rigorous benchmarking standards in optimizer evaluation for pretraining, including:

• Transparent reporting of hyperparameter tuning, systematic grid searches, and reporting of all "failure" scenarios.
• Open-source experimental infrastructure, such as that provided by (Semenov et al., 1 Sep 2025), to facilitate reproducibility and cross-method validation.
• Further investigation into scalable, robust matrix preconditioners that minimize computational overhead.
• Understanding the interplay between optimizer design and training scale, including how schedule-free and sign-based methods can be made robust for LLM pretraining.

7. Summary and Limitations

The contemporary landscape of fantastic pretraining optimizers is characterized by incremental improvements over industry standards when rigorous benchmarking is performed. While sophisticated matrix-based optimizers provide measurable, but modest, gains for small-to-midscale models, the fundamental limits of these methods become apparent at LLM scale. The key determinants of real-world utility are robust hyperparameter tuning, evaluation methodology, and careful consideration of architectural scale. The future development of optimizers will likely focus on reducing computational and memory cost, increasing robustness to scale, and integrating structural parameter information in a manner that justifies their overhead for large models.

Optimizer	Speedup over AdamW (small–large scale)	Requires Tuning	Compute Overhead	Notes
Matrix-based	1.4× → 1.1×	Yes	Moderate	Diminishing return at scale
Scalar/AdamW	1.0× (baseline)	Yes	Low	Highly robust
Sign/Lion	~1.0× (large-batch)	Yes	Low	Sensitive to batch/decay
Schedule-free	Variable	Yes	Low–Moderate	Needs further research

The domain continues to evolve, with rigorous head-to-head evaluations clarifying genuine progress and refocusing research efforts toward scalable, efficient, and robust optimization strategies for pretraining large neural networks.