Optimistic Dual Averaging Unifies Modern Optimizers

Abstract: We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded $1/k$ decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.

• The paper introduces SODA, a unified dual averaging framework that encapsulates diverse methods like AdamW and Muon, eliminating explicit weight decay tuning.
• The paper demonstrates that SODA’s averaging schedule and anchor point selection yield improved performance and rigorous convergence guarantees in various settings.
• The paper shows that SODA adapts to different geometries and momentum schemes, offering a versatile, parameter-free optimizer for modern deep learning models.

Optimistic Dual Averaging as a Unifying Framework for Modern Deep Learning Optimization

Context and Motivation

Contemporary deep learning optimizers have evolved along two primary axes: enhancements in geometry and adaptation, and compositionally sophisticated use of momentum, averaging, and scheduling. The proliferation of optimizers such as Adam, Lion, Muon, NAdam, and their spectral or sign-based variants has highlighted both the importance of geometry-induced updates and the subtleties of combining gradient feedback for stability and efficiency. However, hyperparameter selection—most notably weight decay—remains a challenge for scaling models and training durations. The paper "Optimistic Dual Averaging Unifies Modern Optimizers" (2605.11172) formalizes a unified theoretical and algorithmic perspective that encompasses these recent advances, centered around an extended dual averaging paradigm—SODA (Schedule-Free Optimistic Dual Averaging).

SODA: Unified Theory and Algorithmic Template

SODA generalizes classical dual averaging with optimistic momentum and primal extrapolation through explicit averaging schedules. The canonical update is parameterized by four interpolation coefficients (dual and optimistic dual, primal and extrapolation) and a geometry-defining regularizer $h_k$, enabling flexible recovery of major optimizer classes. Notably, SODA can instantiate Euclidean, $\ell_\infty$, and spectral norm geometries, and various momentum schemes (including Nesterov), through choice of regularizer and schedule.

The SODA algorithm update comprises:

• Dual (Gradient) Averaging and Optimism: Controlled via $\alpha_k$ (momentum) and $\bar{\alpha}_k$ (optimism), forming a dual sequence that generalizes classical and optimistic momentum.
• Primal Averaging and Extrapolation: $\lambda_k$ (stepsize/averaging) and $\bar{\lambda}_k$ (extrapolation), governing iterate averaging and anchoring updates.
• Geometry: Regularizer $h_k$ selects the induction of norm (e.g., spectral, $\ell_\infty$) and influences noise robustness and scaling.
• Weight Decay as Averaging: The $1/(k+2)$ scheduled decay is derived theoretically, interpreted as averaging with respect to the initialization, rather than as a classical regularizer.

A practical SODA wrapper eliminates the need for explicit weight decay tuning by embedding the averaging schedule in the optimizer structure, yielding a $1/k$ decay that is theoretically justified.

Theoretical Guarantees and Parameterization

SODA admits rigorous convergence guarantees under both bounded gradient and smoothness assumptions. Key theoretical results include:

• Rate Bounds: SODA achieves $\ell_\infty$0 convergence for general convex objectives, and $\ell_\infty$1, $\ell_\infty$2 rates under increased smoothness and accelerated schedules, matching optimal statistical lower bounds for stochastic optimization.
• Parameter Choices: Theory dictates schedules for the four averaging parameters, tying primal extrapolation to acceleration, and prescribing small values for extrapolation under smoothness, enabling universality.
• Anchor Point: The choice of reference point $\ell_\infty$3 is nontrivial; empirical evidence and rate bounds show that anchoring at the initialization yields optimal behavior, while centering at the origin is suboptimal where the solution is distant (Figure 1).

Empirical Results

Experiments rigorously benchmark SODA against tuned baselines (AdamW, Muon, Scion/uScion) with NanoGPT models on FineWeb100, varying both horizon and scale. Major empirical findings include:

• SODA Wrapper Outperforms Tuned Baselines: SODA applied as a wrapper around Muon, Adam, and Scion systematically outperforms baselines across training durations and scales, even when those baselines are carefully tuned for weight decay (Figure 2, Figure 3).

Figure 2: Muon with swept weight decay is outperformed by SODA(Muon), without any additional tuning, for both short and extended training horizons.

Figure 3: SODA wrapper consistently exhibits improvement across diverse base optimizers, without additional tuning.

• Optimism and Simplification: The SODAdag variant (optimistic SODA with unified momentum parameters and Lion replacing Adam) is competitive or superior to best-tuned configurations, with simplified hyperparameter requirements (Figure 4).

Figure 4: SODAdag matches or outperforms the best Muon-based optimizer, using a simpler setup.

• Scaling Behavior: Under Chinchilla scaling (width and horizon grow proportionally), SODA delivers increasingly significant benefit as model size grows, indicating favorable scaling and robustness in large regimes (Figure 5).

  Figure 5: SODA(uScion) and SODAdag outperform baselines more clearly at larger scale.

• Anchor Point Importance: Using $\ell_\infty$4 (instead of initialization) results in marked deterioration in performance, confirming theoretical predictions regarding anchor-dependent convergence (Figure 1).

  Figure 1: Performance degrades when SODA’s anchor is set to zero rather than the initialization, emphasizing the criticality of anchoring.

• Layer-Wise Decay Scheduling: SODA’s averaging-induced decay can be applied selectively (hidden layers only or all layers). Empirical ablation indicates improvement in hidden layers while not harming performance on first and last layers.

Practical and Theoretical Implications

SODA unifies the landscape of modern deep learning optimizers—encompassing sign-based, spectral, and adaptive momentum recipes—via a rigorous framework that clarifies and demystifies their design axes. Central implications include:

• Weight Decay Reframed: Rather than a classical regularizer, weight decay is cast as an averaging schedule, rooted in dual averaging theory. This shift eliminates per-optimizer hyperparameter tuning and enables parameter-free schedules.
• Hyperparameter Transfer: SODA’s schedules avoid horizon- and model-size-dependent heuristics, facilitating robust transfer across width and duration; empirical evidence suggests $\ell_\infty$5 decay is optimal outside Chinchilla scaling.
• Geometry-Aware Recipes: Spectral and multi-norm updates can be injected as regularizer selections, yielding per-layer norm control and improved scaling, especially in overparameterized settings.

The framework’s universality and parameter-free operation present clear practical advantages for large-scale training and finetuning, where optimizer configuration is costly.

Future Directions

Key speculative avenues include:

• Finetuning Protocols: SODA’s anchoring at the initialization (or pretrained weights) may be instrumental in finetuning, where classical weight decay against the origin is suboptimal.
• Accelerated SODA: Empirical instantiations of the theoretically accelerated schedules are yet to be fully explored, offering potential for further speedups.
• Non-Convex Extensions: Although current guarantees are convex, extending analysis to non-convex settings may yield deeper insights, especially for deep networks.

Conclusion

SODA provides a principled, theoretically rigorous, and empirically validated framework for optimizing modern deep learning models. It conceptually unifies disparate optimizers, offers parameter-free practical deployment, and reframes weight decay through the lens of primal averaging and dual processing. These developments lay the groundwork for robust, scalable optimization protocols, and offer fertile direction for future research in optimization algorithm design, geometry selection, and training schedule automation.