Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods (2510.10777v1)

Abstract: Optimization lies at the core of modern deep learning, yet existing methods often face a fundamental trade-off between adapting to problem geometry and leveraging curvature utilization. Steepest descent algorithms adapt to different geometries through norm choices but remain strictly first-order, whereas quasi-Newton and adaptive optimizers incorporate curvature information but are restricted to Frobenius geometry, limiting their applicability across diverse architectures. In this work, we propose a unified framework generalizing steepest descent, quasi-Newton methods, and adaptive methods through the novel notion of preconditioned matrix norms. This abstraction reveals that widely used optimizers such as SGD and Adam, as well as more advanced approaches like Muon and KL-Shampoo, and recent hybrids including SOAP and SPlus, all emerge as special cases of the same principle. Within this framework, we provide the first systematic treatment of affine and scale invariance in the matrix-parameterized setting, establishing necessary and sufficient conditions under generalized norms. Building on this foundation, we introduce two new methods, $\texttt{MuAdam}$ and $\texttt{MuAdam-SANIA}$, which combine the spectral geometry of Muon with Adam-style preconditioning. Our experiments demonstrate that these optimizers are competitive with, and in some cases outperform, existing state-of-the-art methods. Our code is available at https://github.com/brain-lab-research/LIB/tree/quasi_descent

• The paper introduces a unified framework showing that steepest descent, quasi-Newton, and adaptive methods are all instances of steepest descent with preconditioned norms.
• It derives theoretical conditions for affine and scale invariance in matrix-parameterized optimization, offering insights into robustness under reparameterizations.
• Novel optimizers MuAdam and MuAdam-SANIA are proposed and empirically validated, demonstrating competitive or superior performance in deep learning and LLM fine-tuning tasks.

Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton, and Adaptive Methods

Introduction and Motivation

This paper introduces a unified geometric framework for optimization in deep learning, based on the concept of preconditioned matrix norms. The framework generalizes and connects steepest descent, quasi-Newton, and adaptive methods, which have traditionally been treated as distinct algorithmic families. The central insight is that these methods can all be viewed as instances of steepest descent under appropriately preconditioned norms, enabling a systematic analysis of their geometric properties—most notably, affine and scale invariance—in the context of matrix-parameterized models.

The work further proposes two new optimizers, MuAdam and MuAdam-SANIA, which combine the spectral geometry of Muon with Adam-style and SANIA-style preconditioning, respectively. Empirical results demonstrate that these methods are competitive with, and sometimes outperform, state-of-the-art optimizers across a range of tasks, including LLM fine-tuning and language modeling.

Unified Framework: Preconditioned Matrix Norms

The core of the framework is the definition of preconditioned matrix norms, which generalize the geometry underlying the update steps of various optimizers. Two main classes are introduced:

• (L, R)-preconditioned norms: For $G \in \mathbb{R}^{m \times n}$, and positive definite $L \in \mathbb{R}^{m \times m}$, $R \in \mathbb{R}^{n \times n}$, the norm is defined as $\|G\|_{L, R, \cdot} = \|L G R\|$ for a base norm $\|\cdot\|$.
• D-preconditioned norms: For diagonal positive $D \in \mathbb{R}^{m \times n}$, $\|G\|_{D, \cdot} = \|D \odot G\|$.

These constructions subsume Kronecker-factored quasi-Newton methods (e.g., Shampoo, K-FAC) and element-wise adaptive methods (e.g., Adam, AdaGrad) as special cases, depending on the choice of $L, R, D$, and the base norm.

The update direction is determined by a linear minimization oracle (LMO) under the chosen norm, which, for a given gradient $G_t$, solves:

$$\Delta W_t = \operatorname{lmo}(G_t) = \arg\max_{T: \|T\| \leq \rho} \langle G_t, T \rangle$$

where $\rho$ is a scaling parameter.

The paper proves that the LMO for preconditioned norms reduces to applying the LMO of the base norm in a transformed gradient space, followed by mapping back via the preconditioners. This result enables a systematic derivation of update rules for a wide range of optimizers.

Geometric Invariance: Affine and Scale Invariance

A major theoretical contribution is the first systematic treatment of affine and scale invariance for matrix-parameterized optimization. The paper formalizes these invariance properties as follows:

• Affine invariance: The optimizer's trajectory is unchanged under reparameterizations of the form $W \mapsto A_L W A_R$ for invertible $A_L, A_R$.
• Scale invariance: The optimizer is invariant to element-wise rescaling $W \mapsto A \odot W$ for positive diagonal $A$.

The necessary and sufficient conditions for these invariances are derived in terms of the transformation rules for the preconditioners. For example, affine invariance for $(L, R)$-norms requires that the preconditioners transform as $L_{\text{new}} = L A_L$, $R_{\text{new}} = A_R R$, while scale invariance for $D$-norms requires $D_{\text{new}} = A \odot D$.

This analysis clarifies why certain optimizers (e.g., Newton's method, SANIA) possess these invariance properties, while others (e.g., SGD, Adam) do not.

Algorithmic Instantiations: MuAdam and MuAdam-SANIA

Building on the unified framework, the paper introduces two new optimizers:

• MuAdam: Combines Muon's spectral LMO with Adam-style diagonal preconditioning. The update involves two applications of the element-wise preconditioner, as dictated by the theory, and a spectral projection step implemented efficiently via Newton–Schulz iteration.
• MuAdam-SANIA: Extends MuAdam by using SANIA-style preconditioning, which replaces the square root in Adam's denominator with a direct normalization, yielding scale invariance.

The algorithmic structure is as follows:

for t in range(T): G_t = grad(Loss, W_t) M_t = beta1 * M_{t-1} + (1 - beta1) * G_t V_t = beta2 * V_{t-1} + (1 - beta2) * (G_t * G_t) N_t = M_t / (V_t ** p + eps) # p=1/4 for MuAdam, p=1/2 for MuAdam-SANIA N_t_prime = spectral_projection(N_t) # via Newton-Schulz N_t_double_prime = N_t_prime / (V_t ** p + eps) W_{t+1} = W_t - lr * N_t_double_prime

This design leverages both the geometry of the spectral norm and the adaptivity of diagonal preconditioning, and is justified by the general LMO result for preconditioned norms.

Empirical Evaluation

Scale Invariance Experiments

The paper empirically verifies scale invariance using a two-layer MLP on the Mushrooms dataset, comparing optimizers on original and coordinate-rescaled inputs. AdamW and Muon, which lack scale invariance, show degraded performance under rescaling, while Adam-SANIA and MuAdam-SANIA maintain identical trajectories.

Figure 1: Scale invariance experiment (Mushrooms, LIBSVM) with a two-layer MLP. Training loss (left, log-scale) and test accuracy (right) on original vs. scaled inputs.

GLUE Benchmark and LLM Fine-Tuning

On GLUE, MuAdam matches or exceeds AdamW and Muon in both LoRA and full fine-tuning settings, demonstrating the practical benefit of combining spectral geometry with adaptive preconditioning.

For LLM fine-tuning (Qwen2-7B on BoolQ, HellaSwag, ARC-Challenge), MuAdam outperforms AdamW and Muon on two out of three tasks, with competitive mean accuracy and low variance across seeds.

Figure 2: LLM fine-tuning results on Qwen2-7B: mean final accuracy with standard deviation across three seeds.

Character-Level Language Modeling

On the Shakespeare dataset, MuAdam outperforms AdamW on deeper transformer models (3 and 4 layers), while both significantly outperform Muon. This supports the claim that spectral geometry and adaptivity can be synergistically combined for improved optimization in deep architectures.

Implications and Future Directions

The unified framework provides a principled foundation for the design and analysis of optimization algorithms in deep learning, particularly for matrix-parameterized models. By clarifying the geometric and invariance properties of existing methods, it enables the systematic construction of new optimizers with desired characteristics.

The empirical results suggest that hybrid methods, which combine norm-based geometry with structured preconditioning, can yield tangible improvements in both robustness and performance. The framework also highlights a rich, underexplored design space for future optimizers, including the potential for further integration of higher-order information and alternative norm geometries.

From a practical perspective, the explicit characterization of invariance properties can inform the choice of optimizer for specific architectures and data regimes, especially in settings where input scaling or reparameterization is common.

Conclusion

This work establishes a comprehensive geometric framework for optimization in deep learning, unifying steepest descent, quasi-Newton, and adaptive methods via preconditioned matrix norms. The framework enables the first systematic analysis of affine and scale invariance in the matrix setting and motivates new hybrid optimizers, MuAdam and MuAdam-SANIA, which demonstrate strong empirical performance. The results indicate that further exploration of generalized norms and structured preconditioning is a promising direction for advancing optimization in large-scale machine learning.