Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer

Abstract: Matrix-based optimizers have demonstrated immense potential in training LLMs, however, designing an ideal optimizer remains a formidable challenge. A superior optimizer must satisfy three core desiderata: efficiency, achieving Muon-like preconditioning to accelerate optimization; stability, strictly adhering to the scale-invariance inherent in neural networks; and speed, minimizing computational overhead. While existing methods address these aspects to varying degrees, they often fail to unify them, either incurring prohibitive computational costs like Muon, or allowing radial jitters that compromise stability like RMNP. To bridge this gap, we propose Nora, an optimizer that rigorously satisfies all three requirements. Nora achieves training stability by explicitly stabilizing weight norms and angular velocities through row-wise momentum projection onto the orthogonal complement of the weights. Simultaneously, by leveraging the block-diagonal dominance of the Transformer Hessian, Nora effectively approximates structured preconditioning while maintaining an optimal computational complexity of $\mathcal{O}(mn)$. Furthermore, we prove that Nora is a scalable optimizer and establish its corresponding scaling theorems. With a streamlined implementation requiring only two lines of code, our preliminary experiments validate Nora as an efficient and highly promising optimizer for large-scale training.

• The paper presents Nora, a novel optimizer that projects momentum onto the row-tangent space to preserve scale invariance and accelerate convergence.
• It employs row-wise normalization to achieve optimal diagonal preconditioning with O(mn) complexity, outperforming methods like Muon and RMNP.
• Empirical results on LLMs show Nora's robust training stability, lower perplexity, and significant speed advantages over existing optimizers.

Normalized Orthogonal Row Alignment for Scalable Matrix Optimization

Introduction

The paper "Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer" (2605.03769) introduces Nora, a matrix-based optimizer explicitly designed for scalable deep learning, with a primary focus on large-scale LLM pretraining. Traditional adaptive optimizers such as Adam treat model parameters as unstructured vectors, missing critical interaction structure prevalent in the loss landscapes of modern DNNs. State-of-the-art matrix optimizers like Muon and RMNP attempt to leverage this structure but either incur prohibitive $O(m^2 n)$ computational costs or insufficiently address scale invariance and instability caused by inappropriate handling of radial updates.

Nora is motivated by three optimizer desiderata: efficiency (strong preconditioning for rapid convergence), stability (consistent with scale invariance of normalized networks), and speed (low computational complexity compatible with highly-parallel LLM workloads). To this end, Nora introduces a row-projection and normalization paradigm that ensures strict alignment to the row-tangent space, maintaining optimal preconditioning effect while enforcing scale-invariant dynamics.

Matrix-Based Optimizer Landscape

Matrix-based optimizers extend beyond per-parameter methods by considering the full structure of weight matrices. Muon leverages orthogonalized updates and Newton–Schulz iterations to achieve a preconditioning matrix that approximates the local Hessian structure, but suffers excessive computational cost $O(m^2 n)$. RMNP acknowledges row-block diagonal dominance of transformer Hessians and achieves $O(mn)$ cost through row-wise normalization, but does not remove destabilizing radial updates, thus compromising training stability.

Both Muon and RMNP fail to consistently enforce row-wise orthogonality with respect to the weights, leading to radial momentum injection and oscillations in weight norms under scale-invariant parametrizations (as induced by BatchNorm, LayerNorm, or RMSNorm). The geometric implication is that each update perturbs the effective angular learning rate, thereby undermining both convergence and network smoothness.

Nora: Methodology and Algorithmic Details

Nora's central innovation is a two-step matrix update: first, the exponential moving average momentum is projected onto the row-wise tangent space to the weight matrix. This is given for each row $i$ by removing the component parallel to $w_{t,i:}$. Second, the resulting matrix undergoes row-wise normalization, scaling each row to unit $\ell_2$ norm. Algebraically, the update direction $d_t$ can be written:

$d_t = \textrm{RowNorm}(v_t^\perp)$

where $v_t^\perp$ is the row-wise perpendicular projection of the momentum.

This procedure guarantees:

• Angular updates: All weight changes are strictly in directions tangent to the current weight norm sphere, preserving scale at each step.
• Diagonal preconditioning: The row-block diagonal dominance of Transformer Hessians justifies efficient approximation of Muon-level preconditioners via simple row-wise normalization, exploiting spectrum concentration for practical efficiency.
• Computational efficiency: The complexity is $O(mn)$, matching the cost profile of SGD/Adam implementations, and the entire logic is collapsible to two matrix code lines, facilitating integration into existing training stacks.

Through maximal update parametrization ($O(m^2 n)$0P) analysis, the paper rigorously proves the optimal scaling of the learning rate with model width for stable feature learning, ensuring that Nora remains effective as layer sizes scale up.

Theoretical Guarantees

A robust convergence analysis is provided under both Frobenius and $O(m^2 n)$1-metric Hessian smoothness. The main theoretical results are:

• Scaling law: The optimal Nora update step size obeys $O(m^2 n)$2 in the infinite-width regime, ensuring non-vanishing, non-exploding feature updates.
• Non-convex convergence: Nora satisfies first-order stationarity bounds analogous to those of RMNP and Adam under appropriate smoothness conditions. Explicit iteration complexity results are proven for $O(m^2 n)$3-stationary points in both projected and standard norms.
• Row-wise orthogonality: The update direction in Nora is always perpendicular to the weights at the row level, which the analysis shows is a necessary condition for fully leveraging scale invariance.

These theoretical claims are presented with formal proofs, utilizing norm-preserving projection and row-block structure arguments that generalize beyond Nora's immediate design.

Empirical Evaluation

Nora is evaluated on LLaMA-style Transformer models at 60M and 135M parameter scales in direct comparison with Muon, Mano, and RMNP. Across an extensive grid of matrix learning rates, Nora demonstrates:

• The lowest final validation loss and perplexity versus all compared baselines at both model scales.
• More consistent late-stage training improvement, with continued loss and perplexity decay observed after other optimizers plateau.
• Wall-clock efficiency: In CUDA bfloat16 benchmarks, Nora's row normalization is an order of magnitude faster than Muon's Newton-Schulz iteration, and the speed advantage grows with larger matrix dimensions (e.g., $O(m^2 n)$4 for 1B-scale matrices).

Additional ablation studies confirm that Nora's performance is robust to weight decay schemes and not contingent on hyperparameter tuning. The core effectiveness is intrinsic to its geometric update structure.

Comparison with Related Optimizers

Mano alternates row- and column-wise orthogonal projections with normalization, aiming to capture both row and column symmetries. However, Nora's exclusive focus on row-level structure is empirically and theoretically justified by the observed dominance of row-block diagonal structure in Transformer Hessians. Further, Nora dispenses with alternating parity logic, simplifying implementation and improving runtime.

Compared to SSO and AdamP, Nora's approach is more directly aligned with the geometric structure induced by normalization-based architectures, avoiding computationally intensive routines such as bisection or approximate manifold projections.

Implications and Future Directions

Nora's design unifies three primary optimizer desiderata—preconditioning efficiency, scale-invariant stability, and computational speed—within a minimal and theoretically transparent algorithm. The implications for scalable deep learning are significant: future LLM and foundation model training workflows can adopt Nora as a drop-in replacement for classical optimizers, achieving faster convergence and improved training stability, especially in the presence of normalization-heavy architectures.

Beyond immediate deployment, Nora provides a template for further research into optimizer design that explicitly leverages geometric symmetries in parametric spaces. Potential directions include extension of projection-based optimizers to other symmetry types (e.g., permutation, block-orthogonal), adaptation for distributed/federated optimization scenarios, and integration with structured second-order preconditioners for even more aggressive conditioning.

Conclusion

Nora advances scalable LLM training by introducing a normalized orthogonal row-alignment approach that rigorously addresses efficiency, stability, and speed within a unifying geometric framework. It is unique in guaranteeing scale-invariant, Muon-like preconditioning at the computational cost of row normalization, simultaneously establishing strong theoretical convergence and scaling properties. Empirical results confirm superior performance and runtime compared to leading matrix optimizers. Nora represents a principled, robust optimizer for modern deep neural architectures, and sets a clear trajectory for future developments in structure-aware optimization methodologies.