Advantage of AOL preconditioning in high-iteration Newton–Schulz regimes

Determine whether Almost Orthogonal Layer (AOL) preconditioning provides any advantage when the Newton–Schulz orthogonalization is run with a high number of iterations (large t), specifically in terms of convergence behavior and approximation accuracy of the polar factor, compared to Frobenius normalization within orthogonality-based optimizers such as Muon and Turbo-Muon.

Background

The paper shows that AOL preconditioning significantly improves convergence and reduces polar error in low-iteration regimes (t ≤ 5), enabling the removal of one Newton–Schulz iteration without loss of accuracy and yielding runtime benefits. It also decomposes the remaining error into an approximation term and a bias term introduced by AOL, and proves the preconditioned update remains a strict descent direction.

However, when the iteration count is large, the authors note that while the bias persists, the practical benefits of preconditioning may be absorbed. They explicitly state uncertainty about whether AOL preconditioning offers any advantage in high-t regimes, leaving this as an unresolved question.