Reconciling the proper bases for spectral normalization and variance adaptation

Investigate how to reconcile the choice of basis in which spectral normalization and variance adaptation are applied within matrix-whitening optimizers, specifically whether variance adaptation should be performed in the rotated eigenbasis (as in SOAP) or in the original elementwise basis after orthogonalization (as in AdaMuon).

Background

The paper shows that variance adaptation provides benefits regardless of the basis in which it is applied: in SOAP, adaptation occurs in the rotated eigenbasis, while in AdaMuon it is applied elementwise after Newton–Schulz orthogonalization. This suggests matrix-whitening serves two roles—spectral normalization and variance adaptation—that can be decoupled.

The authors present preliminary evidence (including a didactic SPA variant) but note that properly reconciling the basis choices for these two components remains unresolved and requires further investigation.