A priori selection of the settling parameter

Determine an a priori procedure for choosing the denominator parameter ε in scale-free trust-bounded Riemannian Adam so that Stiefel attention frames cease moving at the intended gradient scale.

Background

The corrected Riemannian Adam update is designed to remain effectively scale-free when gradients are larger than ε. This prevents the attention frames from freezing but also means that the optimizer may continue making nearly full-sized steps when gradients have become small, causing late-training motion and preventing settling.

The paper explains that ε should be selected according to the gradient scale below which frame motion is intended to stop. Nevertheless, the authors state that they cannot yet determine this value a priori and identify that inability as the method’s most important open issue.

References

Remark~\ref{rem:settling} gives the remedy---choose $\varepsilon$ at the gradient scale below which motion should cease---but we cannot yet set it a priori, which makes this the most important open issue with the method.

— Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not  (2609.19363 - Guerrero, 16 Sep 2026) in Section 8, Limitations, first bullet; see also Remark “ε is the settling threshold, not numerical hygiene”