Scalability of AURA to billion-parameter networks

Assess whether the empirical results obtained with AURA extend to neural networks with billions of parameters, particularly in view of the additional memory required by AURA’s per-parameter state.

Background

The experiments evaluate AURA on relatively small fully connected complex-valued neural networks. AURA stores additional per-parameter quantities beyond the state of its base optimizer, creating a memory overhead that may become more consequential as model size increases.

The authors explicitly leave unresolved whether the observed optimization benefits and practical behavior of AURA extend to networks with billions of parameters, where the memory overhead is expected to matter most.

References

Finally, for clear computational cost reasons, we tested the proposed method on relatively small networks; whether the results extend to networks with billions of parameters, where the memory overhead matters most, remains to be assessed.

— AURA: Angular Update Rate Adaptation for training complex-valued neural networks  (2609.25914 - Ballini et al., 22 Sep 2026) in Section Conclusions