Scaling DeltaMomentum beyond the reported experiments

Establish the performance and applicability of DeltaMomentum at scales larger than 1 billion parameters and in non-language generative domains, including image generation and reinforcement learning.

Background

The experiments evaluate DeltaMomentum on LLMs up to 1 billion parameters and on CIFAR-10 classification tasks. The authors explicitly identify larger-scale evaluations and non-language generative applications as unresolved areas, particularly because the current evidence does not establish whether the reported optimization benefits transfer to substantially larger models or to generative settings such as image synthesis and reinforcement learning.

References

Experiments cover language modeling up to 1B parameters and CIFAR-10 classifiers, so larger scale and non-language generative domains such as image generation and reinforcement learning remain open.

— DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule  (2608.19491 - Hong et al., 19 Aug 2026) in Section 7, paragraph "Limitations and Future Work"