Norm-Constrained Flows and Sign-Based Optimization: Theory and Algorithms

Abstract: Sign Gradient Descent (SignGD) is a simple yet robust optimization method, widely used in machine learning for its resilience to gradient noise and compatibility with low-precision computations. While its empirical performance is well established, its theoretical understanding remains limited. In this work, we revisit SignGD from a continuous-time perspective, showing that it arises as an Euler discretization of a norm-constrained gradient flow. This viewpoint reveals a trust-region interpretation and connects SignGD to a broader class of methods defined by different norm constraints, such as normalized gradient descent and greedy coordinate descent. We further study the discontinuous nature of the underlying dynamics using Filippov's differential inclusion framework, which allows us to derive new algorithmic variants, such as the convex-combination sliding update for the $\ell_1$-constrained flow, that faithfully approximate Filippov solutions even at discontinuity points. While we do not provide convergence guarantees for these variants, we demonstrate that they preserve descent properties and perform well empirically. We also introduce an accelerated version of SignGD based on a momentum-augmented discretization of the sign-gradient flow, and show its effectiveness in practice. Finally, we establish provable convergence guarantees for standard SignGD in the setting of strongly convex optimization. Our results provide new geometric, algorithmic, and analytical insights into SignGD and its norm-constrained extensions.