Memoryless Stability-Annealed Smoothed-Sign Descent

• Memoryless Stability-Annealed Smoothed-Sign Descent is a family of sign-based optimization methods that smooths the hard sign map while employing an annealed stability parameter to improve convergence.
• It leverages temperature-controlled soft-sign maps and a minimal, memoryless update structure to balance robust, sign-like steps with fine, magnitude-sensitive refinements.
• The approach interpolates between ℓ∞ and Euclidean geometries, yielding enhanced convergence properties for deep learning and linear classification tasks.

Searching arXiv for the most relevant papers on “Memoryless Stability-Annealed Smoothed-Sign Descent” and closely related smoothed-sign / sign-based optimization work. {"query":"\"Memoryless Stability-Annealed Smoothed-Sign Descent\" OR SoftSignum SoftMuon smoothed sign descent stability annealing arXiv","max_results":10,"sort_by":"submittedDate"} {"query":"\"A Mirror Descent Perspective of Smoothed Sign Descent\" arXiv OR \"Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent\" arXiv","max_results":10,"sort_by":"submittedDate"} Memoryless Stability-Annealed Smoothed-Sign Descent denotes a family of sign-based optimization procedures that replace the discontinuous sign map by a bounded smooth surrogate, regulate the transition between sign-like and magnitude-sensitive behavior through an annealed stability parameter, and retain a low-state or explicitly memoryless update structure. In the formulation underlying SoftSignum and SoftMuon, the principle combines a temperature-controlled soft-sign map, a robust quantile-based annealing schedule, and a geometry-relaxation analysis based on strongly convex regularizers and Fenchel conjugates (Feoktistov et al., 29 May 2026). In closely related work, the same phrase or directly adjacent terminology also refers to updates of the form $g/(|g|+\epsilon_t)$ with an exponentially annealed stability constant, for which the normalized iterates admit an exact entropic mirror-ascent description and converge to a rate-indexed barrier solution on separable linear classification problems (Wang et al., 7 Jul 2026).

1. Terminological scope and defining features

The expression combines four ideas that recur across the recent literature. “Smoothed-sign” refers to a continuous map that saturates toward $\operatorname{sign}(\cdot)$ for large arguments but remains linear or approximately linear near zero. “Stability-annealed” refers to the deliberate evolution of a temperature, stability constant, or coordinate-wise scale so that late-phase oscillation is reduced and terminal convergence is improved. “Memoryless” refers to minimal auxiliary state, although the exact meaning is paper-dependent. In SoftSignum and SoftMuon, the auxiliary state is a single momentum buffer together with two optional scalars $(\mu,\sigma)$ for temperature scheduling, with no second-moment accumulators or per-parameter statistics (Feoktistov et al., 29 May 2026). In the full-batch separable-data analysis of smoothed-sign descent, memoryless is stricter: there is no past-gradient accumulation, no first moments, no second moments, and no RMSProp/AdaGrad state (Wang et al., 7 Jul 2026).

This terminological variation is important because the family is not a single algorithm with a single update rule. One branch studies minimal-state training optimizers for deep learning, with momentum retained but second-moment state removed (Feoktistov et al., 29 May 2026). A second branch studies fully memoryless smoothed-sign dynamics in linear classification and identifies their implicit bias through a rate-controlled stability schedule (Wang et al., 7 Jul 2026). A third branch reinterprets sign-based adversarial optimization through monotonically decreasing coordinate-wise step-sizes, where the effective direction becomes a tempered sign without introducing an explicit analytic relaxation of $\operatorname{sign}$ (Tao et al., 22 Feb 2026).

A recurrent misconception is that all such methods are simply signSGD with a smaller learning rate. The papers instead describe a geometric interpolation. In SoftSignum, the update saturates to $\pm 1$ for large coordinates while remaining linear near zero, so the same iteration behaves sign-like on large or noisy coordinates and SGD-like on small or converged coordinates (Feoktistov et al., 29 May 2026). In the separable-data theory, the stability constant selects a point on a rate-indexed barrier path between an $\ell_2$ hard-margin limit and an $\ell_\infty$ max-margin limit, so annealing changes the asymptotic geometry rather than only the transient step size (Wang et al., 7 Jul 2026).

2. Smoothed-sign maps and update rules

The canonical SoftSignum construction smooths the hard sign by an elementwise temperature-controlled saturating map $s_T$. Two scalar instantiations are used. The hyperbolic soft-sign is

$s_T(x)=\tanh(Tx),$

and the algebraic soft-sign is

$s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$

Both are strictly increasing, satisfy $\operatorname{sign}(\cdot)$0, obey $\operatorname{sign}(\cdot)$1, converge to $\operatorname{sign}(\cdot)$2 as $\operatorname{sign}(\cdot)$3, and satisfy $\operatorname{sign}(\cdot)$4 as $\operatorname{sign}(\cdot)$5 (Feoktistov et al., 29 May 2026).

SoftSignum uses momentum

$\operatorname{sign}(\cdot)$6

with decoupled weight decay $\operatorname{sign}(\cdot)$7, and the coordinate-wise update

$\operatorname{sign}(\cdot)$8

This is contrasted directly with signSGD, where $\operatorname{sign}(\cdot)$9, and with SGD, where $(\mu,\sigma)$0 (Feoktistov et al., 29 May 2026). The intended effect is bounded, robust steps for large coordinates together with fine-grained contraction near zero.

The matrix-valued extension SoftMuon applies the same idea spectrally. For a momentum matrix $(\mu,\sigma)$1, the spectral soft-sign map is

$(\mu,\sigma)$2

with computation via Newton–Schulz iterations rather than an explicit SVD in practice (Feoktistov et al., 29 May 2026).

Related papers use other smoothed-sign parameterizations. In the linear-classification analysis, the memoryless update is

$(\mu,\sigma)$3

with absolute values and divisions taken coordinatewise (Wang et al., 7 Jul 2026). In the mirror-descent analysis for overparameterized regression, the discrete-time update is

$(\mu,\sigma)$4

and the corresponding coordinate-wise smoothed-sign map is $(\mu,\sigma)$5 (Wang et al., 2024). In the adversarial-optimization reinterpretation, the effective direction is $(\mu,\sigma)$6 with

$(\mu,\sigma)$7

so the method recovers $(\mu,\sigma)$8 early and becomes a tempered or “smoothed” sign later as the diagonal scale shrinks monotonically (Tao et al., 22 Feb 2026).

3. Stability annealing and low-state scheduling

In SoftSignum, stability annealing is implemented through a temperature schedule designed to keep a prescribed fraction of coordinates near saturation while progressively relaxing the geometry. Training progress is parameterized by $(\mu,\sigma)$9 and

$\operatorname{sign}$0

clipped to $\operatorname{sign}$1. For $\operatorname{sign}$2 the method remains in a pure sign regime; for $\operatorname{sign}$3 it anneals. If $\operatorname{sign}$4 is a magnitude threshold derived from the distribution of $\operatorname{sign}$5, the temperature is chosen by

$\operatorname{sign}$6

followed by the lower clip $\operatorname{sign}$7 (Feoktistov et al., 29 May 2026).

The schedule is deliberately robust. At $\operatorname{sign}$8, the method computes

$\operatorname{sign}$9

approximates the magnitude distribution by a folded Cauchy $\pm 1$0, and then obtains subsequent quantiles $\pm 1$1 from that parametric model. An optional EMA refinement with decay $\pm 1$2 is described, but the one-shot estimate is reported as sufficient in practice (Feoktistov et al., 29 May 2026). The resulting state remains minimal: one momentum buffer plus two optional scalars, with no second-moment or per-parameter buffers (Feoktistov et al., 29 May 2026).

The separable-data theory uses a different annealing variable. It defines cumulative learning-rate time

$\pm 1$3

and an exponentially decaying stability constant

$\pm 1$4

where $\pm 1$5 is the annealing rate parameter (Wang et al., 7 Jul 2026). Here the stability schedule is not a heuristic overlay on training dynamics; it is the quantity that indexes the implicit-bias selector.

The adversarial-optimization literature replaces temperature annealing by coordinate-wise step-size annealing. The rule

$\pm 1$6

enforces monotonically decreasing coordinate-wise step-sizes, and the global schedule uses $\pm 1$7 in theory or $\pm 1$8 in practice (Tao et al., 22 Feb 2026). This construction is memoryless in the I-FGSM-style variant except for the stored diagonal scale, and it is explicitly motivated by the observation that sign-based per-coordinate step-sizes can blow up as gradients shrink near optima (Tao et al., 22 Feb 2026).

A further nuance appears in the mirror-descent treatment of smoothed sign descent for regression. That paper analyzes fixed $\pm 1$9, not $\ell_2$0, but proves that within a non-degenerate interval larger $\ell_2$1 tightens the upper bound on the KKT error. This suggests a stability-annealed design in which $\ell_2$2 is increased during the convergence stage, although the time-varying schedule itself is not part of the formal theorem (Wang et al., 2024).

4. Geometry, duality, and convergence theory

The modern theory of smoothed-sign descent is geometric rather than purely heuristic. SoftSignum derives its updates from a proximal-like subproblem

$\ell_2$3

where $\ell_2$4 is proper, closed, convex, and $\ell_2$5-strongly convex on $\ell_2$6 (Feoktistov et al., 29 May 2026). With an entropic regularizer on $\ell_2$7, the first-order condition yields $\ell_2$8; with the algebraic regularizer $\ell_2$9, it yields the algebraic soft-sign (Feoktistov et al., 29 May 2026). The same framework gives the mirror-descent/Fenchel identity

$\ell_\infty$0

with conjugates

$\ell_\infty$1

Under $\ell_\infty$2-smoothness of $\ell_\infty$3, bounded gradient-noise variance $\ell_\infty$4, momentum $\ell_\infty$5, and the step-size condition

$\ell_\infty$6

SoftSignum obtains the stochastic non-convex guarantee

$\ell_\infty$7

with the corresponding matrix-valued analogue for SoftMuon obtained by lifting $\ell_\infty$8 to a spectral function and using Lewis’s conjugacy identity (Feoktistov et al., 29 May 2026). The analysis emphasizes that $\ell_\infty$9 is quadratic for small gradients and linear for large ones, yielding a mixed $s_T$0 progress measure (Feoktistov et al., 29 May 2026).

A separate line of work studies smoothed sign descent through mirror descent. For overparameterized regression with diagonal-linear parameterization, the dynamics admit a time-varying mirror map

$s_T$1

and the dual dynamics become a sign-modulated, smoothed update rather than a pure gradient flow (Wang et al., 2024). The convergent solution satisfies $s_T$2-KKT conditions for minimizing a Bregman-divergence-style function $s_T$3 subject to $s_T$4, and the paper shows that the KKT residual bound decreases linearly with $s_T$5 on a stated interval (Wang et al., 2024).

For full-batch linear classification on separable data with weighted exponential loss, the dual structure is exact. Defining

$s_T$6

the memoryless smoothed-sign update can be rewritten exactly as entropic mirror ascent on the concave dual objective

$s_T$7

with a KL recursion controlling the dual gap (Wang et al., 7 Jul 2026). The theory yields the normalized-iterate envelope

$s_T$8

which is an $s_T$9 envelope with an $s_T(x)=\tanh(Tx),$0 initialization term (Wang et al., 7 Jul 2026).

5. Implicit bias, geometry selection, and favorable regimes

The most explicit implicit-bias theorem in this area is the separable-data result for stability-annealed smoothed-sign descent. For $s_T(x)=\tanh(Tx),$1, where

$s_T(x)=\tanh(Tx),$2

the normalized iterates converge to the unique minimizer of the Burg-type barrier program

$s_T(x)=\tanh(Tx),$3

where

$s_T(x)=\tanh(Tx),$4

As $s_T(x)=\tanh(Tx),$5, $s_T(x)=\tanh(Tx),$6, the Euclidean hard-margin separator; as $s_T(x)=\tanh(Tx),$7, $s_T(x)=\tanh(Tx),$8, where $s_T(x)=\tanh(Tx),$9 is the $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$0 max-margin solution set (Wang et al., 7 Jul 2026). Stability annealing therefore selects a rate-indexed barrier path between gradient-descent-like and sign-like implicit biases.

This picture aligns with broader geometric analyses of sign methods. Sign gradient descent is steepest descent with respect to the $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$1 norm, and its improvement relative to Euclidean gradient descent depends on the Hessian geometry. The sign-based regime is preferable when the Hessian is to some degree concentrated on its diagonal and its maximal eigenvalue is much larger than the average eigenvalue; the paper identifies both properties as common in deep networks (Balles et al., 2020). That result explains why annealed smoothed-sign methods are naturally framed as interpolations between $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$2-type and Euclidean-like geometries rather than as simple learning-rate modifications.

A complementary data-distribution argument appears in next-token prediction under heavy-tailed class imbalance. In the softmax unigram model with Zipf’s law $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$3, the paper proves faster convergence of coordinate-wise sign descent over normalized GD. The norm-dependent complexity satisfies

$s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$4

yielding

$s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$5

under the stated schedules (Yadav et al., 30 Nov 2025). A plausible implication is that annealed smoothed-sign methods can exploit the same heavy-tailed coordinate structure while softening the terminal oscillations that pure sign descent may incur.

6. Implementations, empirical evidence, and limitations

The best-developed deep-learning instantiations are SoftSignum and SoftMuon. Their reported empirical evidence spans next-character prediction, graph learning, and LLM pretraining. On the Transformer next-character task, SoftMuon reaches $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$6 accuracy versus Muon $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$7, while SoftSignum reaches $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$8 versus AdamW $s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.$9 and Signum $\operatorname{sign}(\cdot)$00 (Feoktistov et al., 29 May 2026). On a 130M LLaMA model trained on C4, SoftMuon reaches validation perplexity $\operatorname{sign}(\cdot)$01 versus Muon $\operatorname{sign}(\cdot)$02; SoftSignum reaches $\operatorname{sign}(\cdot)$03 versus Signum $\operatorname{sign}(\cdot)$04 and AdamW $\operatorname{sign}(\cdot)$05 (Feoktistov et al., 29 May 2026). On FineWeb-Edu, the reported SoftMuon perplexities are $\operatorname{sign}(\cdot)$06 versus Muon $\operatorname{sign}(\cdot)$07 for 360M and $\operatorname{sign}(\cdot)$08 versus Muon $\operatorname{sign}(\cdot)$09 for 720M (Feoktistov et al., 29 May 2026).

The practical recommendations are correspondingly specific. The papers recommend reusing signSGD or Muon learning-rate baselines, choosing $\operatorname{sign}(\cdot)$10 and $\operatorname{sign}(\cdot)$11 as defaults, clipping $\operatorname{sign}(\cdot)$12 from below at $\operatorname{sign}(\cdot)$13, using one-shot folded-Cauchy quantile estimation at $\operatorname{sign}(\cdot)$14, and taking momentum $\operatorname{sign}(\cdot)$15 in the range $\operatorname{sign}(\cdot)$16–$\operatorname{sign}(\cdot)$17. For large-scale pretraining, global norm clipping such as $\operatorname{sign}(\cdot)$18 remains beneficial, and SoftMuon uses approximately five Newton–Schulz iterations for the inverse square root (Feoktistov et al., 29 May 2026).

The adversarial-optimization branch provides a distinct application domain and a different notion of stability annealing. On NIPS2017 with ResNet-50 as surrogate and Inception-v3 as target, the success rate for I-FGSM drops from $\operatorname{sign}(\cdot)$19 at $\operatorname{sign}(\cdot)$20 to $\operatorname{sign}(\cdot)$21 at $\operatorname{sign}(\cdot)$22, and the paper attributes this degradation to overshooting or oscillations caused by non-decaying per-coordinate step-sizes (Tao et al., 22 Feb 2026). The proposed MDCS variants enforce monotone coordinate-wise scales and report consistent transferability gains across image classification, cross-modal retrieval, and VQA (Tao et al., 22 Feb 2026). That work also stresses a limitation of the smoothed-sign label in this setting: the method does not introduce an explicit smoothing of the sign operator, but rather an implicit regularization through annealed coordinate-wise scaling (Tao et al., 22 Feb 2026).

The formal scope of current theorems remains narrow. The rate-indexed barrier-path result assumes full-batch linear classification on separable data, weighted exponential loss, and the memoryless update with $\operatorname{sign}(\cdot)$23 (Wang et al., 7 Jul 2026). The mirror-descent implicit-bias analysis assumes overparameterized regression with block-diagonal structure in $\operatorname{sign}(\cdot)$24 and a specific diagonal-linear parameterization (Wang et al., 2024). The adversarial MDCS theorem assumes concavity of the local objective on the $\operatorname{sign}(\cdot)$25-budget set together with bounded gradients and bounded diameter (Tao et al., 22 Feb 2026). For SoftSignum and SoftMuon, the reported caveats are operational rather than asymptotic: too-small $\operatorname{sign}(\cdot)$26 early reduces effective step size and slows progress, too-large $\operatorname{sign}(\cdot)$27 weakens terminal relaxation, and mis-specified spectral groups in SoftMuon can underperform Muon (Feoktistov et al., 29 May 2026).

Taken together, these results present Memoryless Stability-Annealed Smoothed-Sign Descent not as a single optimizer but as a coherent research program. Across its variants, the central claim is consistent: replace hard sign by a bounded smooth map or an annealed sign-equivalent scaling, schedule the stability parameter so that late-stage geometry relaxes, and use the resulting interpolation to preserve the robustness and low-state profile of sign-based methods while recovering finer terminal behavior (Feoktistov et al., 29 May 2026).