---
title: Memoryless Stability-Annealed Smoothed-Sign Descent
url: https://www.emergentmind.com/topics/memoryless-stability-annealed-smoothed-sign-descent
type: topic
---

# Memoryless Stability-Annealed Smoothed-Sign Descent

Searching arXiv for the most relevant papers on “Memoryless Stability-Annealed Smoothed-Sign Descent” and closely related smoothed-sign / sign-based optimization work.
{"query":"\"Memoryless Stability-Annealed Smoothed-Sign Descent\" OR SoftSignum SoftMuon smoothed sign descent stability annealing arXiv","max_results":10,"sort_by":"submittedDate"}
{"query":"\"A Mirror Descent Perspective of Smoothed Sign Descent\" arXiv OR \"Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent\" arXiv","max_results":10,"sort_by":"submittedDate"}
Memoryless Stability-Annealed Smoothed-Sign Descent denotes a family of sign-based optimization procedures that replace the discontinuous sign map by a bounded smooth surrogate, regulate the transition between sign-like and magnitude-sensitive behavior through an annealed stability parameter, and retain a low-state or explicitly memoryless update structure. In the formulation underlying SoftSignum and SoftMuon, the principle combines a temperature-controlled soft-sign map, a robust quantile-based annealing schedule, and a geometry-relaxation analysis based on strongly convex regularizers and Fenchel conjugates [2605.31371]. In closely related work, the same phrase or directly adjacent terminology also refers to updates of the form $g/(|g|+\epsilon_t)$ with an exponentially annealed stability constant, for which the normalized iterates admit an exact entropic mirror-ascent description and converge to a rate-indexed barrier solution on separable linear classification problems [2607.06013].

## 1. Terminological scope and defining features

The expression combines four ideas that recur across the recent literature. “Smoothed-sign” refers to a continuous map that saturates toward $\operatorname{sign}(\cdot)$ for large arguments but remains linear or approximately linear near zero. “Stability-annealed” refers to the deliberate evolution of a temperature, stability constant, or coordinate-wise scale so that late-phase oscillation is reduced and terminal convergence is improved. “Memoryless” refers to minimal auxiliary state, although the exact meaning is paper-dependent. In SoftSignum and SoftMuon, the auxiliary state is a single momentum buffer together with two optional scalars $(\mu,\sigma)$ for temperature scheduling, with no second-moment accumulators or per-parameter statistics [2605.31371]. In the full-batch separable-data analysis of smoothed-sign descent, memoryless is stricter: there is no past-gradient accumulation, no first moments, no second moments, and no RMSProp/AdaGrad state [2607.06013].

This terminological variation is important because the family is not a single algorithm with a single update rule. One branch studies minimal-state training optimizers for deep learning, with momentum retained but second-moment state removed [2605.31371]. A second branch studies fully memoryless smoothed-sign dynamics in linear classification and identifies their implicit bias through a rate-controlled stability schedule [2607.06013]. A third branch reinterprets sign-based adversarial optimization through monotonically decreasing coordinate-wise step-sizes, where the effective direction becomes a tempered sign without introducing an explicit analytic relaxation of $\operatorname{sign}$ [2602.19096].

A recurrent misconception is that all such methods are simply signSGD with a smaller learning rate. The papers instead describe a geometric interpolation. In SoftSignum, the update saturates to $\pm 1$ for large coordinates while remaining linear near zero, so the same iteration behaves sign-like on large or noisy coordinates and SGD-like on small or converged coordinates [2605.31371]. In the separable-data theory, the stability constant selects a point on a rate-indexed barrier path between an $\ell_2$ hard-margin limit and an $\ell_\infty$ max-margin limit, so annealing changes the asymptotic geometry rather than only the transient step size [2607.06013].

## 2. Smoothed-sign maps and update rules

The canonical SoftSignum construction smooths the hard sign by an elementwise temperature-controlled saturating map $s_T$. Two scalar instantiations are used. The hyperbolic soft-sign is
$$
s_T(x)=\tanh(Tx),
$$
and the algebraic soft-sign is
$$
s_T(x)=\frac{Tx}{\sqrt{1+T^2x^2}}.
$$
Both are strictly increasing, satisfy $\sup_x |s_T'(x)|=T$, obey $|s_T(x)|\le 1$, converge to $\operatorname{sign}(x)$ as $T\to\infty$, and satisfy $s_T(x)\approx Tx$ as $T\to 0$ [2605.31371].

SoftSignum uses momentum
$$
m_t=\beta m_{t-1}+(1-\beta)g_t
$$
with decoupled weight decay $\lambda$, and the coordinate-wise update
$$
\theta_{t+1}^{(i)}=(1-\eta_t\lambda)\theta_t^{(i)}-\eta_t s_{T_t}(m_{t+1}^{(i)}).
$$
This is contrasted directly with signSGD, where $u_t^{(i)}=\operatorname{sign}(m_{t+1}^{(i)})$, and with SGD, where $u_t^{(i)}=m_{t+1}^{(i)}$ [2605.31371]. The intended effect is bounded, robust steps for large coordinates together with fine-grained contraction near zero.

The matrix-valued extension SoftMuon applies the same idea spectrally. For a momentum matrix $M=U\operatorname{diag}(\sigma)V^\top$, the spectral soft-sign map is
$$
\Phi_T(M)=U\operatorname{diag}\!\left(\frac{T\sigma_i}{\sqrt{1+T^2\sigma_i^2}}\right)V^\top
= M(M^\top M+T^{-2}I)^{-1/2},
$$
with computation via Newton–Schulz iterations rather than an explicit SVD in practice [2605.31371].

Related papers use other smoothed-sign parameterizations. In the linear-classification analysis, the memoryless update is
$$
w_{t+1}=w_t+\eta_t\frac{-\nabla L(w_t)}{|\nabla L(w_t)|+\epsilon_t},
$$
with absolute values and divisions taken coordinatewise [2607.06013]. In the mirror-descent analysis for overparameterized regression, the discrete-time update is
$$
w_{t+1}=w_t-\eta\frac{\nabla_w L(w_t)}{|\nabla_w L(w_t)|+\epsilon\mathbf{1}},
$$
and the corresponding coordinate-wise smoothed-sign map is $\phi_\epsilon(g)=g/(|g|+\epsilon)$ [2410.14158]. In the adversarial-optimization reinterpretation, the effective direction is $D_t v_{t+1}$ with
$$
d_{t,i}=\min\!\left(\frac{1}{|v_{t+1,i}|},\,d_{t-1,i}\right),
$$
so the method recovers $\operatorname{sign}(v_{t+1})$ early and becomes a tempered or “smoothed” sign later as the diagonal scale shrinks monotonically [2602.19096].

## 3. Stability annealing and low-state scheduling

In SoftSignum, stability annealing is implemented through a temperature schedule designed to keep a prescribed fraction of coordinates near saturation while progressively relaxing the geometry. Training progress is parameterized by $\alpha_t=t/N$ and
$$
p_t=\frac{\alpha_t-\alpha_{\mathrm{sign}}}{1-\alpha_{\mathrm{sign}}}
$$
clipped to $[0,1]$. For $\alpha_t<\alpha_{\mathrm{sign}}$ the method remains in a pure sign regime; for $\alpha_t\ge \alpha_{\mathrm{sign}}$ it anneals. If $q_{p_t}$ is a magnitude threshold derived from the distribution of $|m_{t+1}^{(i)}|$, the temperature is chosen by
$$
\tanh(T_t q_{p_t})=1-\epsilon
\quad\Rightarrow\quad
T_t=\operatorname{arctanh}(1-\epsilon)/q_{p_t},
$$
followed by the lower clip $T_t\leftarrow \max\{1,T_t\}$ [2605.31371].

The schedule is deliberately robust. At $\alpha_{\mathrm{sign}}$, the method computes
$$
\mu\leftarrow \operatorname{median}_i(m_{t+1}^{(i)}),\qquad
\sigma\leftarrow \operatorname{median}_i|m_{t+1}^{(i)}-\mu|,
$$
approximates the magnitude distribution by a folded Cauchy $|\operatorname{Cauchy}(\mu,\sigma)|$, and then obtains subsequent quantiles $q_{p_t}$ from that parametric model. An optional EMA refinement with decay $\rho$ is described, but the one-shot estimate is reported as sufficient in practice [2605.31371]. The resulting state remains minimal: one momentum buffer plus two optional scalars, with no second-moment or per-parameter buffers [2605.31371].

The separable-data theory uses a different annealing variable. It defines cumulative learning-rate time
$$
S_t=\sum_{s=0}^{t-1}\eta_s
$$
and an exponentially decaying stability constant
$$
\epsilon_t=\epsilon_0\exp(-\kappa S_t),
$$
where $\kappa$ is the annealing rate parameter [2607.06013]. Here the stability schedule is not a heuristic overlay on training dynamics; it is the quantity that indexes the implicit-bias selector.

The adversarial-optimization literature replaces temperature annealing by coordinate-wise step-size annealing. The rule
$$
d_{t,i}=\min\!\left(\frac{1}{|v_{t+1,i}|},\,d_{t-1,i}\right)
$$
enforces monotonically decreasing coordinate-wise step-sizes, and the global schedule uses $\alpha_t=\gamma/\sqrt{t}$ in theory or $\alpha_t=(\epsilon\cdot \gamma)/T$ in practice [2602.19096]. This construction is memoryless in the I-FGSM-style variant except for the stored diagonal scale, and it is explicitly motivated by the observation that sign-based per-coordinate step-sizes can blow up as gradients shrink near optima [2602.19096].

A further nuance appears in the mirror-descent treatment of smoothed sign descent for regression. That paper analyzes fixed $\epsilon$, not $\epsilon_t$, but proves that within a non-degenerate interval larger $\epsilon$ tightens the upper bound on the KKT error. This suggests a stability-annealed design in which $\epsilon$ is increased during the convergence stage, although the time-varying schedule itself is not part of the formal theorem [2410.14158].

## 4. Geometry, duality, and convergence theory

The modern theory of smoothed-sign descent is geometric rather than purely heuristic. SoftSignum derives its updates from a proximal-like subproblem
$$
d_t\in\arg\min_{d\in\mathcal{D}}
\left\{\langle m_t,d\rangle+\frac{1}{T_t}V(d)\right\},
$$
where $V$ is proper, closed, convex, and $1$-strongly convex on $\mathcal{D}$ [2605.31371]. With an entropic regularizer on $(-1,1)^d$, the first-order condition yields $d_t^{(i)}=-\delta_t\tanh(T_t m_t^{(i)})$; with the algebraic regularizer $V(u)=\sum_i[1-\sqrt{1-u_i^2}]$, it yields the algebraic soft-sign [2605.31371]. The same framework gives the mirror-descent/Fenchel identity
$$
d_t=\nabla V^*(-T_t m_t),
$$
with conjugates
$$
V_{\tanh}^*(y)=\sum_i \ln[\cosh(y_i)],\qquad
V_{\mathrm{alg}}^*(y)=\sum_i(\sqrt{1+y_i^2}-1).
$$

Under $L$-smoothness of $f$, bounded gradient-noise variance $\sigma^2$, momentum $m_t=\beta m_{t-1}+(1-\beta)g_t$, and the step-size condition
$$
\delta_t T_t\le \frac{1}{2L}\min\!\left\{1,\frac{1-\beta}{\beta}\right\},
$$
SoftSignum obtains the stochastic non-convex guarantee
$$
\frac{1}{K}\sum_{t=0}^{K-1}\frac{1}{T_t^2}\,
\mathbb{E}\big[V^*(-T_t\nabla f(\theta_t))\big]
\le
\frac{f(\theta_0)-f^*}{K\bar{\delta}\bar{T}}
+(1-\beta)\sigma^2
+\frac{\|\nabla f(\theta_0)\|_2^2}{K(1-\beta)},
$$
with the corresponding matrix-valued analogue for SoftMuon obtained by lifting $V$ to a spectral function and using Lewis’s conjugacy identity [2605.31371]. The analysis emphasizes that $V^*(\cdot)$ is quadratic for small gradients and linear for large ones, yielding a mixed $\ell_1/\ell_2$ progress measure [2605.31371].

A separate line of work studies smoothed sign descent through mirror descent. For overparameterized regression with diagonal-linear parameterization, the dynamics admit a time-varying mirror map
$$
\Phi_t(\beta)=\frac{2}{3}\sum_{i=1}^D \big(|\beta_i|+v_{i,t}^2\big)^{3/2},
$$
and the dual dynamics become a sign-modulated, smoothed update rather than a pure gradient flow [2410.14158]. The convergent solution satisfies $\delta$-KKT conditions for minimizing a Bregman-divergence-style function $E(\beta,\beta_0)$ subject to $X\beta=y$, and the paper shows that the KKT residual bound decreases linearly with $\epsilon$ on a stated interval [2410.14158].

For full-batch linear classification on separable data with weighted exponential loss, the dual structure is exact. Defining
$$
L(w)=\sum_{i=1}^n a_i e^{-z_i^\top w},
\qquad
\epsilon_t=\epsilon_0 e^{-\kappa S_t},
$$
the memoryless smoothed-sign update can be rewritten exactly as entropic mirror ascent on the concave dual objective
$$
D(\lambda)=\kappa \mathbf{1}^\top\lambda-\Phi^*(Z^\top\lambda),\qquad \lambda\ge 0,
$$
with a KL recursion controlling the dual gap [2607.06013]. The theory yields the normalized-iterate envelope
$$
\left\|\frac{w_t}{S_t}-u_\kappa\right\|_2
\le
\frac{\|w_0\|_2}{S_t}
+
\sqrt{\frac{C_\Phi C_D}{S_t}},
$$
which is an $S_t^{-1/2}$ envelope with an $S_t^{-1}$ initialization term [2607.06013].

## 5. Implicit bias, geometry selection, and favorable regimes

The most explicit implicit-bias theorem in this area is the separable-data result for stability-annealed smoothed-sign descent. For $0<\kappa<\gamma_\infty$, where
$$
\gamma_\infty=\max_{\|u\|_\infty\le 1}\gamma(u),
\qquad
\gamma(u)=\min_i z_i^\top u,
$$
the normalized iterates converge to the unique minimizer of the Burg-type barrier program
$$
u_\kappa=\arg\min_{u\in(-1,1)^d} B(u)
\quad\text{s.t.}\quad
z_i^\top u\ge \kappa\ \forall i,
$$
where
$$
B(u)=\sum_{j=1}^d[-|u_j|-\log(1-|u_j|)].
$$
As $\kappa\downarrow 0$, $u_\kappa/\kappa\to v_2$, the Euclidean hard-margin separator; as $\kappa\uparrow \gamma_\infty$, $\operatorname{dist}(u_\kappa,U_\infty)\to 0$, where $U_\infty$ is the $\ell_\infty$ max-margin solution set [2607.06013]. Stability annealing therefore selects a rate-indexed barrier path between gradient-descent-like and sign-like implicit biases.

This picture aligns with broader geometric analyses of sign methods. Sign gradient descent is steepest descent with respect to the $\ell_\infty$ norm, and its improvement relative to Euclidean gradient descent depends on the Hessian geometry. The sign-based regime is preferable when the Hessian is to some degree concentrated on its diagonal and its maximal eigenvalue is much larger than the average eigenvalue; the paper identifies both properties as common in deep networks [2002.08056]. That result explains why annealed smoothed-sign methods are naturally framed as interpolations between $\ell_\infty$-type and Euclidean-like geometries rather than as simple learning-rate modifications.

A complementary data-distribution argument appears in next-token prediction under heavy-tailed class imbalance. In the softmax unigram model with Zipf’s law $p_k\propto k^{-1}$, the paper proves faster convergence of coordinate-wise sign descent over normalized GD. The norm-dependent complexity satisfies
$$
C(f,\|\cdot\|_\infty)=2\log^2 d \ll d \sim C(f,\|\cdot\|_2),
$$
yielding
$$
f(\theta_T)-f^*\le \frac{2\log^2 d}{T+1}
\quad\text{for sign descent,}
\qquad
f(\theta_T)-f^*\le \frac{d}{T+1}
\quad\text{for normalized GD}
$$
under the stated schedules [2512.00763]. A plausible implication is that annealed smoothed-sign methods can exploit the same heavy-tailed coordinate structure while softening the terminal oscillations that pure sign descent may incur.

## 6. Implementations, empirical evidence, and limitations

The best-developed deep-learning instantiations are SoftSignum and SoftMuon. Their reported empirical evidence spans next-character prediction, graph learning, and LLM pretraining. On the Transformer next-character task, SoftMuon reaches $57.84\%$ accuracy versus Muon $57.36\%$, while SoftSignum reaches $57.05\%$ versus AdamW $56.91$ and Signum $56.17$ [2605.31371]. On a 130M LLaMA model trained on C4, SoftMuon reaches validation perplexity $18.288$ versus Muon $18.591$; SoftSignum reaches $18.519$ versus Signum $18.688$ and AdamW $18.709$ [2605.31371]. On FineWeb-Edu, the reported SoftMuon perplexities are $12.387$ versus Muon $12.541$ for 360M and $16.216$ versus Muon $16.362$ for 720M [2605.31371].

The practical recommendations are correspondingly specific. The papers recommend reusing signSGD or Muon learning-rate baselines, choosing $\alpha_{\mathrm{sign}}=0.9$ and $\epsilon=10^{-4}$ as defaults, clipping $T$ from below at $1$, using one-shot folded-Cauchy quantile estimation at $\alpha_{\mathrm{sign}}$, and taking momentum $\beta$ in the range $0.9$–$0.95$. For large-scale pretraining, global norm clipping such as $0.1$ remains beneficial, and SoftMuon uses approximately five Newton–Schulz iterations for the inverse square root [2605.31371].

The adversarial-optimization branch provides a distinct application domain and a different notion of stability annealing. On NIPS2017 with ResNet-50 as surrogate and Inception-v3 as target, the success rate for I-FGSM drops from $37.1\%$ at $t=2$ to $15.5\%$ at $t=20$, and the paper attributes this degradation to overshooting or oscillations caused by non-decaying per-coordinate step-sizes [2602.19096]. The proposed MDCS variants enforce monotone coordinate-wise scales and report consistent transferability gains across image classification, cross-modal retrieval, and VQA [2602.19096]. That work also stresses a limitation of the smoothed-sign label in this setting: the method does not introduce an explicit smoothing of the sign operator, but rather an implicit regularization through annealed coordinate-wise scaling [2602.19096].

The formal scope of current theorems remains narrow. The rate-indexed barrier-path result assumes full-batch linear classification on separable data, weighted exponential loss, and the memoryless update with $\epsilon_t=\epsilon_0 e^{-\kappa S_t}$ [2607.06013]. The mirror-descent implicit-bias analysis assumes overparameterized regression with block-diagonal structure in $X^\top X$ and a specific diagonal-linear parameterization [2410.14158]. The adversarial MDCS theorem assumes concavity of the local objective on the $L_\infty$-budget set together with bounded gradients and bounded diameter [2602.19096]. For SoftSignum and SoftMuon, the reported caveats are operational rather than asymptotic: too-small $T$ early reduces effective step size and slows progress, too-large $\epsilon$ weakens terminal relaxation, and mis-specified spectral groups in SoftMuon can underperform Muon [2605.31371].

Taken together, these results present Memoryless Stability-Annealed Smoothed-Sign Descent not as a single optimizer but as a coherent research program. Across its variants, the central claim is consistent: replace hard sign by a bounded smooth map or an annealed sign-equivalent scaling, schedule the stability parameter so that late-stage geometry relaxes, and use the resulting interpolation to preserve the robustness and low-state profile of sign-based methods while recovering finer terminal behavior [2605.31371].

Source: https://www.emergentmind.com/topics/memoryless-stability-annealed-smoothed-sign-descent