---
title: 'SignSoftSGD (S3): Adaptive Optimizer'
url: https://www.emergentmind.com/topics/signsoftsgd-s3
type: topic
---

# SignSoftSGD (S3): Adaptive Optimizer

Searching arXiv for the cited optimizer papers to ground the article in the current record.
{"query":"arXiv 2507.06464 SoftSignSGD(S3) An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam", "max_results": 5}
to=arxiv_search  东臣on?

to=arxiv_search {"query":"2507.06464 SoftSignSGD(S3)", "max_results": 10}
SignSoftSGD (S3), titled “SoftSignSGD (S3)” in the paper and also referred to in some passages as “SignSoftSGD (S3),” is a sign-like adaptive optimizer proposed for practical deep neural network training and loss-spike minimization beyond Adam. Its defining design choices are a flexible $p$-th order moment in the denominator, a unified exponential moving average (EMA) coefficient for numerator and denominator, and an equivalent Nesterov’s accelerated gradient module. The method is intended to preserve the robustness of sign-based updates under large gradient fluctuations while preventing uncontrolled coordinate-wise scaling; in the unified-EMA setting, each coordinate of its normalized update is provably bounded in $[-1,1]$ [2507.06464].

## 1. Nomenclature and disambiguation

The paper’s title and Algorithm 1 use **SoftSignSGD (S3)**, while some prose uses **SignSoftSGD (S3)**. The authors note that “SoftSignSGD” was chosen because the per-parameter update is inherently bounded in $[-1,1]$, which they describe as softsign-like behavior. The intended short name is **S3**, and the “SignSoftSGD” form is a naming inconsistency in the prose [2507.06464].

The acronym **S3** is not unique in the sign-optimization literature. In “Sparse-SignSGD with Majority Vote,” $S^3$ denotes **Sparse-SignSGD**, a distributed top-$K$ sign-compression method with majority-vote aggregation rather than a soft-sign adaptive optimizer [2302.07475]. In “Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling,” the relevant optimizer is **SoftSignum**, a temperature-controlled $\tanh$ relaxation of Signum; that paper does not use the S3 name for the method itself [2605.31371]. A separate 2025 paper on parameter-free Sign-SGD likewise does **not** introduce a soft-sign operator and uses only the hard coordinate-wise $\operatorname{sign}(\cdot)$ map [2506.03725].

This nomenclatural overlap matters because the term “SignSoftSGD (S3)” can otherwise be conflated with three distinct lines of work: bounded adaptive sign-like optimization, smooth sign relaxation, and sparse distributed sign compression.

## 2. Update rule and optimizer structure

S3 operates on parameters $x_t \in \mathbb{R}^d$ with stochastic gradient $g_t = \nabla f(x_t; \zeta_t)$, learning rate $\eta_t$, unified EMA coefficient $\beta \in [0,1)$, momentum order $p \geq 1$, and optional decoupled weight decay $\lambda \geq 0$. No $\epsilon$ term is required, and the method does not use gradient clipping or bias correction [2507.06464].

Its elementwise moving averages are
$$
m_t = \beta m_{t-1} + (1-\beta) g_t,
$$
$$
s_t(p) = \beta s_{t-1} + (1-\beta) |g_t|^p.
$$

S3 then forms Nesterovized numerator and denominator terms,
$$
n_t = \beta m_t + (1-\beta) g_t,
$$
$$
b_t(p) = \left(\beta s_t(p) + (1-\beta)|g_t|^p\right)^{1/p}.
$$

The core update is
$$
x_{t+1} = x_t - \eta_t \frac{n_t}{b_t(p)}.
$$
With decoupled weight decay, the update becomes
$$
x_{t+1} = (1-\eta_t\lambda)x_t - \eta_t \frac{n_t}{b_t(p)}.
$$

The optimizer therefore generalizes Adam-style normalization in two ways. First, it replaces the conventional second-order denominator by a flexible $p$-th order preconditioner. Second, it uses a **single** EMA coefficient for all moving averages. The paper explicitly presents this as a sign-like generalization beyond Adam, not as an explicit application of the nonlinearity
$$
\operatorname{softsign}(x)=\frac{x}{1+|x|}.
$$
Instead, the “softsign-like” property comes from the bounded ratio $n_t/b_t(p)$ rather than from an explicit softsign transform [2507.06464].

## 3. Bounded updates, stability, and the relation to Adam

The central motivation of S3 is a specific instability mechanism attributed to Adam. The paper argues that Adam is effective partly because $m_t/\sqrt{v_t}$ is already sign-like, but also vulnerable because this normalized update can substantially exceed magnitude $1$. For Adam, under $\beta_1^2 < \beta_2$, the paper gives the coordinate-wise upper bound
$$
\frac{|m_t^{(j)}|}{\sqrt{v_t^{(j)}}}
\le
\frac{(1-\beta_1)\sqrt{1-\beta_2^t}\sqrt{1-(\beta_1^2/\beta_2)^t}}
{(1-\beta_1^t)\sqrt{1-\beta_2}\sqrt{1-\beta_1^2/\beta_2}}
\approx
\frac{1-\beta_1}{\sqrt{1-\beta_2}\sqrt{1-\beta_1^2/\beta_2}}.
$$
For the commonly used $\beta_1=0.9$ and $\beta_2=0.999$, this upper bound is approximately $7.27$ [2507.06464].

S3 is designed to eliminate this failure mode. Let $q$ satisfy $1/p + 1/q = 1$. The paper proves that if $\beta_1 < \beta_2^{1/p}$ and $p \ge 1$, then for any coordinate $j$,
$$
\frac{|n_t^{(j)}|}{b_t^{(j)}(p)}
\le
\frac{1-\beta_1}
{(1-\beta_2)^{1/p}\left(1-\beta_1^q/\beta_2^{q/p}\right)^{1/q}}.
$$
In the special case $\beta_1=\beta_2=\beta$, which is precisely the unified-EMA design used by S3, this upper bound is minimized and becomes
$$
\frac{|n_t^{(j)}|}{b_t^{(j)}(p)} \le 1.
$$

This bound is the optimizer’s principal stability mechanism. The normalized step is coordinatewise capped in $[-1,1]$, so large learning rates do not combine with occasional oversized coordinate updates in the same manner as in Adam. The paper also proves monotonicity in the denominator with respect to $p$: for $1 \le p_1 \le p_2$, one has $b_t(p_1) \le b_t(p_2)$ coordinatewise. This means that larger $p$ values increase the denominator and further attenuate spikes. The authors interpret this as a direct mechanism for reducing destabilizing loss spikes during abrupt gradient fluctuations [2507.06464].

## 4. Nesterov acceleration and convergence theory

S3 incorporates an equivalent Nesterov’s accelerated gradient module without introducing additional persistent state beyond $m_t$ and $s_t(p)$. The paper proves the equivalence of three NAG forms and identifies the relevant form for S3 as the NAG (II)-style update
$$
g_t=\nabla f(x_t;\zeta_t), \qquad
m_t=\beta m_{t-1}+(1-\beta)g_t,
$$
$$
x_{t+1}=x_t-\eta\bigl(\beta m_t + (1-\beta)g_t\bigr).
$$
S3 instantiates this lookahead structure in both numerator and denominator, which is why the paper describes the method as providing acceleration “without memory overhead” [2507.06464].

The theoretical analysis considers minimizing
$$
F(x)=\mathbb{E}_{\zeta}[f(x;\zeta)]
$$
under three assumptions: $F$ is bounded below; a generalized local smoothness condition holds with constants $L_0$, $L_1$, and $R$; and the stochastic gradient is unbiased with bounded variance. The smoothness assumption is
$$
\|\nabla F(y)-\nabla F(x)\|_2
\le
\bigl(L_0 + L_1\|\nabla F(x)\|_2\bigr)\|y-x\|_2
\quad\text{for }\|x-y\|_2 \le R.
$$

For the nonconvex stochastic setting, the paper sets
$$
\beta = 1-\frac{1}{\sqrt{T}}, \qquad
\eta = \frac{1}{L_0 T^{3/4}},
$$
defines
$$
u_t^{(j)}=\frac{|n_t^{(j)}|}{b_t^{(j)}(p)},
$$
and assumes $u_t^{(j)} \ge 1/U_{\max}$. The resulting convergence theorem states
$$
\frac{1}{T}\sum_{t=1}^T \mathbb{E}\bigl[\|\nabla F(x_t)\|_1\bigr]
\le
\frac{2L_0U_{\max}(F(x_1)-F^*)}{T^{1/4}}
+
\frac{4\beta U_{\max}\sqrt{d}\,\mathbb{E}[\|\nabla F(x_1)\|_2]}{T^{1/2}}
+
\frac{4}{U_{\max}}\frac{\sqrt{d}\sigma}{T^{1/4}}
+
\frac{4\beta^2 U_{\max} d}{T^{1/4}}
+
\frac{U_{\max} d}{T^{7/4}}.
$$

The paper presents this as the optimal $O(T^{-1/4})$ rate for general nonconvex stochastic optimization under a weaker “non-uniform” smoothness assumption than many Adam analyses. A plausible implication is that S3 is intended not merely as a heuristic stabilization of Adam-like behavior, but as an optimizer whose bounded-update mechanism and acceleration module are designed to remain analyzable in the stochastic nonconvex regime [2507.06464].

## 5. Implementation characteristics and practical guidance

The default settings reported for S3 are $\beta=0.95$ and $p=3$, with $p=3$ described as a robust choice and $p=1$ as slightly cheaper but sometimes slightly worse in accuracy. The paper states that S3 supports aggressive learning rates because of the bounded normalized step and the larger-$p$ preconditioner. Example maxima include $\eta_{\max} \approx 6\times 10^{-3}$ with cosine decay for ViT-B/16 on ImageNet, and $\eta_{\max} \approx 3\times 10^{-3}$ with $5$k-step warmup and cosine decay for GPT-2 (345M) on OpenWebText [2507.06464].

The per-step implementation is structurally simple:

1. compute $g_t$;
2. update $m$ and $s$;
3. form $n$ and $b$;
4. optionally apply decoupled weight decay;
5. update parameters by $-\eta_t(n/b)$.

Memory usage is comparable to Adam because S3 stores two state tensors, $m$ and $s$, and does not allocate additional persistent buffers for NAG. The paper contrasts this with Adan, which stores additional $r_t$ state and requires $g_{t-1}$. Compute overhead is described as modest: for $p=1$ it is minimal, and for $p=3$ it remains limited to extra elementwise operations such as $|g|^p$ and power evaluations [2507.06464].

The practical tuning advice given in the paper is correspondingly narrow. One starts with $\beta=0.95$ and $p=3$, uses an AdamW-like learning rate, and then scales it upward if the loss remains smooth. If slight oscillations appear, the paper recommends increasing $p$ or mildly reducing the learning rate. Weight decay is used in decoupled AdamW style, and the product $\mathrm{lr}\times \mathrm{wd}$ is matched to AdamW by the “keep lr×wd constant” rule borrowed from Lion [2507.06464].

## 6. Empirical performance, ablations, and limitations

The paper reports experiments on ImageNet classification with ResNet-50 and ViT-B/16, and on language modeling with GPT-2 at 345M and 7B scales. S3 is evaluated with $p=3$, $\beta=0.95$, and no gradient clipping; for other optimizers, gradient clipping at $1.0$ is standard [2507.06464].

| Setting | S3 | Comparator |
|---|---:|---:|
| ResNet-50, ImageNet, 150 epochs | 78.76% | AdamW 77.29% |
| ViT-B/16, ImageNet, 150 epochs | 80.93% | AdamW 79.52% |
| GPT-2 345M, 50k steps, val perplexity | 4.59 | AdamW 4.78 |
| GPT-2 7B, 50k steps, val perplexity | 19.69 | AdamW 21.13 |

For GPT-2 (345M), the paper also reports that AdamW reaches validation perplexity $4.57$ at $100$k steps, whereas S3 reaches $4.59$ at $50$k steps, which it interprets as roughly $2\times$ efficiency. The abstract further states that S3 “rarely experiences loss spikes, even with a $10\times$ larger learning rate,” and that it delivers performance comparable to or better than AdamW with $2\times$ the training steps [2507.06464].

The ablation study assigns the largest single gain to the NAG module. Flexible $p>1$ improves stability and final performance, with $p=3$ serving as a strong default. Using the same $\beta$ for numerator and denominator is reported as neutral to mildly negative in isolation, but crucial when combined with larger learning rates because it enables the $[-1,1]$ bound on the normalized step. On downstream zero-shot evaluation, S3-pretrained GPT-2 models outperform AdamW counterparts on an OpenCompass suite, with larger gains at larger model size [2507.06464].

The paper also states several limitations. Extremely aggressive learning rates may still require tuning of warmup length or $p$, and for $p=1$ the theoretical bound remains valid but final accuracy can be slightly lower than for $p=3$ on some vision tasks. More broadly, the optimizer should not be confused with two other sign-based developments: the smooth $\tanh$-based **SoftSignum** relaxation of Signum, which addresses parameter heterogeneity through temperature scheduling [2605.31371], and the distributed **Sparse-SignSGD with Majority Vote** method, whose $S^3$ acronym refers to sparsification plus sign quantization rather than to bounded adaptive preconditioning [2302.07475].

Source: https://www.emergentmind.com/topics/signsoftsgd-s3