SignSoftSGD (S3): Adaptive Optimizer

Updated 6 July 2026

SignSoftSGD (S3) is a sign-based adaptive optimizer that employs a flexible p-th order moment and unified EMA to ensure each update is bounded within [-1,1].
It generalizes Adam by incorporating a Nesterov-accelerated gradient module, reducing loss spikes and improving training stability.
Empirical evaluations on models like ResNet-50, ViT-B/16, and GPT-2 demonstrate its efficiency and superior performance compared to conventional AdamW.

Searching arXiv for the cited optimizer papers to ground the article in the current record. {"query":"arXiv (Peng et al., 9 Jul 2025) SoftSignSGD(S3) An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam", "max_results": 5} to=arxiv_search 东臣on?

to=arxiv_search {"query":"(Peng et al., 9 Jul 2025) SoftSignSGD(S3)", "max_results": 10} SignSoftSGD (S3), titled “SoftSignSGD (S3)” in the paper and also referred to in some passages as “SignSoftSGD (S3),” is a sign-like adaptive optimizer proposed for practical deep neural network training and loss-spike minimization beyond Adam. Its defining design choices are a flexible $p$ -th order moment in the denominator, a unified exponential moving average (EMA) coefficient for numerator and denominator, and an equivalent Nesterov’s accelerated gradient module. The method is intended to preserve the robustness of sign-based updates under large gradient fluctuations while preventing uncontrolled coordinate-wise scaling; in the unified-EMA setting, each coordinate of its normalized update is provably bounded in $[-1,1]$ (Peng et al., 9 Jul 2025).

1. Nomenclature and disambiguation

The paper’s title and Algorithm 1 use SoftSignSGD (S3), while some prose uses SignSoftSGD (S3). The authors note that “SoftSignSGD” was chosen because the per-parameter update is inherently bounded in $[-1,1]$ , which they describe as softsign-like behavior. The intended short name is S3, and the “SignSoftSGD” form is a naming inconsistency in the prose (Peng et al., 9 Jul 2025).

The acronym S3 is not unique in the sign-optimization literature. In “Sparse-SignSGD with Majority Vote,” $S^3$ denotes Sparse-SignSGD, a distributed top- $K$ sign-compression method with majority-vote aggregation rather than a soft-sign adaptive optimizer (Park et al., 2023). In “Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling,” the relevant optimizer is SoftSignum, a temperature-controlled $\tanh$ relaxation of Signum; that paper does not use the S3 name for the method itself (Feoktistov et al., 29 May 2026). A separate 2025 paper on parameter-free Sign-SGD likewise does not introduce a soft-sign operator and uses only the hard coordinate-wise $\operatorname{sign}(\cdot)$ map (Medyakov et al., 4 Jun 2025).

This nomenclatural overlap matters because the term “SignSoftSGD (S3)” can otherwise be conflated with three distinct lines of work: bounded adaptive sign-like optimization, smooth sign relaxation, and sparse distributed sign compression.

2. Update rule and optimizer structure

S3 operates on parameters $x_t \in \mathbb{R}^d$ with stochastic gradient $g_t = \nabla f(x_t; \zeta_t)$ , learning rate $\eta_t$ , unified EMA coefficient $[-1,1]$ 0, momentum order $[-1,1]$ 1, and optional decoupled weight decay $[-1,1]$ 2. No $[-1,1]$ 3 term is required, and the method does not use gradient clipping or bias correction (Peng et al., 9 Jul 2025).

Its elementwise moving averages are

$[-1,1]$ 4

$[-1,1]$ 5

S3 then forms Nesterovized numerator and denominator terms,

$[-1,1]$ 6

$[-1,1]$ 7

The core update is

$[-1,1]$ 8

With decoupled weight decay, the update becomes

$[-1,1]$ 9

The optimizer therefore generalizes Adam-style normalization in two ways. First, it replaces the conventional second-order denominator by a flexible $[-1,1]$ 0-th order preconditioner. Second, it uses a single EMA coefficient for all moving averages. The paper explicitly presents this as a sign-like generalization beyond Adam, not as an explicit application of the nonlinearity

$[-1,1]$ 1

Instead, the “softsign-like” property comes from the bounded ratio $[-1,1]$ 2 rather than from an explicit softsign transform (Peng et al., 9 Jul 2025).

3. Bounded updates, stability, and the relation to Adam

The central motivation of S3 is a specific instability mechanism attributed to Adam. The paper argues that Adam is effective partly because $[-1,1]$ 3 is already sign-like, but also vulnerable because this normalized update can substantially exceed magnitude $[-1,1]$ 4. For Adam, under $[-1,1]$ 5, the paper gives the coordinate-wise upper bound

$[-1,1]$ 6

For the commonly used $[-1,1]$ 7 and $[-1,1]$ 8, this upper bound is approximately $[-1,1]$ 9 (Peng et al., 9 Jul 2025).

S3 is designed to eliminate this failure mode. Let $S^3$ 0 satisfy $S^3$ 1. The paper proves that if $S^3$ 2 and $S^3$ 3, then for any coordinate $S^3$ 4,

$S^3$ 5

In the special case $S^3$ 6, which is precisely the unified-EMA design used by S3, this upper bound is minimized and becomes

$S^3$ 7

This bound is the optimizer’s principal stability mechanism. The normalized step is coordinatewise capped in $S^3$ 8, so large learning rates do not combine with occasional oversized coordinate updates in the same manner as in Adam. The paper also proves monotonicity in the denominator with respect to $S^3$ 9: for $K$ 0, one has $K$ 1 coordinatewise. This means that larger $K$ 2 values increase the denominator and further attenuate spikes. The authors interpret this as a direct mechanism for reducing destabilizing loss spikes during abrupt gradient fluctuations (Peng et al., 9 Jul 2025).

4. Nesterov acceleration and convergence theory

S3 incorporates an equivalent Nesterov’s accelerated gradient module without introducing additional persistent state beyond $K$ 3 and $K$ 4. The paper proves the equivalence of three NAG forms and identifies the relevant form for S3 as the NAG (II)-style update

$K$ 5

$K$ 6

S3 instantiates this lookahead structure in both numerator and denominator, which is why the paper describes the method as providing acceleration “without memory overhead” (Peng et al., 9 Jul 2025).

The theoretical analysis considers minimizing

$K$ 7

under three assumptions: $K$ 8 is bounded below; a generalized local smoothness condition holds with constants $K$ 9, $\tanh$ 0, and $\tanh$ 1; and the stochastic gradient is unbiased with bounded variance. The smoothness assumption is

$\tanh$ 2

For the nonconvex stochastic setting, the paper sets

$\tanh$ 3

defines

$\tanh$ 4

and assumes $\tanh$ 5. The resulting convergence theorem states

$\tanh$ 6

The paper presents this as the optimal $\tanh$ 7 rate for general nonconvex stochastic optimization under a weaker “non-uniform” smoothness assumption than many Adam analyses. A plausible implication is that S3 is intended not merely as a heuristic stabilization of Adam-like behavior, but as an optimizer whose bounded-update mechanism and acceleration module are designed to remain analyzable in the stochastic nonconvex regime (Peng et al., 9 Jul 2025).

5. Implementation characteristics and practical guidance

The default settings reported for S3 are $\tanh$ 8 and $\tanh$ 9, with $\operatorname{sign}(\cdot)$ 0 described as a robust choice and $\operatorname{sign}(\cdot)$ 1 as slightly cheaper but sometimes slightly worse in accuracy. The paper states that S3 supports aggressive learning rates because of the bounded normalized step and the larger- $\operatorname{sign}(\cdot)$ 2 preconditioner. Example maxima include $\operatorname{sign}(\cdot)$ 3 with cosine decay for ViT-B/16 on ImageNet, and $\operatorname{sign}(\cdot)$ 4 with $\operatorname{sign}(\cdot)$ 5k-step warmup and cosine decay for GPT-2 (345M) on OpenWebText (Peng et al., 9 Jul 2025).

The per-step implementation is structurally simple:

compute $\operatorname{sign}(\cdot)$ 6;
update $\operatorname{sign}(\cdot)$ 7 and $\operatorname{sign}(\cdot)$ 8;
form $\operatorname{sign}(\cdot)$ 9 and $x_t \in \mathbb{R}^d$ 0;
optionally apply decoupled weight decay;
update parameters by $x_t \in \mathbb{R}^d$ 1.

Memory usage is comparable to Adam because S3 stores two state tensors, $x_t \in \mathbb{R}^d$ 2 and $x_t \in \mathbb{R}^d$ 3, and does not allocate additional persistent buffers for NAG. The paper contrasts this with Adan, which stores additional $x_t \in \mathbb{R}^d$ 4 state and requires $x_t \in \mathbb{R}^d$ 5. Compute overhead is described as modest: for $x_t \in \mathbb{R}^d$ 6 it is minimal, and for $x_t \in \mathbb{R}^d$ 7 it remains limited to extra elementwise operations such as $x_t \in \mathbb{R}^d$ 8 and power evaluations (Peng et al., 9 Jul 2025).

The practical tuning advice given in the paper is correspondingly narrow. One starts with $x_t \in \mathbb{R}^d$ 9 and $g_t = \nabla f(x_t; \zeta_t)$ 0, uses an AdamW-like learning rate, and then scales it upward if the loss remains smooth. If slight oscillations appear, the paper recommends increasing $g_t = \nabla f(x_t; \zeta_t)$ 1 or mildly reducing the learning rate. Weight decay is used in decoupled AdamW style, and the product $g_t = \nabla f(x_t; \zeta_t)$ 2 is matched to AdamW by the “keep lr×wd constant” rule borrowed from Lion (Peng et al., 9 Jul 2025).

6. Empirical performance, ablations, and limitations

The paper reports experiments on ImageNet classification with ResNet-50 and ViT-B/16, and on language modeling with GPT-2 at 345M and 7B scales. S3 is evaluated with $g_t = \nabla f(x_t; \zeta_t)$ 3, $g_t = \nabla f(x_t; \zeta_t)$ 4, and no gradient clipping; for other optimizers, gradient clipping at $g_t = \nabla f(x_t; \zeta_t)$ 5 is standard (Peng et al., 9 Jul 2025).

Setting	S3	Comparator
ResNet-50, ImageNet, 150 epochs	78.76%	AdamW 77.29%
ViT-B/16, ImageNet, 150 epochs	80.93%	AdamW 79.52%
GPT-2 345M, 50k steps, val perplexity	4.59	AdamW 4.78
GPT-2 7B, 50k steps, val perplexity	19.69	AdamW 21.13

For GPT-2 (345M), the paper also reports that AdamW reaches validation perplexity $g_t = \nabla f(x_t; \zeta_t)$ 6 at $g_t = \nabla f(x_t; \zeta_t)$ 7k steps, whereas S3 reaches $g_t = \nabla f(x_t; \zeta_t)$ 8 at $g_t = \nabla f(x_t; \zeta_t)$ 9k steps, which it interprets as roughly $\eta_t$ 0 efficiency. The abstract further states that S3 “rarely experiences loss spikes, even with a $\eta_t$ 1 larger learning rate,” and that it delivers performance comparable to or better than AdamW with $\eta_t$ 2 the training steps (Peng et al., 9 Jul 2025).

The ablation study assigns the largest single gain to the NAG module. Flexible $\eta_t$ 3 improves stability and final performance, with $\eta_t$ 4 serving as a strong default. Using the same $\eta_t$ 5 for numerator and denominator is reported as neutral to mildly negative in isolation, but crucial when combined with larger learning rates because it enables the $\eta_t$ 6 bound on the normalized step. On downstream zero-shot evaluation, S3-pretrained GPT-2 models outperform AdamW counterparts on an OpenCompass suite, with larger gains at larger model size (Peng et al., 9 Jul 2025).

The paper also states several limitations. Extremely aggressive learning rates may still require tuning of warmup length or $\eta_t$ 7, and for $\eta_t$ 8 the theoretical bound remains valid but final accuracy can be slightly lower than for $\eta_t$ 9 on some vision tasks. More broadly, the optimizer should not be confused with two other sign-based developments: the smooth $[-1,1]$ 00-based SoftSignum relaxation of Signum, which addresses parameter heterogeneity through temperature scheduling (Feoktistov et al., 29 May 2026), and the distributed Sparse-SignSGD with Majority Vote method, whose $[-1,1]$ 01 acronym refers to sparsification plus sign quantization rather than to bounded adaptive preconditioning (Park et al., 2023).