--- title: Singularity-aware Adam Optimizers url: https://www.emergentmind.com/topics/singularity-aware-adam-s-adam type: topic --- # Singularity-aware Adam Optimizers Singularity-aware Adam, variously written **SAdam**, **Sadam**, or **S-Adam**, denotes several distinct Adam-derived optimization methods that alter Adam’s adaptive denominator or effective step size in order to address specific failure modes of the original algorithm. In the literature represented here, the name has been used for a strongly convex online-learning variant that replaces Adam’s $v_t^{-1/2}$ scaling by $v_t^{-1}$ and attains a data-dependent $O(\log T)$ regret bound [1905.02957], a softplus-calibrated adaptive method intended to regularize highly anisotropic coordinatewise learning rates and improve convergence and generalization [1908.00700], and a non-smooth optimization method that damps Adam updates according to a randomized estimate of local geometric instability near Clarke-singular regions [2605.29547]. ## 1. Nomenclature and scope A persistent source of confusion is that **“singularity-aware Adam” is not a single standardized optimizer**. The label has been attached to different modifications of Adam, each targeting a different pathology. One line of work focuses on **strongly convex online convex optimization**, another on **adaptive-learning-rate calibration in smooth or weakly structured stochastic optimization**, and a third on **non-smooth deep-learning regimes** involving ReLU kinks, quantization, and Clarke-subdifferential effects. | Variant | Core modification | Primary setting | |---|---|---| | SAdam [1905.02957] | Uses $\hat v_t^{-1}$ instead of $\hat v_t^{-1/2}$ and controls decay through time-varying $\beta_{2t}$ | Strongly convex online convex optimization | | Sadam [1908.00700] | Replaces $\sqrt{v_t}+\epsilon$ by $\mathrm{softplus}(\sqrt{v_t})$ | Nonconvex, convex, P-L, and strongly convex stochastic optimization | | S-Adam [2605.29547] | Multiplies Adam’s step by $\exp(-\lambda \rho_t)$ using a Local Geometric Instability score | Non-smooth stochastic optimization, QAT, and small-batch learning | The shared theme is **modification of Adam’s adaptivity near problematic regimes**. The problematic regime, however, differs substantially across papers: strongly convex geometry in [1905.02957], anisotropic adaptive learning rates and $\epsilon$-sensitivity in [1908.00700], and non-smooth singularities characterized through the Clarke subdifferential in [2605.29547]. ## 2. Strongly convex SAdam In the 2019 paper “SAdam: A Variant of Adam for Strongly Convex Functions,” SAdam is the strongly-convex adaptation of Adam [1905.02957]. The motivating question is whether the strong-convexity structure can be exploited to improve on the data-dependent $O(\sqrt{T})$ regret bounds available for Adam-type methods under general convexity. The answer given is affirmative: SAdam is designed so that the **overall effective learning rate decays approximately as $1/t$**, matching the classical strongly convex online-gradient regime while preserving diagonal adaptivity and momentum. The setting is **online convex optimization** over a convex decision set $D\subseteq \mathbb{R}^d$. At round $t$, the learner selects $x_t\in D$, observes $f_t(\cdot)$, and incurs loss $f_t(x_t)$, with regret $$ R(T):=\sum_{t=1}^T f_t(x_t)-\min_{x\in D}\sum_{t=1}^T f_t(x). $$ The analysis assumes that every $f_t$ is $\lambda$-strongly convex, that gradients satisfy $$ \max_{x\in D}\|\nabla f_t(x)\|_\infty < G_\infty, $$ and that the domain is bounded in $\ell_\infty$ diameter: $$ \max_{x_1,x_2\in D}\|x_1-x_2\|_\infty \le D_\infty. $$ The method also uses a weighted projection $$ \Pi_D^H(x)=\arg\min_{y\in D}\|y-x\|_H $$ for positive definite $H$. Relative to standard Adam, $$ m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t,\qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)\operatorname{diag}(g_t g_t^\top), $$ with update $$ x_{t+1}=x_t-\alpha\, v_t^{-1/2} m_t, $$ SAdam modifies two components. First, it removes the square root from the preconditioner. Second, it makes $\beta_2$ time-varying so that the inverse preconditioner grows in a controlled manner. The paper’s Algorithm 1 is $$ g_t=\nabla f_t(x_t), $$ $$ m_t=\beta_{1t}m_{t-1}+(1-\beta_{1t})g_t, $$ $$ v_t=\beta_{2t}v_{t-1}+(1-\beta_{2t})\operatorname{diag}(g_t g_t^\top), $$ $$ \hat v_t=v_t+\epsilon I, $$ $$ x_{t+1}=\Pi_D^{\hat v_t}\!\left(x_t-\alpha_t \hat v_t^{-1}m_t\right). $$ A typical choice is $$ \beta_{1t}=\beta_1\nu^{t-1},\qquad \beta_1,\nu\in[0,1). $$ The central analytic device is control of the effective $1/t$-like step through conditions on the diagonal preconditioner. The key monotonicity-style requirement is: > There exists $C>0$ such that for any $\alpha>C$, for all $t$ and coordinates $i$, > $$ > t\,\hat v_{t,i}^{-1}-(t-1)\hat v_{t-1,i}^{-1}<\alpha(1-\beta_1). > $$ A second technical condition bounds the cumulative effect of the EMA denominator relative to $\sum_{j=1}^t g_{j,i}^2$. Under these assumptions, Theorem 1 yields a regret bound whose characteristic term is $$ O\!\left(\sum_{i=1}^d \log\!\Big(\sum_{j=1}^T g_{j,i}^2 + 1\Big)\right), $$ so the worst-case rate becomes $O(d\log T)$ [1905.02957]. The bound is explicitly data-dependent and becomes tighter when coordinatewise gradient histories are small or sparse. This SAdam also contains **SC-RMSprop** as a special case. When $\beta_{1t}=0$ and $1-\gamma<\beta_{2t}\le 1-\eta$ for suitable constants, the method reduces to SC-RMSprop, and the paper provides what it describes as the **first data-dependent logarithmic regret bound** for that method. In the $\beta_{1t}=0$ case, Corollary 3 gives a bound of the form $$ R(T)\le \frac{D_\infty^2}{2\alpha} + \frac{\alpha\gamma}{2}\sum_{i=1}^d \log\!\left( \frac{\sum_{j=1}^T g_{j,i}^2}{\gamma\epsilon}+1 \right), $$ up to the exact constants and normalization used in the paper. Empirically, this strongly convex SAdam is evaluated on $\ell_2$-regularized softmax regression over MNIST, CIFAR-10, and CIFAR-100, where it achieves the **lowest regret** among the compared methods; strongly convex methods such as SC-Adagrad, SC-RMSprop, and SAdam outperform Adam, AMSgrad, and AdamNC, while plain OGD performs worst overall. On a 4-layer CNN trained on the same datasets, SAdam yields the **lowest training loss** and strong test accuracy [1905.02957]. ## 3. Softplus-calibrated Sadam The 2019 paper “Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM” uses **Sadam** to denote a different algorithmic idea [1908.00700]. Its diagnosis is that Adam’s adaptive learning rate, $$ \frac{\eta_t}{\sqrt{v_t}+\epsilon}, $$ is often **highly anisotropic across coordinates and over training time**. The authors report that some coordinates of $$ \frac{1}{\sqrt{v_t}+\epsilon} $$ can become extremely large, even reaching about $10^8$ when $\epsilon=10^{-8}$. This motivates what the paper calls a **“small learning-rate dilemma”**: choosing the base learning rate $\eta$ small enough to prevent unstable large updates in some coordinates may cause other coordinates to update too slowly later in training. Sadam retains Adam’s first- and second-moment recursions, $$ m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, $$ $$ v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2, $$ but replaces Adam’s denominator $\sqrt{v_t}+\epsilon$ by $$ \mathrm{softplus}(\sqrt{v_t}), $$ where $$ \mathrm{softplus}(x)=\frac{1}{\beta}\log\!\bigl(1+e^{\beta x}\bigr),\qquad \beta>0. $$ The update becomes $$ x_{t+1}=x_t-\frac{\eta_t}{\mathrm{softplus}(\sqrt{v_t})}\odot m_t. $$ For large $x$, $\mathrm{softplus}(x)\approx x$; for small $x$, it lifts the denominator away from zero. The paper recommends the Adam-like defaults $\beta_1=0.9$ and $\beta_2=0.999$, and reports that $\beta$ is typically chosen from $\{10,50,100\}$, with $\beta=50$ recommended based on observed performance [1908.00700]. The companion method **SAMSGrad** applies the same calibration to AMSGrad’s monotone second-moment construction: $$ \tilde v_t=\beta_2 \tilde v_{t-1} + (1-\beta_2)g_t^2,\qquad v_t=\max\{v_{t-1},\tilde v_t\}, $$ followed by $$ x_{t+1}=x_t-\frac{\eta_t}{\mathrm{softplus}(\sqrt{v_t})}\odot m_t. $$ The conceptual distinction from Adam is that Adam stabilizes through a fixed additive $\epsilon$, whereas Sadam stabilizes by **smoothly bounding and compressing** the denominator via softplus. According to the paper, this reduces extreme A-LR values in coordinates where $v_{t,j}$ is tiny and removes the explicit need for $\epsilon$ as a tuning knob. When $\beta$ is large, $\mathrm{softplus}(\sqrt{v_t})\approx \sqrt{v_t}$ and the method approaches Adam-like behavior; when $\beta$ is small, the behavior becomes more like momentum SGD [1908.00700]. The theoretical contribution is organized around explicit bounds on the adaptive learning rate. For Adam, the A-LR bounds are denoted by $(\mu_1,\mu_2)$; for Sadam, by $(\mu_3,\mu_4)$. A central claim is that Adam’s convergence constants depend materially on $\epsilon$, a dependence said to have been largely overlooked in earlier analyses. Under $L$-smoothness, bounded gradients, and bounded stochastic-gradient variance, the paper derives: - in the **nonconvex** setting with $\eta=O(1/\sqrt{T})$, Adam has a convergence rate with explicit $1/\epsilon^2$ dependence, whereas Sadam has leading dependence summarized as $O(\beta^2/\sqrt{T})$; - in the **convex, non-strongly convex** setting for the averaged iterate $\bar x_T=\frac{1}{T}\sum_{t=1}^T x_t$, Adam satisfies $$ f(\bar x_T)-f^* \le O\!\left(\frac{d}{\epsilon^2\sqrt{T}}\right), $$ while Sadam satisfies $$ f(\bar x_T)-f^* \le O\!\left(\frac{d\beta^2}{\sqrt{T}}\right); $$ - under the **Polyak–Łojasiewicz condition** $$ \|\nabla f(x)\|^2 \ge 2\lambda(f(x)-f^*), $$ and in the **strongly convex case**, with $\eta=O(1/T^2)$, the paper gives linear-contraction-style expressions multiplied by an $O(1/T)$ term, with Sadam constants determined by $\mu_3$ and $\mu_4$ [1908.00700]. Empirical studies are conducted on MNIST with a CNN; CIFAR-10 with ResNet-20, ResNet-56, and DenseNet-40; CIFAR-100 with VGG and ResNet-18; and Penn Treebank and WikiText-2 with 3-layer LSTMs. The reported findings are that Sadam and SAMSGrad improve test accuracy, often close the gap with momentum SGD, improve perplexity curves and convergence behavior in language modeling, and compress the extreme range of the A-LR distribution. ## 4. S-Adam for non-smooth optimization The 2026 paper “Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization” introduces **S-Adam** in a third, more explicitly geometric sense [2605.29547]. The motivating claim is that modern deep networks are **not globally smooth** because of ReLU activations, quantization operators in QAT, and sparsity or regularization terms. At such singular points, the gradient is modeled not as a single vector but as a **set-valued Clarke subdifferential** $\partial_C f(x)$. In this framework, Adam is said to suffer from **gradient chattering**: violent oscillations in the update direction caused by conflicting signals within the Clarke subdifferential. The paper attributes to this phenomenon a mismatch between Adam’s momentum buffer and local geometry, overshooting across kinks or discrete boundaries, poor convergence, and worse generalization. The method is presented as a **drop-in Adam variant** that detects geometric singularity and damps updates only when necessary [2605.29547]. The core diagnostic is the **Local Geometric Instability (LGI)** score, intended to estimate the diameter of the Clarke subdifferential, $$ \operatorname{diam}(\partial_C f(x))=\sup_{g_1,g_2\in\partial_C f(x)}\|g_1-g_2\|. $$ At iterate $x_t$, one samples random directions $$ u_1,\dots,u_k\sim \mathcal{U}(\mathbb{S}^{d-1}), $$ forms directional probes $$ D_i=\frac{f(x_t+\delta u_i)-f(x_t)}{\delta}, $$ and defines $$ \rho_t(x_t)=\frac{\operatorname{Var}(\{D_i\}_{i=1}^k)}{\operatorname{Mean}(\{D_i^2\})+\epsilon}. $$ The appendix also writes the denominator as $\frac{1}{k}\sum_{i=1}^k D_i^2+\epsilon$. The paper proves $$ 0\le \rho_t(x_t)<1, $$ so the damping factor $\exp(-\lambda \rho_t)$ lies in $(e^{-\lambda},1]$. A finite-sample guarantee states $$ |\hat{\rho}_k-\rho|\le \Delta\quad\text{with probability at least }1-\delta, $$ using $$ k=O\!\left(\frac{L^6}{\epsilon^4\Delta^2}\log\frac{1}{\delta}\right). $$ The appendix also gives a randomized-smoothing interpretation. For $$ f_\delta(x)=\mathbb{E}_u[f(x+\delta u)], $$ LGI approximates a relative-curvature quantity $$ \rho(x)\approx \frac{1 + \frac{1}{2d}\kappa_\delta(x)^2}{1 + \frac{1}{2d}\kappa_\delta(x)^2 + \epsilon'}, $$ where $$ \kappa_\delta(x)=\frac{\delta \|\nabla^2 f_\delta(x)\|_F}{\|\nabla f_\delta(x)\|}. $$ Within the paper’s interpretation, high curvature or high nonsmoothness implies $\rho\approx 1$, whereas flat or smooth regions imply $\rho\approx 0$ [2605.29547]. S-Adam keeps Adam’s moments, $$ m_t=\beta_1 m_{t-1} + (1-\beta_1) g_t,\qquad v_t=\beta_2 v_{t-1} + (1-\beta_2) g_t^2, $$ but uses the LGI-modulated step size $$ \hat{\eta}_t=\eta_t\exp(-\lambda \rho_t), $$ leading to the update $$ w_{t+1}=w_t-\eta_t\exp(-\lambda \rho_t)\frac{m_t}{\sqrt{v_t}+\epsilon}. $$ The experiments use perturbation scale $\delta=0.01$, probe count $k\in\{2,8\}$, damping intensity $\lambda=2.0$, stabilization constant $\epsilon=10^{-6}$, and LGI score cap $10.0$. The convergence analysis is developed under local Lipschitzness, lower boundedness, path differentiability, almost-sure boundedness of iterates, a Robbins–Monro-style step-size condition $$ \sum_{t=1}^\infty \eta_t=\infty,\qquad \sum_{t=1}^\infty \eta_t^2<\infty, $$ and consistency conditions for the moment estimators. The algorithm is written as a stochastic approximation whose mean-field limit is a differential inclusion, $$ \dot{x}(t)\in -\bar{\alpha}(x(t))\mathcal{H}(x(t)), $$ with $$ \bar{\alpha}(x)=\mathbb{E}\!\left[\exp(-\lambda \rho(x))\mid x_t=x\right]. $$ Using this framework, the paper proves $$ \liminf_{t\to\infty}\operatorname{dist}(0,\partial_C f(x_t))=0\quad\text{a.s.}, $$ so every limit point is Clarke stationary, and it establishes the rate $$ \min_{0\le t