Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

Abstract: Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S_t^{-1/2} normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-epsilon crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-epsilon crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory.

• The paper establishes an exact implicit bias theorem, showing normalized iterates converge to a unique minimizer defined by a rate-indexed Burg-type barrier program.
• It employs a decaying stability parameter in sign descent dynamics to interpolate between gradient-descent and signSGD behaviors via exponential annealing.
• It offers finite-time mirror ascent analysis with empirical validations, highlighting the non-transference of these bias guarantees to Adam and RMSProp.

Stability Annealing, Implicit Bias, and the Barrier Path in Smoothed Sign Descent

Overview and Objectives

The paper "Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data" (2607.06013) provides a rigorous characterization of the implicit bias of memoryless smoothed-sign descent with a decaying stability parameter, addressing a core ambiguity in the behavior of sign-like adaptive methods under separability conditions. Specifically, the work analyzes full-batch linear classification with weighted exponential loss, focusing on the regime where the coordinatewise stability constant in the update is annealed at a controlled exponential rate. The main contribution is an exact implicit bias theorem: the normalized iterates converge to a unique minimizer of a Burg-type barrier program constrained by a margin slice determined by the stability decay rate.

Background and Motivation

Implicit bias in optimization algorithms, particularly gradient-based methods, has emerged as a central theme in understanding generalization in overparameterized models. Classical results show that gradient descent with exponential or logistic loss drives iterates towards the $\ell_2$ hard-margin separator in the separable regime, while adaptive and sign-based optimizers typically produce different geometric biases, e.g., $\ell_\infty$-based separators [soudry2018implicit, nacson2019stochastic, zhang2024implicitadam]. Introducing a fixed stability constant, as in Adam-style updates, changes this bias, but ambiguity remains for the case where stability is neither strictly fixed nor negligible but instead decays at a prescribed rate. The authors identify and resolve this ambiguity by showing an exact, rate-indexed transition of the limiting separator via a specific barrier program.

The Stability-Annealed Smoothed Sign Descent Dynamics

The analyzed dynamic is

$$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$

where updates are coordinatewise, and $L(w)$ is the weighted exponential loss. The annealing schedule for $\epsilon_t$ is exponential in cumulative learning-rate time $S_t$, indexed by rate parameter $\kappa$. This setting explicitly interpolates between signSGD-like behavior ($\epsilon_t \to 0$ quickly) and gradient-descent-like behavior (large or constant $\epsilon_t$), enabling examination of the limiting implicit bias as a function of the annealing rate.

Figure 1: The two-dimensional barrier geometry, illustrating the rate-indexed selection path within the margin–feasible slice for varying $\kappa$ and geometric endpoints for $\ell_\infty$0 and $\ell_\infty$1-based separators.

Barrier Geometry, Optimization Program, and Limiting Behavior

The central concept is a convex "Burg-type" barrier function

$\ell_\infty$2

The limiting separator for a given $\ell_\infty$3 is the unique minimizer:

$\ell_\infty$4

where $\ell_\infty$5 are the signed datapoints. This barrier program interpolates between $\ell_\infty$6 and $\ell_\infty$7 solutions: as $\ell_\infty$8, $\ell_\infty$9 (the $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$0-maximum margin separator), while as $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$1, $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$2 (the $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$3-margin maximizer), where $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$4.

Key Theoretical Result: For $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$5, the normalized iterates $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$6 converge to $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$7, and the rescaled negative gradient $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$8 converges to the image of the dual multipliers under the KKT mapping.

Duality and Discrete-Time Mirror Ascent Representation

The analysis recasts the dynamics as entropic mirror ascent on a concave dual objective associated with the barrier program. Explicit definition of the Fenchel conjugate and strong duality structure provides:

• An exact recursion for dual gap reduction via a KL divergence identity,
• Finite-time envelope bounds for the approach of the normalized iterates to the barrier minimizer, with rate $$w_{t+1} = w_t + \eta_t \frac{-\nabla L(w_t)}{|\nabla L(w_t)| + \epsilon_t}, \qquad \epsilon_t = \epsilon_0 \exp(-\kappa S_t), \quad S_t = \sum_{s=0}^{t-1}\eta_s$$9,
• Tight characterization of the path to the barrier-constrained solution independent of active-set identification or generic position assumptions.

  Figure 2: Theorem validation dashboard, showing error distributions and the validation of discrete-time dual algebra at floating-point accuracy across >2700 successful runs and extended long-horizon checkpoints.

Numerical Results and Empirical Diagnostics

The theoretical claims are substantiated by extensive experiments on synthetic separable datasets, varying dimension, support-vector configurations, and data distributions. Critical evidence includes:

• Floating-point verification of dual identities (i.e., mirror ascent algebra) under the prescribed schedule,
• Robust recovery of the rate-indexed barrier path for a wide range of hyperparameters,
• Empirical confirmation of the $L(w)$0 convergence envelope without post-hoc parameter tuning,
• Strong agreement between observed and predicted transition scaling in the fixed-$L(w)$1 crossover regime.

  Figure 3: Mechanism scorecard comparing error dynamics under different schedule settings, highlighting the specific role of the exponential $L(w)$2-clock in aligning path behavior across instances.

  

  Figure 4: Fixed-$L(w)$3 crossover scaling, empirically validating the predicted law $L(w)$4 for the memoryless proxy scheme.

Adam, RMSProp, and Other Boundary Methods

Diagnostic experiments contrast the main smoothed-sign descent path with full-batch Adam and RMSProp updates. Empirical transfer diagnostics show that, under tested schedules, Adam and RMSProp do not satisfy the perturbation residual conditions necessary to inherit the same implicit bias trajectory, thus failing to transfer the smoothed-sign theoretical guarantee. The separation is numerically quantified: mean path errors to the barrier solution are >10$L(w)$5 larger for Adam/RMSProp than for the memoryless scheme.

Figure 5: Adaptive-method boundary diagnostics showing the empirical distribution of path errors and transfer residuals for Adam, RMSProp, and the smoothed-sign method.

Theoretical and Practical Implications

The results provide several important theoretical clarifications:

• Selector Mechanism: The exponential annealing rate $L(w)$6 provides a direct and continuous selector for the implicit bias between $L(w)$7 and $L(w)$8 geometries for separable data under sign-based preconditioning.
• Mirror Ascent Equivalence: The explicit link to entropic mirror ascent offers a rigorous finite-time analysis that bypasses requirement for active-set identification typically needed in classical margin analyses.
• Non-transference to Adam/RMSProp: The negative results for Adam and RMSProp under the prescribed schedules delimit the scope of the theorem and highlight the necessity for specific residual control if adaptive methods are to inherit the same implicit bias as the idealized smoothed-sign proxy.

Practical consequences include:

• Schedule design for adaptive methods: if precise geometry selection is desirable (e.g., interpolating between sparse and dense margin solutions), direct stability annealing regimes provide algorithmic control.
• Finite-time guarantees: the $L(w)$9 envelope provides effective stopping criteria and confidence levels for trajectory proximity to the limiting solution.

Limitations and Open Questions

The established results require exponential loss and separability, with $\epsilon_t$0. The endpoint and critical regime ($\epsilon_t$1), as well as generic logistic-loss tails, lie outside the current proof, though empirical diagnostics suggest robustness. The non-transference of the guarantee to adaptive methods signals a need for new proof techniques or more refined algorithmic prescriptions in these wider contexts. Two-sided uniform gradient tail bounds for fixed-$\epsilon_t$2 crossover scaling remain an open theoretical challenge.

Conclusion

This study delivers a precise, explicit selection principle for the implicit bias of smoothed-sign descent under stability annealing, identifying a unique, rate-indexed Burg-type barrier path in separable settings with exponential loss. The dual mirror-ascent proof structure ensures robust finite-time guarantees, and rigorous numerical diagnostics confirm theory–experiment alignment. The work offers algorithmic and geometric guidelines for both researchers and practitioners seeking to understand or control the geometry induced by adaptive learning rates in linear models. Future directions include extending the barrier path paradigm to general loss tails, nonlinear architectures, and systematic analysis of adaptive preconditioning schemes.