LyAm: Lyapunov-Based Adaptive Optimizer

• LyAm is a novel Adam-style optimizer that integrates Lyapunov stability by using the loss as a Lyapunov function, ensuring monotonic decrease.
• It combines adaptive moment estimation with a per-parameter damping rule to improve convergence in noisy, non-convex settings.
• Empirical results on vision benchmarks show enhanced stability, faster convergence, and improved robustness compared to traditional optimizers.

Searching arXiv for the exact term "LyAm" and the cited paper to ensure the article is grounded in current arXiv records. LyAm is an Adam-style optimizer for deep neural network training that integrates adaptive moment estimation with Lyapunov-based stability mechanisms in order to improve robustness on noisy gradients and highly non-convex loss surfaces (Mirzabeigi et al., 15 Jul 2025). It treats optimization as the evolution of a dynamical system and uses the loss itself as a Lyapunov function, so that the update rule is explicitly shaped by the requirement that the loss decreases monotonically or at least does not increase. In the formulation reported for computer vision benchmarks, LyAm combines Adam-like first- and second-moment tracking with a Lyapunov-inspired per-parameter step-size rule that damps updates when the squared-gradient estimate is large, with the stated goal of enhancing convergence robustness, mitigating training noise, and stabilizing learning in complex non-convex regimes (Mirzabeigi et al., 15 Jul 2025).

1. Problem setting and design objective

LyAm was proposed for the training regime characteristic of modern deep networks, especially in computer vision, where stochastic optimization proceeds over highly non-convex objectives and high-dimensional parameter spaces (Mirzabeigi et al., 15 Jul 2025). In that setting, the gradients are described as noisy due to mini-batching, label noise, corrupted inputs, poisoned data, and adversarial examples, while the geometry of the loss landscape introduces additional instability through sharp valleys, saddle points, oscillations, and divergence risks.

The optimizer is positioned relative to two standard families. SGD is simple and can generalize well, but it converges slowly in ill-conditioned or highly noisy settings and requires careful hand tuning of the learning rate. Adam, RMSProp, and AdaGrad use per-parameter adaptive steps based on gradient statistics and often converge faster, but their adaptivity is characterized as largely heuristic; in highly non-convex regions they can overshoot or oscillate, remain sensitive to gradient noise, exhibit erratic loss curves, and lack strong convergence guarantees in general non-convex settings (Mirzabeigi et al., 15 Jul 2025).

The design goal of LyAm is therefore to explicitly enforce stability during optimization. The method treats training as a dynamical system, chooses an energy-like scalar function, and designs updates so that this function decreases along the optimization trajectory. In the reported formulation, that function is the loss itself. This suggests a conceptual synthesis: Adam contributes practical moment-based adaptivity, while Lyapunov stability theory constrains the admissible step magnitude so that training remains stable even when gradients are noisy or curvature is unfavorable (Mirzabeigi et al., 15 Jul 2025).

2. Algorithmic structure

LyAm maintains Adam-style first and second moments of the stochastic gradient. Let $\theta_t \in \mathbb{R}^n$ denote the parameters at iteration $t$, let $L(\theta)$ be the loss, and let $g_t = \nabla_\theta L(\theta_t)$ be the stochastic gradient. For each coordinate $i = 1,\dots,n$, the moment recursions are

$$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$

$$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$

with decay rates $\beta_1, \beta_2 \in [0,1)$ (Mirzabeigi et al., 15 Jul 2025). As in Adam, bias correction is applied through $\hat m_t$ and $\hat v_t$, described as the bias-corrected first and second moment estimates.

The defining difference from Adam is the effective learning-rate rule. Instead of using a normalized step of the form $t$0, LyAm uses

$t$1

together with the update

$t$2

This replaces square-root normalization by multiplicative damping through the factor $t$3 (Mirzabeigi et al., 15 Jul 2025). If the second moment is large, the coordinatewise learning rate shrinks; if it is small, the learning rate remains close to the base value $t$4. The paper states that this scaling is derived from Lyapunov stability analysis rather than heuristic design.

The resulting procedure differs from Adam in only one structural component: the computation of the effective per-parameter step size. Initialization uses $t$5, $t$6, $t$7, and $t$8, with $t$9 and $L(\theta)$0; each iteration then computes the gradient, updates the moments, applies bias correction, forms the elementwise adaptive learning rate $L(\theta)$1, and updates parameters through $L(\theta)$2 (Mirzabeigi et al., 15 Jul 2025).

3. Lyapunov stability mechanism

The Lyapunov mechanism is built by selecting the loss itself as the Lyapunov function,

$L(\theta)$3

so that stability is analyzed through the discrete-time drift

$L(\theta)$4

The stated objective is to make this drift non-positive, either directly or in expectation (Mirzabeigi et al., 15 Jul 2025).

The analysis assumes that $L(\theta)$5 is differentiable with Lipschitz continuous gradient,

$L(\theta)$6

for some Lipschitz constant $L(\theta)$7. Under this assumption, a Taylor upper bound yields a decomposition of the Lyapunov drift into a descent term and a quadratic penalty term. Substituting the LyAm update gives

$L(\theta)$8

The first term is favorable when the first moment is aligned with the gradient, while the second penalizes large steps in high-curvature regions (Mirzabeigi et al., 15 Jul 2025).

The analysis then introduces an alignment assumption, namely that the first moment tracks the true gradient sufficiently well, $L(\theta)$9. Under that approximation, the LyAm damping rule causes the descent term to scale roughly as

$g_t = \nabla_\theta L(\theta_t)$0

while the quadratic penalty scales as

$g_t = \nabla_\theta L(\theta_t)$1

From this comparison, the stated coordinatewise condition for non-positive drift is

$g_t = \nabla_\theta L(\theta_t)$2

In words, if the base learning rate is sufficiently small relative to curvature and local variance, the loss decreases. Because the effective step is scaled by $g_t = \nabla_\theta L(\theta_t)$3, the step size automatically contracts when the squared-gradient estimate is large (Mirzabeigi et al., 15 Jul 2025). This is the central theoretical motivation for the optimizer’s damping form.

4. Convergence properties in non-convex optimization

The theoretical analysis is described as Lyapunov-based and qualitative rather than a finite-time rate theorem. Under the assumptions that the loss has Lipschitz continuous gradient, the moment estimate remains sufficiently aligned with the gradient, and the base step size satisfies the stability condition above, the paper states that the Lyapunov drift obeys

$g_t = \nabla_\theta L(\theta_t)$4

so that the loss is monotonically non-increasing (Mirzabeigi et al., 15 Jul 2025).

In non-convex settings, the consequence is that the sequence $g_t = \nabla_\theta L(\theta_t)$5 converges because it is bounded below and non-increasing. The paper further states that the updates stabilize and, under standard conditions, $g_t = \nabla_\theta L(\theta_t)$6 tends to zero, implying convergence to a critical point rather than necessarily a global optimum. As usual in first-order non-convex optimization, such a critical point may be a local minimum or a saddle point (Mirzabeigi et al., 15 Jul 2025).

A notable limitation is explicit in the formulation: the analysis does not derive bounds such as $g_t = \nabla_\theta L(\theta_t)$7. The guarantees are instead framed in terms of monotonicity and qualitative stability. A plausible implication is that LyAm’s distinctive contribution lies less in asymptotic rate refinement than in linking a practical adaptive update rule to a control-theoretic stability criterion (Mirzabeigi et al., 15 Jul 2025).

5. Noise robustness

The noise model used in the analysis writes the stochastic gradient as

$g_t = \nabla_\theta L(\theta_t)$8

where $g_t = \nabla_\theta L(\theta_t)$9 is a stochastic noise vector with $i = 1,\dots,n$0 (Mirzabeigi et al., 15 Jul 2025). This is intended to cover mini-batch noise as well as label or input corruption under an unbiased-gradient assumption.

The paper identifies two mechanisms by which LyAm handles such noise. First, because $i = 1,\dots,n$1 is an exponential moving average of $i = 1,\dots,n$2, it reflects both signal and noise variance. When noise inflates the gradient magnitude, $i = 1,\dots,n$3 increases and the effective learning rate

$i = 1,\dots,n$4

decreases correspondingly. This is claimed to prevent large noise-induced jumps and to reduce oscillations and overshooting due to gradient spikes (Mirzabeigi et al., 15 Jul 2025).

Second, the optimizer retains bias-corrected first moments, so $i = 1,\dots,n$5 is used rather than raw gradients or uncorrected exponential moving averages. The reported motivation is that bias correction helps the first moment track the true gradient better in expectation and avoids the early-iteration bias that can make step sizes inconsistent (Mirzabeigi et al., 15 Jul 2025).

The Lyapunov-drift argument is then transferred to the noisy setting in expectation. Because the noise is zero-mean, the alignment term remains favorable on average, whereas the main effect of noise is to inflate $i = 1,\dots,n$6, which in turn shrinks the step size and controls the quadratic error term. Under sufficiently small $i = 1,\dots,n$7, the paper argues that

$i = 1,\dots,n$8

yielding robust convergence in expectation even under noisy gradients (Mirzabeigi et al., 15 Jul 2025).

6. Empirical evaluation

LyAm is evaluated on four computer vision benchmarks: CIFAR-10, CIFAR-100, Tiny ImageNet, and GTSRB (Mirzabeigi et al., 15 Jul 2025). The experiments cover both benign data and noisy or poisoned conditions. The noisy condition includes an example on CIFAR-10 in which 10% of training images are replaced by MNIST digits to simulate distribution shift and label noise.

The reported model architectures are ViT-16-B, ResNet-50, and VGG-16. Baselines include AdaGrad, Adam, AdamW, AdaBelief, and Adan, all run under comparable settings with the same model, dataset, and training length (Mirzabeigi et al., 15 Jul 2025).

Several concrete results are highlighted. On GTSRB with ViT-16-B, the reported accuracies are 87.78% for AdaGrad, 91.67% for Adam, 91.92% for AdamW, 92.56% for AdaBelief, 92.91% for Adan, and 94.37% for LyAm, which is the best value in that comparison (Mirzabeigi et al., 15 Jul 2025). On CIFAR-100 with ResNet-50, the reported values are 68.34% for Adam, 68.67% for AdamW, 69.34% for AdaBelief, 69.78% for Adan, and 70.56% for LyAm, again the best in the listed comparison. The optimizer is also reported to achieve the lowest validation loss and to have training times per epoch similar to other adaptive methods.

Under noisy conditions, the paper reports that on poisoned CIFAR-10 with ResNet-50, training and validation loss curves converge faster, appear smoother, and avoid the oscillations visible in Adam and AdamW. Training and validation accuracy curves are likewise described as stabilizing earlier and with less variability; one example states that LyAm stabilizes around approximately 65% accuracy under noise, while Adam and AdamW remain a few points lower with more fluctuations (Mirzabeigi et al., 15 Jul 2025). These observations are presented as empirical confirmation of the Lyapunov-guided damping mechanism.

7. Ablations, practical use, and limitations

The ablation study is conducted on CIFAR-10 with ResNet-50 (Mirzabeigi et al., 15 Jul 2025). Removing bias correction causes an accuracy drop, exemplified by a change from approximately 93.5% to approximately 90%, and also slows convergence while making it less stable. Disabling the adaptive scaling $i = 1,\dots,n$9 leads to higher final loss in noisy data, with an example of 0.90 versus 0.80, and produces more oscillatory training curves. Hyperparameter tuning indicates that the best performance is obtained for $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$0, $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$1, and $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$2 roughly in the range $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$3 to $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$4 (Mirzabeigi et al., 15 Jul 2025). The ablation table’s best configuration, designated Setup R, uses $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$5, $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$6, and a tuned $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$7, reaching 95.0% accuracy on benign CIFAR-10 and 86.0% on poisoned CIFAR-10.

From an implementation standpoint, LyAm introduces no new exotic hyperparameters beyond those already familiar from Adam: $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$8, $$m_{t,(i)} = \beta_1\, m_{t-1,(i)} + (1 - \beta_1)\, g_{t,(i)},$$9, and a base learning rate $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$0 (Mirzabeigi et al., 15 Jul 2025). The practical modification is confined to replacing Adam’s normalized step by the damped update $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$1. The paper suggests that existing Adam configurations can be adapted by keeping $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$2 and $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$3 unchanged, using a similar or slightly smaller $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$4, and substituting the update rule accordingly. It also notes that LyAm may be especially beneficial for noisy or corrupted datasets, unstable loss curves, and complex non-convex vision architectures such as ResNets and ViTs.

The limitations are explicitly stated. The convergence analysis remains qualitative rather than rate-based; the method still requires tuning $$v_{t,(i)} = \beta_2\, v_{t-1,(i)} + (1 - \beta_2)\, g_{t,(i)}^2,$$5; an excessively large base learning rate can violate the Lyapunov condition and produce instability in practice; and LyAm retains the common limitations of first-order methods, including convergence to saddle points and possible dependence on scheduling or warmup in some architectures (Mirzabeigi et al., 15 Jul 2025). The broader significance claimed for the optimizer is that it exemplifies a control-theoretic approach to robust deep learning optimization: training is treated as a dynamical system, the update rule is coupled to a stability guarantee, and robustness to imperfect data quality is elevated from a heuristic desideratum to an explicit design principle (Mirzabeigi et al., 15 Jul 2025).