Incremental Adam: Online & Adaptive Variants

• Incremental Adam is a family of Adam-based optimizers that uses per-sample or online updates to adapt moment statistics, leading to distinct implicit bias compared to full-batch methods.
• It includes an online learning perspective where update vectors are incrementally chosen via methods like discounted FTRL and confidence gating to handle nonstationary data.
• Variants such as AdamX and CAdam progressively modify momentum decay and update gating, highlighting how scheduling and data geometry affect convergence and stability.

Searching arXiv for papers on Incremental Adam and closely related Adam variants in incremental, online, and per-sample settings. Incremental Adam denotes a family of interpretations and variants of the Adam optimizer in which adaptation occurs at the level of individual updates, samples, or iterations rather than only through a fixed full-batch recursion. In the strictest sense, it refers to Adam run in a one-sample-per-step or online regime, where the optimizer updates its first- and second-moment statistics from incremental gradients. More broadly, the term also covers Adam-derived methods that modify Adam progressively over training time or gate Adam-style updates in streaming settings. The literature does not present a single canonical algorithm uniformly called “Incremental Adam”; instead, it spans several distinct lines of work: per-sample implicit-bias analysis for Adam (Baek et al., 30 Oct 2025), online-learning formulations in which Adam incrementally chooses update vectors (Ahn et al., 2024), online recommendation variants that preserve Adam’s moments but selectively suppress stale updates (Wang et al., 2024), schedule-based Adam-to-SGD transitions (Zhu et al., 17 Nov 2025), and foundational results showing that Adam’s non-convergence is not confined to stochastic or incremental settings (Bock et al., 2022). Together, these works show that the behavior of Adam under incremental operation depends strongly on the batching scheme, the data geometry, and the specific mechanism by which historical information is retained or attenuated.

1. Terminological scope and optimization settings

The literature distinguishes several meanings of “incremental Adam,” and keeping them separate is essential for technical precision. One meaning is incremental mode in the standard stochastic or online sense: Adam is run with individual samples or incremental gradients rather than full gradients. The batch-versus-incremental distinction is made explicitly in the non-convergence analysis of Adam, where batch mode uses the full gradient $\nabla f(w_t)$ of a deterministic objective at each iteration, whereas incremental mode refers to stochastic, online, or per-sample updates (Bock et al., 2022).

A second meaning arises in online learning of updates. In that formulation, the increment itself is the object chosen online: $\theta_{t+1} = \theta_t + \Delta_t,$ and Adam is interpreted as an online learner that incrementally selects $\Delta_t$ from past gradients (Ahn et al., 2024). This use of “incremental” concerns update vectors rather than sample access.

A third meaning concerns Adam variants that change continuously over training time. AdamX is exemplary: it is not incremental in the dataset-streaming sense, but it modifies Adam incrementally across iterations by using a time-varying second-moment decay that tends to $1$, thereby causing the method to become more SGD-like in the stable phase (Zhu et al., 17 Nov 2025). A fourth meaning is application-specific, as in the Adam-enhanced PSO-LFA model, where an “Adam incremental vector” denotes a PSO motion increment treated as a gradient-like quantity and adaptively rescaled by Adam-style moments (Chen et al., 2023).

This suggests that “Incremental Adam” is best treated as a topic family rather than a single optimizer. Within that family, the strictest and most theoretically developed usage is per-sample or online Adam, especially in separable linear classification (Baek et al., 30 Oct 2025) and online recommendation systems (Wang et al., 2024).

2. Classical Adam and the incremental/per-sample recursion

The core Adam recursion, as used across the cited literature, maintains first and second moment estimates: $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$

With bias correction,

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$

and the parameter update is

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$

This formulation appears directly in the online-learning and recommendation-system treatments of Adam (Wang et al., 2024).

In the strict per-sample setting studied in the implicit-bias literature, incremental Adam uses one sample per iteration. At time $t$, with cyclic order $i_t = t \bmod N$, the gradient is

$\theta_{t+1} = \theta_t + \Delta_t,$0

and the same Adam moment recursions are applied: $\theta_{t+1} = \theta_t + \Delta_t,$1

$\theta_{t+1} = \theta_t + \Delta_t,$2

$\theta_{t+1} = \theta_t + \Delta_t,$3

In that analysis, the stability term $\theta_{t+1} = \theta_t + \Delta_t,$4 is intentionally omitted because asymptotically it can dominate the vanishing second-moment estimate and alter the bias (Baek et al., 30 Oct 2025).

A related but distinct generalized Adam framework is used in nonconvex stochastic optimization. There, Adam-type algorithms are written abstractly as

$\theta_{t+1} = \theta_t + \Delta_t,$5

$\theta_{t+1} = \theta_t + \Delta_t,$6

$\theta_{t+1} = \theta_t + \Delta_t,$7

with $\theta_{t+1} = \theta_t + \Delta_t,$8 emphasized as the effective stepsize (Chen et al., 2018). That framework includes Adam, AMSGrad, AdaGrad, and AdaFom, but it does not define a deterministic cyclic optimizer explicitly named “Incremental Adam” (Chen et al., 2018).

3. Per-sample implicit bias: departure from the full-batch regime

The most direct theoretical treatment of incremental Adam studies binary linear classification with separable data and logistic or exponential loss: $\theta_{t+1} = \theta_t + \Delta_t,$9 or

$\Delta_t$0

Under linear separability, optimization drives $\Delta_t$1, so the asymptotic direction $\Delta_t$2 encodes the optimizer’s implicit bias (Baek et al., 30 Oct 2025).

Prior full-batch analyses had established that Adam favors $\Delta_t$3-geometry and converges to the $\Delta_t$4-max-margin classifier. The per-sample analysis shows that this full-batch conclusion does not transfer automatically to incremental Adam (Baek et al., 30 Oct 2025). The paper derives an epoch-wise approximation for incremental Adam: $\Delta_t$5 where $\Delta_t$6. By contrast, full-batch Adam asymptotically satisfies a sign-like approximation: $\Delta_t$7 with $\Delta_t$8 (Baek et al., 30 Oct 2025). This difference in asymptotic update geometry is the core reason incremental Adam need not inherit the $\Delta_t$9-bias of deterministic Adam.

The clearest theorem concerns Generalized Rademacher (GR) data, for which every sample has coordinates of equal magnitude: $1$0 On such data, the incremental Adam epoch update simplifies to a weighted normalized-GD form: $1$1 with uniformly bounded weights $1$2. Under the stated assumptions, the resulting direction converges to the $1$3-max-margin classifier: $1$4 This establishes a direct separation: full-batch Adam aligns with $1$5-margin, whereas incremental Adam can align with $1$6-margin on the same structured separable problem (Baek et al., 30 Oct 2025).

For general datasets, the same work introduces the proxy update

$1$7

which captures the $1$8 regime of incremental Adam (Baek et al., 30 Oct 2025). Its asymptotic direction is characterized through a data-dependent fixed point. There exists $1$9 such that

$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$0

with support restricted to support vectors. The associated primal problem is

$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$1

where

$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$2

Under the continuity and LICQ assumptions, the direction is given implicitly by some fixed point $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$3 of the normalized dual map: $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$4 Thus the asymptotic bias of incremental Adam is neither universally $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$5 nor universally $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$6; it is data-dependent and batching-dependent (Baek et al., 30 Oct 2025).

4. Incremental Adam as online learning of updates

A conceptually different line of work treats Adam as an algorithm that incrementally chooses update vectors rather than directly as a parameter-space recursion. The optimization step is written

$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$7

and the online learner chooses $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$8 from past gradients $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,$9 (Ahn et al., 2024). In this framework, Adam is shown to be a coordinate-wise discounted FTRL method. Using a quadratic regularizer $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$0, FTRL chooses

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$1

With the specific discounted inputs

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$2

the resulting coordinate-wise solution becomes

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$3

This is the paper’s simplified Adam form without debiasing and $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$4 (Ahn et al., 2024).

The significance of this view for Incremental Adam is twofold. First, it gives a precise sense in which Adam is already incremental: it incrementally selects update vectors online. Second, it recasts Adam’s algorithmic components in online-learning terms. Momentum becomes cumulative discounted loss information, exponential discounting becomes forgetting of stale losses, and coordinate-wise adaptive scaling becomes a scale-free learning-rate mechanism (Ahn et al., 2024).

The associated theory is framed through dynamic regret rather than descent-to-stationarity. Discounted FTRL over updates yields discounted regret bounds, which are then converted to dynamic regret via an exact decomposition (Ahn et al., 2024). This suggests that “incremental Adam” can be understood not only as per-sample stochastic Adam, but also as a principled online learner over increments.

5. Online and streaming variants: confidence gating and stale-state control

In online learning settings with distribution shift and noisy feedback, Adam’s historical state can become harmful because its first- and second-moment EMAs summarize outdated data. CAdam addresses this by leaving Adam’s moment recursions unchanged while introducing a per-coordinate confidence gate (Wang et al., 2024). Standard Adam uses

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$5

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$6

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$7

with the usual Adam update

$v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$8

CAdam computes the same $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2.$9 and $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$0, but only uses coordinate $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$1 when momentum and gradient agree in sign: $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$2 equivalently,

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$3

The update is then

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$4

or with AMSGrad-style denominator when that variant is used (Wang et al., 2024).

This mechanism is explicitly motivated by online recommendation systems with concept drift and noisy feedback. If the old trend encoded by $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$5 conflicts with current evidence $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$6, the coordinate is treated as low-confidence and its update is withheld. The paper defines an Alignment Ratio

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$7

used diagnostically rather than as part of the rule (Wang et al., 2024).

Under convexity, bounded gradients, bounded domain, and decaying $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$8, the paper proves sublinear regret: $$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t},$$9 and in the appendix states the explicit upper bound

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$0

In this sense CAdam is one of the most literal online or incremental Adam variants in the literature: it preserves Adam’s moment machinery and modifies only whether a coordinate is allowed to apply the current Adam step (Wang et al., 2024).

6. Progressive and schedule-based Adam modifications

A separate branch of work uses “incremental” in the sense of iteration-dependent morphing of Adam over training time. AdamX modifies only the second-moment decay mechanism, replacing Adam’s effective second-moment coefficient with

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$1

while also expressing the first moment in recursive bias-corrected form

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$2

Its second moment becomes

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$3

The normalized direction is

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$4

and the parameter update is

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$5

with

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$6

The crucial distinction from vanilla Adam is asymptotic: $$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$7 The intended consequence is that the denominator becomes increasingly inertial, so late training behaves more like momentum SGD (Zhu et al., 17 Nov 2025).

AdamX is therefore an “incremental Adam” in a schedule-based sense: the optimizer is changed a little at every step rather than switched abruptly. The paper presents this as a gradual Adam-to-SGD transition, not a hard handoff (Zhu et al., 17 Nov 2025). This suggests a broader taxonomy in which Incremental Adam includes not only sample-wise Adam but also optimizers that progressively reduce Adam’s adaptivity.

Other Adam-family refinements operate by modifying the preconditioner signal itself. AdaMomentum keeps Adam’s first-moment update

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$8

but replaces the second moment

$$\theta_t = \theta_{t-1} - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}.$$9

with

$t$0

Bias-corrected moments are then used in

$t$1

This is a small structural change, but it replaces raw-gradient second-moment tracking by momentumized second-moment tracking, which the paper argues improves both optimization behavior and generalization (Wang et al., 2021). This suggests another plausible sense of “incremental Adam”: a minimal refinement of Adam that preserves the moment structure and state size while changing only one internal statistic.

7. Convergence pathologies, theory limits, and practical implications

The non-convergence literature is critical context for Incremental Adam because earlier failures of Adam could be attributed to stochasticity or online formulations. A decisive result is that even deterministic batch Adam with bias correction can exhibit period-2 limit cycles on the scalar quadratic

$t$2

In one dimension, the updates are

$t$3

$t$4

and with bias correction

$t$5

For the bias-uncorrected autonomous system at $t$6, the paper gives an explicit 2-cycle

$t$7

with second point

$t$8

These cycles persist for small $t$9 except at an exceptional value

$i_t = t \bmod N$0

and therefore Adam is not globally convergent even on strictly convex quadratics (Bock et al., 2022).

This matters for Incremental Adam because the same paper states that prior work had already provided “an example in incremental mode where the regret does not converge, neither do the arguments of the objective function,” citing Reddi et al. and emphasizing that its own results have “no direct implication on efforts to prove convergence of Adam in the incremental mode under additional assumptions” (Bock et al., 2022). The conceptual point is that non-convergence is not merely a stochastic artifact. Incremental failures occur in a broader context of unstable adaptive-moment dynamics.

By contrast, the general nonconvex Adam-type theory provides positive guarantees only under controlled assumptions. For a broad generalized Adam framework, the stationarity rate

$i_t = t \bmod N$1

is obtained for Adam-type methods when the oscillation of effective stepsizes is controlled (Chen et al., 2018). The theory emphasizes two critical terms: $i_t = t \bmod N$2 and

$i_t = t \bmod N$3

The second term measures effective-stepsize oscillation and is identified as a major reason Adam can fail (Chen et al., 2018). That analysis supports AMSGrad and AdaFom, not original Adam in a deterministic incremental finite-sum sense (Chen et al., 2018).

A related misconception is that full-batch theory automatically captures per-sample Adam. The implicit-bias results show the opposite: batching can fundamentally alter Adam’s geometry (Baek et al., 30 Oct 2025). Another misconception is that bias correction guarantees convergence; the quadratic limit-cycle results explicitly show otherwise in batch mode (Bock et al., 2022). A plausible implication is that any encyclopedia treatment of Incremental Adam must foreground not only its variants and applications but also the nontrivial dependence of its asymptotic behavior on batch regime, second-moment construction, and stale-state handling.

8. Representative formulations and variants

Variant or interpretation	Core modification	Principal setting
Per-sample incremental Adam	Uses $i_t = t \bmod N$4 in Adam recursion	Separable logistic/exponential regression (Baek et al., 30 Oct 2025)
Adam as online learning of increments	Chooses $i_t = t \bmod N$5 by discounted FTRL	Online learning of updates (Ahn et al., 2024)
CAdam	Masks $i_t = t \bmod N$6 by $i_t = t \bmod N$7	Online recommendation under drift/noise (Wang et al., 2024)
AdamX	Uses time-varying $i_t = t \bmod N$8	Progressive Adam-to-SGD transition (Zhu et al., 17 Nov 2025)
AdaMomentum	Replaces $i_t = t \bmod N$9 by $\theta_{t+1} = \theta_t + \Delta_t,$00 in second moment	Adam-family refinement (Wang et al., 2021)

These formulations are not interchangeable. Per-sample incremental Adam concerns stochastic data access and asymptotic bias. The FTRL formulation concerns online selection of update vectors. CAdam concerns robustness to stale momentum in nonstationary streams. AdamX concerns late-stage reduction of adaptivity. AdaMomentum concerns a preconditioner refinement within Adam’s standard structure.

9. Applications and domain-specific interpretations

Incremental Adam variants are most directly motivated by settings in which full-batch stationary assumptions are implausible. Online recommendation is the clearest case. There, the learner faces a sequence of losses

$\theta_{t+1} = \theta_t + \Delta_t,$01

and seeks to minimize regret against the best fixed comparator: $\theta_{t+1} = \theta_t + \Delta_t,$02 CAdam is explicitly designed for this regime, where user behavior, traffic mix, and item popularity shift over time, making Adam’s retained state potentially stale (Wang et al., 2024).

Another application-specific interpretation appears in latent factor analysis with PSO. There, the “Adam incremental vector” is not a stochastic sample gradient but a PSO displacement term: $\theta_{t+1} = \theta_t + \Delta_t,$03 which is then used in Adam-style moment recursions

$\theta_{t+1} = \theta_t + \Delta_t,$04

$\theta_{t+1} = \theta_t + \Delta_t,$05

leading to the adaptive increment

$\theta_{t+1} = \theta_t + \Delta_t,$06

The resulting velocity is set to

$\theta_{t+1} = \theta_t + \Delta_t,$07

This is an Adam-inspired adaptive increment inside PSO rather than deep-learning Adam proper (Chen et al., 2023).

These examples show that “Incremental Adam” is a flexible design pattern: a history-dependent adaptive scaling rule applied incrementally to whatever object is being updated—model parameters, online coordinates, optimizer increments, or particle displacements.

10. Synthesis

Incremental Adam is not a single optimizer with a universally accepted definition. The term covers at least three major ideas. In the narrow and most important theoretical sense, it denotes per-sample or online Adam, whose asymptotic bias can differ fundamentally from full-batch Adam; in separable linear classification, it can converge to the $\theta_{t+1} = \theta_t + \Delta_t,$08-max-margin classifier rather than the $\theta_{t+1} = \theta_t + \Delta_t,$09-max-margin classifier (Baek et al., 30 Oct 2025). In the online-learning sense, Adam is itself incremental because it can be derived as a discounted FTRL algorithm over update vectors $\theta_{t+1} = \theta_t + \Delta_t,$10, making it an online learner of increments (Ahn et al., 2024). In the optimizer-design sense, Incremental Adam includes variants that change Adam gradually over time or gate Adam-style updates in nonstationary streams, such as AdamX and CAdam (Zhu et al., 17 Nov 2025, Wang et al., 2024).

The literature also establishes important limits. Counterexamples to incremental-mode convergence already existed, and deterministic batch Adam itself exhibits non-convergent period-2 limit cycles on simple convex quadratics, so Adam’s difficulties cannot be dismissed as mere stochastic artifacts (Bock et al., 2022). General convergence guarantees for Adam-type methods exist, but they rely on assumptions controlling effective-stepsize oscillation and support variants such as AMSGrad and AdaFom more directly than original Adam (Chen et al., 2018).

Taken together, these results imply that Incremental Adam should be understood not as “Adam plus small-batch noise,” but as a family of adaptive moment methods whose long-run geometry, stability, and practical utility are shaped by the interaction among samplewise updating, retained state, and the internal construction of the adaptive denominator.