First-Order Nonlocal Adam Flow

• The paper introduces a continuous-time Adam model where parameter updates follow a first-order evolution combined with nonlocal memory integrals.
• It employs exponentially weighted kernels to capture past gradients, bridging the gap between AdaGrad, RMSProp, and Adam through continuous bias correction.
• Numerical validations show that the model closely replicates discrete dynamics, offering insights for refinements such as second-order inertial systems.

First-Order Nonlocal Adam Flow denotes a continuous-time representation of Adam in which the parameter trajectory satisfies a first-order evolution equation, while the velocity depends on memory terms built from past gradients and past squared gradients. In this sense, the flow is first-order in the parameter variable but nonlocal in time, because it depends on Volterra-type history integrals rather than only on the instantaneous state. The foundational formulation appears in the continuous-time modeling of AdaGrad, RMSProp, and Adam, where Adam is written as a first-order integro-differential equation; subsequent work treats that construction as an existing baseline continuous-time model and refines it, rather than replacing it (Heredia, 2024, Heredia, 9 Feb 2026).

1. Foundational definition

The core continuous-time Adam model is given by the equation

$$\dot{\theta}^i(t)=-\alpha(t)\,\frac{m^i(t,\theta)}{\sqrt{v(t,\theta)}+\epsilon(t)},$$

with continuous bias-correction factors

$$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$

The first and second moments are represented by history-dependent functionals,

$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$

with exponential kernels

$$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$

The same paper notes a small notational inconsistency: the appendix formulas for $m$ and $v$ include an extra factor of $\tau$, and that factor does not appear in the proposition statements for AdaGrad and RMSProp; it is reported there as likely a typographical artifact, but the intended structure is still clear—both $m$ and $v$ are exponentially weighted memory functionals of past gradients (Heredia, 2024).

The phrase “first-order nonlocal Adam flow” is therefore an accurate characterization of the model even though the original paper does not literally use that exact phrase. The continuous-time Adam state consists of the parameter trajectory $\theta(t)$, the first-moment functional $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$0, and the second-moment functional $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$1. The model is first-order because only $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$2 appears, and it is nonlocal because the right-hand side depends on the entire past trajectory $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$3, not only on $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$4 (Heredia, 2024).

2. Derivation from discrete Adam

The derivation begins from standard Adam with bias correction: $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$5

$$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$6

$$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$7

$$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$8

The continuum limit is obtained under the identifications $$\alpha(t)=\frac{\sqrt{1-\beta_2^{\,t/\alpha}}}{1-\beta_1^{\,t/\alpha}}, \qquad \epsilon(t)=\epsilon\sqrt{1-\beta_2^{\,t/\alpha}}.$$9 and

$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$0

which is an Euler-type scaling in which one optimizer step corresponds to a time increment of length $$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$1 (Heredia, 2024).

After shifting indices so that gradients are evaluated at $$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$2, the moment recursions become first-order difference equations. Under the scaling $$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$3, these produce the filter ODEs

$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$4

$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$5

Solving these by integrating factors yields exponentially weighted convolution formulas for $$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$6 and $$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$7. The resulting integro-differential form and the coupled ODE form are equivalent descriptions of the same continuous-time Adam dynamics: one keeps the auxiliary moment variables explicitly, while the other eliminates them and exposes the temporal nonlocality directly (Heredia, 2024).

This equivalence is structurally important. It shows that the first-order nonlocal Adam flow is not a separate optimizer from the filter-ODE representation; it is the same continuous model viewed after eliminating the auxiliary states. A common misconception is to treat the integral formulation as fundamentally different from a first-order system with extra variables. In this literature, the distinction is representational rather than dynamical.

3. Memory kernels and relation to AdaGrad and RMSProp

The continuous-time models for AdaGrad, RMSProp, and Adam fit a common pattern: the parameter update is first-order in time and is adaptively normalized by a history-dependent denominator. What distinguishes them is the memory kernel and, for Adam, the presence of a filtered numerator as well (Heredia, 2024).

Method	Memory kernel	Defining feature
AdaGrad	$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$8	full accumulation without decay
RMSProp	$$m^i(t,\theta)=\int_0^t \tau\,K_1(t-\tau)\,\partial^i f(\theta(\tau))\,d\tau, \qquad v(t,\theta)=\int_0^t \tau\,K_2(t-\tau)\,\partial^i f(\theta(\tau))\,\partial_i f(\theta(\tau))\,d\tau,$$9	exponentially fading second-moment memory
Adam	$$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$0, $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$1	two memory channels and continuous bias correction

For AdaGrad, the model is

$$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$2

Because the kernel is constant, AdaGrad has infinite memory without decay. RMSProp replaces this with exponentially fading memory: $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$3

$$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$4

Adam then combines an RMSProp-type denominator with a first-moment exponential moving average in the numerator and continuous-time bias correction through $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$5 and $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$6 (Heredia, 2024).

The nonlocality is explicit in convolution notation: $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$7 modulo the previously noted appendix $$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$8-factor issue. This makes Adam qualitatively different from standard gradient flow

$$K_a(t)=\frac{1-\beta_a}{\alpha}e^{-\frac{1-\beta_a}{\alpha}t}, \qquad a=1,2.$$9

which is local in state. In Adam, recent gradients are weighted more heavily, but older gradients remain present through exponentially decaying kernels. The paper explicitly interprets these integral terms as “memory effects” and “cumulative effects” (Heredia, 2024).

4. Mathematical interpretation and later theoretical development

The original continuous-time modeling paper is primarily a modeling and numerical paper. It does not provide a full well-posedness theory, convergence proof, or Lyapunov analysis for the Adam integro-differential equation. Its main theoretical claim is instead that the continuous equations are accurate approximations of the corresponding discrete algorithms, especially for small learning rates, under the formal Euler-limit expansion used in the derivation (Heredia, 2024).

Later work makes this first-order nonlocal flow mathematically explicit as the baseline continuous-time Adam model. In that formulation,

$m$0

where

$m$1

$m$2

and

$m$3

The same quantities are equivalently characterized by first-order filter ODEs for $m$4 and $m$5, with $m$6. In that paper, the second moment is scalar or isotropic rather than coordinatewise (Heredia, 9 Feb 2026).

That later analysis assumes, for comparison and Lyapunov-transfer arguments, that $m$7, is bounded below, has compact sublevel set

$m$8

that the trajectory remains in $m$9, that $v$0 is Lipschitz on $v$1, and often that $v$2. It then introduces a second-order inertial Adam model and proves that this inertial system is an $v$3-refinement of the pre-existing first-order nonlocal Adam flow. In particular, away from the initial layer, the second-order dynamics reduce to the first-order flow up to a remainder

$v$4

and for fixed $v$5, $v$6 (Heredia, 9 Feb 2026).

The same work also imports a Lyapunov structure from the first-order model: $v$7 For the perturbed inertial model it proves inequalities of the form

$v$8

and

$v$9

followed by PL-, strong-convexity-, and KL-based residual bounds. These are not first-order theorems proved from scratch for the original nonlocal flow; rather, they are second-order stability results obtained by perturbing around it (Heredia, 9 Feb 2026).

5. Related notions of “nonlocal” Adam dynamics

The phrase “nonlocal Adam flow” has acquired several distinct meanings in later work. The original meaning is temporal nonlocality induced by memory kernels. Later papers preserve the first-order character but change what counts as nonlocality.

At the edge of stability, “rod flow for Adam” is a first-order ODE on the augmented state $\tau$0, where nonlocality arises from endpoint evaluations at the two rod endpoints $\tau$1 extracted from the extent tensor $\tau$2. The governing equations involve

$\tau$3

$\tau$4

$\tau$5

This model is first-order, but its nonlocality is geometric rather than temporal: the vector field depends on gradients at spatially separated rod endpoints, not on convolutional time memory alone (Regis et al., 7 May 2026).

In probability space, AdamFlow generalizes Adam to a continuity-equation formulation on the augmented state $\tau$6. Its PDE is

$\tau$7

with $\tau$8. Here the nonlocality comes from the full measure dependence of $\tau$9, and for sliced Wasserstein objectives it is global through projected empirical distributions and one-dimensional transport maps. This is again a first-order nonlocal Adam flow, but now in measure space rather than Euclidean parameter space (Ma et al., 2 Apr 2026).

A different ODE approximation arises in the fast–slow regime with fixed $m$0 and vanishing step sizes. There the limiting dynamics close on $m$1 alone: $m$2 where the Adam vector field $m$3 is defined by infinite-history stationary averages of frozen-$m$4 gradients and squared gradients. This produces a first-order autonomous ODE whose drift incorporates stationary memory, so it can be interpreted as a memory-induced effective flow even though the final equation is local in $m$5 as written (Dereich et al., 6 Nov 2025).

These constructions are not interchangeable. The original nonlocal Adam flow is a causal time-memory IDE; rod flow is nonlocal in phase-space geometry; AdamFlow is nonlocal through measure coupling; and the fast–slow Adam vector field is an effective first-order closure after averaging out fast adaptive states. The shared theme is first-order dynamics with nonlocal dependence that is absent from ordinary local gradient flow.

6. Numerical validation, accuracy, and interpretation

The original numerical validation compares discrete Adam with its continuous nonlocal model on the scalar convex objective

$m$6

The reported configurations include $m$7, $m$8, and learning rates $m$9 and $v$0. The paper reports that the continuous nonlocal Adam model reproduces the discrete dynamics very closely, that agreement improves as the learning rate decreases, that at $v$1 the continuous and discrete trajectories are “almost exactly the same,” and that at $v$2 small differences appear, such as slightly different peaks or mild initial divergence in some $v$3 configurations. Oscillations for higher $v$4 at larger learning rates are also qualitatively captured. The same paper reports close agreement on a simple MSE fitting problem $v$5 approximating $v$6, again with better matching at smaller learning rates (Heredia, 2024).

The numerical solver there is an iterative integro-differential solver inspired by IDESolver, using Gaussian quadrature for the integral terms and Euler discretization for the differential part. For AdaGrad and RMSProp, interpolation is used to evaluate the $v$7 denominator shift; Adam does not require that future-time shift in the paper’s final formulation. This numerical setup reinforces the paper’s central claim that the integro-differential flow is a practical approximation, not only a formal analogy (Heredia, 2024).

Subsequent numerical work keeps the first-order nonlocal flow as a reference model while comparing it to more elaborate continuous-time dynamics. The second-order inertial refinement paper reports that on a one-dimensional Rosenbrock-type loss, the second-order model becomes consistently more accurate than the first-order one as $v$8 decreases, supporting the claim that the inertial system is an $v$9-refinement of the first-order nonlocal flow. At the same time, the first-order nonlocal flow already tracks discrete Adam well and serves as the baseline limit recovered as $\theta(t)$0 (Heredia, 9 Feb 2026). In the edge-of-stability setting, Adam rod flow is reported to track the centers of discrete iterates several orders of magnitude more closely than stable flow, across MLP, CNN, and ViT experiments, though that construction targets a different oscillatory regime and a different notion of nonlocality (Regis et al., 7 May 2026).

Two clarifications follow from this body of work. First, the first-order nonlocal Adam flow is not a second-order inertial model; when inertial terms are introduced, they are presented as refinements or generalizations, not as the original continuous-time Adam baseline. Second, the phrase does not designate a single universally adopted equation across all later papers. It most precisely refers to the continuous-time Adam IDE with exponentially weighted causal memory in the numerator and denominator, from which later rod, inertial, measure-valued, and averaged-vector-field formulations depart in different directions.