Delayed Momentum Aggregation

• Delayed momentum aggregation is a set of optimization strategies that integrate stale model updates to construct robust momentum buffers in asynchronous and federated learning scenarios.
• It employs techniques like ordered momentum and momentum approximation to counteract bias from delayed gradients, ensuring correct chronological weighting.
• These methods improve convergence speed, robustness, and communication efficiency by retuning momentum based on system delays and asynchronous update patterns.

Delayed momentum aggregation denotes optimization schemes in which momentum information is formed from updates that are delayed, stale, or only periodically synchronized rather than being computed solely from the current gradient at the current iterate. In the most specific usage, it is the federated-learning principle of aggregating the most recently received gradients from non-participating clients alongside fresh momentum from active clients (Otsuka et al., 3 Sep 2025). Closely related mechanisms appear in asynchronous SGD, asynchronous and private federated learning, hierarchical federated learning, and DiLoCo-style low-communication training, where delay variables, synchronization intervals, or outer-round pseudo-gradients cause momentum to be computed from temporally misaligned local trajectories (Mitliagkas et al., 2016, Yu et al., 2024, Yang et al., 2022, Horoi et al., 3 Jul 2026). This suggests that delayed momentum aggregation is best understood as a family of delay-aware momentum constructions rather than a single algorithm.

1. Formal scope and recurring mathematical structure

Across the literature, the common object is a momentum buffer or momentum-like recurrence that receives information computed on earlier models. In fully asynchronous SGD, a worker reads a stale iterate

$v_t = w_{t-\tau_t},$

and performs

$w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$

where $\tau_t\in\mathbb N$ is the staleness. In partial-participation federated learning, the server stores a client-wise momentum vector $m_i^t$ and aggregates both refreshed and retained vectors. In DiLoCo, workers take $H$ local steps and communicate only an outer-round displacement

$\Delta_i^k = w_i^{k,H} - w^k,$

so momentum is applied only to periodically aggregated pseudo-gradients (Mitliagkas et al., 2016, Otsuka et al., 3 Sep 2025, Horoi et al., 3 Jul 2026).

These formulations differ in topology and fault model, but they share the same structural issue: the optimizer must decide how delayed information should contribute to a momentum buffer. Some methods reinterpret delay as implicit momentum, some compensate for the distortion introduced by stale arrivals, and some exploit stored stale vectors to preserve robustness or reduce communication.

Setting	Delayed object	Representative mechanism
Asynchronous SGD	stale gradients or stale iterates	implicit momentum, ordered momentum, ordered weights
Asynchronous/private FL	buffered model updates and version tags	momentum approximation via reweighting
Partial-participation Byzantine FL	server-stored client momentum vectors	aggregation of fresh and stale momentum
DiLoCo and related methods	outer-round pseudo-gradients after $H$ local steps	Nesterov momentum on aggregated task vectors
Hierarchical FL	worker, edge, and cloud momentum streams	multi-tier momentum aggregation every $\tau$ and $\tau\pi$ steps

A key distinction within this family is whether delay is treated as a nuisance to be attenuated, as an implicit source of acceleration, or as a resource to be retained for stability and robustness.

2. Asynchrony as an implicit momentum mechanism

A foundational result is that asynchronous SGD itself can be rewritten as a momentum-like method. Under the assumption that delays are independent of the sample-selection process and with constant step size $\alpha$, the expected increment obeys

$w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$0

where $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$1 is the staleness distribution. Whenever delays have variance $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$2, past gradients reappear with positive weight, creating a momentum-like memory (Mitliagkas et al., 2016).

The geometric case is especially explicit. If

$w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$3

then the expected async update collapses to

$w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$4

In expectation, the asynchronous update exactly matches a Polyak-momentum step with implicit momentum coefficient $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$5, and this equivalence does not require convexity of $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$6 (Mitliagkas et al., 2016).

The same work connects the coefficient $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$7 to a homogeneous queueing model. If $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$8 workers have i.i.d. $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$9 update times, the induced staleness law is geometric with

$\tau_t\in\mathbb N$0

yielding

$\tau_t\in\mathbb N$1

This forges a direct link between worker count and effective momentum (Mitliagkas et al., 2016).

Once explicit algorithmic momentum $\tau_t\in\mathbb N$2 is added, the effective momentum is approximately additive, $\tau_t\in\mathbb N$3. That observation leads to two design consequences. First, momentum must be retuned as asynchrony changes. Second, when $\tau_t\in\mathbb N$4 becomes large, negative algorithmic momentum can be optimal. The paper reports that on CIFAR with 16 CPU workers, tuning $\tau_t\in\mathbb N$5 from the “industry standard” $\tau_t\in\mathbb N$6 to its $\tau_t\in\mathbb N$7-dependent optimum yields a $\tau_t\in\mathbb N$8 reduction in the number of training steps needed to reach the same loss, and allowing $\tau_t\in\mathbb N$9 yields a further $m_i^t$0 speed-up at very large staleness (Mitliagkas et al., 2016).

3. Chronological weighting, bias correction, and ordered asynchronous momentum

A recurrent criticism of naive momentum under delay is that it mixes gradients of different vintages in a single buffer and can repeatedly recycle stale information. In parameter-server ASGD, the naive rule

$m_i^t$1

often impedes convergence as delay grows. Ordered Momentum (OrMo) addresses this by grouping gradients by their original iteration index into buckets of size $m_i^t$2, maintaining a head-bucket index $m_i^t$3, and ensuring that the $m_i^t$4-th bucket contributes exactly with weight $m_i^t$5. When a stale gradient arrives, the server compensates in one shot for all the micro-steps it missed before arrival, so no gradient is over-counted. Under standard non-convex assumptions, OrMo attains the same $m_i^t$6 rate as serial SGD up to a small $m_i^t$7 term, without any bounded-delay assumption (Shi et al., 2024).

The chronological-weighting problem also appears in asynchronous federated learning with buffered updates. In that setting, a cohort of clients returns stale model displacements $m_i^t$8, which the server averages into a pseudo-gradient

$m_i^t$9

If one then applies ordinary server-side momentum, the resulting asynchronous trajectory differs from the synchronous one by an implicit momentum bias matrix

$H$0

where $H$1 is the ideal exponential-weight matrix and $H$2 is the received-average operator determined by staleness. Momentum Approximation (MA) corrects this by solving, row by row,

$H$3

with closed-form solution

$H$4

A light-weight variant uses a scalar recursion $H$5 and solves a $H$6 least-squares problem each round. Because MA is post-processing of aggregated sums, it is compatible with secure aggregation and inherits the same $H$7 differential-privacy guarantee as the underlying noisy sums. Empirically, MA and MA-light achieve $H$8 speed up in convergence compared to naively combining asynchronous FL with momentum, with $H$9 relative improvement in macro-AP or accuracy; under DP with $\Delta_i^k = w_i^{k,H} - w^k,$0, the speed-up remains $\Delta_i^k = w_i^{k,H} - w^k,$1 (Yu et al., 2024).

A more recent line studies data-dependent delays directly. The update

$\Delta_i^k = w_i^{k,H} - w^k,$2

assigns each delayed gradient exactly the weight it would have had in a synchronous exponential moving average. The paper establishes the first optimal convergence rates for data-dependent delays in both convex and non-convex smooth setups, avoids filtering or discarding delayed gradients, and supports up to $\Delta_i^k = w_i^{k,H} - w^k,$3 workers without degrading the $\Delta_i^k = w_i^{k,H} - w^k,$4 rate (Dahan et al., 3 May 2026). This positions ordered weighting as an alternative to staleness filtering, which the same work characterizes as systematically favoring simpler or faster-to-process samples.

4. Federated learning with partial participation, robustness, privacy, and hierarchy

In partial-participation Byzantine-robust FL, delayed momentum aggregation is formulated explicitly as a server principle. Each client $\Delta_i^k = w_i^{k,H} - w^k,$5 has a momentum vector $\Delta_i^k = w_i^{k,H} - w^k,$6. If $\Delta_i^k = w_i^{k,H} - w^k,$7, it is refreshed; otherwise it is retained. The server then aggregates all $\Delta_i^k = w_i^{k,H} - w^k,$8 vectors,

$\Delta_i^k = w_i^{k,H} - w^k,$9

The purpose is to ensure that the robust aggregator always sees $H$0 vectors with only a $H$1-fraction Byzantine, even when the sampled set itself could contain a Byzantine majority. Under a $H$2-robust aggregator and standard smoothness, variance, heterogeneity, and bounded-gradient assumptions, D-Byz-SGDM satisfies

$H$3

and a lower bound shows that the non-vanishing $H$4 term is information-theoretically optimal (Otsuka et al., 3 Sep 2025).

The same work emphasizes that communication efficiency is preserved. The per-round communication cost is $H$5, storage is $H$6, and no extra communication is required to retrieve stale momenta because they are stored server-side. In experiments on MNIST with a small CNN, $H$7, $H$8, and $H$9, D-Byz-SGDM consistently achieves the highest final accuracy across all aggregators and attacks, while FedAvg and FedAvg-M diverge within a few epochs whenever a Byzantine majority is sampled (Otsuka et al., 3 Sep 2025).

Hierarchical federated learning extends delayed momentum aggregation to a three-tier worker-edge-cloud network. In HierMo, workers maintain local momentum vectors $\tau$0, edge nodes aggregate stale worker momenta every $\tau$1 local steps, and the cloud aggregates stale edge-level quantities every $\tau$2 steps. The delays are explicitly represented by $\tau$3 and $\tau$4. The analysis introduces virtual edge and cloud sequences to characterize model aggregation, momentum aggregation, and their interactions, establishing an $\tau$5 convergence rate and a tighter convergence upper bound compared with HierFAVG without momentum (Yang et al., 2022). This broadens the topic from stale gradients at a single server to delayed momentum exchange across communication tiers.

5. Periodic outer-loop momentum in low-communication distributed training

DiLoCo and related local-SGD methods instantiate delayed momentum aggregation through periodic synchronization. With global model $\tau$6, worker $\tau$7 takes $\tau$8 local steps from $\tau$9 to $\tau\pi$0, forming a task vector

$\tau\pi$1

The outer optimizer then applies Nesterov momentum to these pseudo-gradients: $\tau\pi$2 Momentum is therefore delayed by $\tau\pi$3 local steps: unlike classical distributed SGD, the buffer is refreshed only at outer communication rounds (Horoi et al., 3 Jul 2026).

The model-merging view makes this explicit. Iso-C treats each $\tau\pi$4 as a task vector and, for each matrix parameter tensor $\tau\pi$5, computes an SVD

$\tau\pi$6

forms the mean singular value

$\tau\pi$7

and reconstructs the isotropic update

$\tau\pi$8

IsoLoCo inserts this correction into the DiLoCo outer loop and retains Nesterov momentum. The paper interprets the degradation at large $\tau\pi$9 or large $\alpha$0 as arising from the aggregation of increasingly divergent local trajectories and from delayed outer momentum that may need retuning (Horoi et al., 3 Jul 2026).

The empirical results quantify the scaling behavior on a 178 M-parameter LLaMa-style model trained on DCLM under a fixed FLOP budget, with $\alpha$1 unless otherwise noted.

Method	Setting	Best validation loss
Data-Parallel AdamW	DP	2.881
DiLoCo	$\alpha$2	2.869, 2.950, 3.355
DiLoCo + Iso-C	$\alpha$3	2.855, 2.929, 3.280
IsoLoCo	$\alpha$4	2.842, 2.889, 3.160

For inner-step scaling at $\alpha$5, DiLoCo changes from $\alpha$6 at $\alpha$7 to $\alpha$8 at $\alpha$9, while IsoLoCo changes from $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$00 to $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$01. The absolute gap between DiLoCo and IsoLoCo widens with $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$02: at $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$03 it is $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$04 $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$05, and at $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$06 it is $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$07 $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$08 (Horoi et al., 3 Jul 2026). This suggests that delayed momentum aggregation in outer-loop optimizers is closely connected to model-merging interference and not only to classical gradient staleness.

6. Trade-offs, misconceptions, and broader algorithmic significance

A central systems trade-off is that delay often improves hardware or communication efficiency while harming statistical efficiency unless the momentum mechanism is redesigned. In asynchronous deep learning, better hardware efficiency comes from no locks, but a single default momentum across all scales of asynchrony can either waste hardware parallelism or suffer slow convergence; fair comparison of synchronous and asynchronous schemes therefore requires retuning momentum for each level of asynchrony (Mitliagkas et al., 2016). In local-SGD and DiLoCo, communication is reduced by a factor of $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$09, but the outer optimizer sees stale pseudo-gradients from divergent local trajectories, so performance degrades as $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$10 or $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$11 grows unless aggregation is improved (Horoi et al., 3 Jul 2026).

Several misconceptions recur in the literature. One is that adding ordinary momentum to an asynchronous method is automatically beneficial. Multiple papers reject this: naive server-side momentum in ASGD can impede convergence, stale updates in AsyncFL create an implicit momentum bias matrix $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$12, and delayed information can be double-counted unless the chronology is preserved (Shi et al., 2024, Yu et al., 2024). Another misconception is that discarding stale gradients is neutral. The data-dependent-delay analysis argues that delay-adaptive attenuation or filtering introduces systematic bias because updates from simpler or faster-to-process samples are overrepresented, while gradients from more complex samples are delayed or suppressed (Dahan et al., 3 May 2026). A third misconception is that Byzantine robustness under full participation transfers directly to partial participation. The partial-participation results show the opposite: methods that aggregate only fresh sampled updates fail immediately after the sampled clients contain a Byzantine majority, motivating stale-momentum retention at the server (Otsuka et al., 3 Sep 2025).

Empirical evidence broadly supports the theoretical picture. OrMo improves final test accuracy by $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$13 points over ASGD and other momentum variants on CIFAR-10/100 with ResNet-20, with 16 or 64 workers and up to $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$14 stragglers, while never requiring tuning $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$15 downward even under large delays (Shi et al., 2024). Momentum Approximation recovers the same optimal $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$16 as in synchronous FL and is compatible with secure aggregation and differential privacy (Yu et al., 2024). D-Byz-SGDM shows that stale momentum can be a robustness resource rather than only an optimization liability (Otsuka et al., 3 Sep 2025).

Outside SGD-like optimization, a related delayed-momentum design appears in eigenvalue computation. DMPower and DMStream perform a pre-momentum warm-up with inexact deflation to estimate $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$17, then activate a heavy-ball phase with $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$18 and iteration $w_{t+1} = w_t - \alpha \nabla f(v_t; z_{i_t}),$19 (Rabbani et al., 2021). This suggests that the core idea of delaying momentum until delayed information can be weighted or estimated correctly has significance beyond distributed learning.

Taken together, the literature presents delayed momentum aggregation as a general response to temporal misalignment in optimization. The main divide is not between “with momentum” and “without momentum,” but between methods that preserve the correct chronological contribution of delayed information and methods that collapse stale and fresh updates into an undifferentiated buffer.