MT-DAO: Distributed Multi-Timescale Optimizers

• MT-DAO is a distributed adaptive optimizer that mitigates time-scale mismatches by maintaining multiple momentum traces with distinct decay factors during local update training.
• It employs separate synchronization schedules for parameters, first momenta, and second moments, effectively closing the performance gap with fully synchronous distributed training.
• Empirical evaluations on language model pre-training show that MT-DAO reduces perplexity and wall-clock time by up to 35%, especially in low-bandwidth, cross-datacenter settings.

Searching arXiv for the MT-DAO paper and closely related distributed/local adaptive optimizer work. arxiv_search(query="^^^^^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^^^^^", max_results=5) Searching arXiv by identifier and title keywords. arxiv_search(query="^^^^^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^^^^^"Multi-Timescale Distributed Adaptive Optimizers with Local Updates\"", max_results=^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^) MT-DAO, short for Multi-Timescale Distributed Adaptive Optimizers with Local Updates, is a family of distributed adaptive optimization methods designed for data-parallel training under infrequent communication. It targets the regime in which workers perform multiple local optimization steps before synchronizing, thereby reducing communication overhead relative to fully synchronous distributed data parallelism (DDP). The central claim is that adaptive optimizers exhibit a time-scale mismatch under local-update training: momentum terms tuned for frequent synchronization decay too quickly across long local intervals, causing noise-dominated optimization. MT-DAO addresses this by maintaining multiple first-moment traces with distinct decay factors, optionally together with a second-moment trace, and by synchronizing these states on different schedules. The paper reports the first convergence guarantees for this setting and empirical results showing that, for language-model pre-training, MT-DAO eliminates the performance gap with DDP, improves perplexity relative to infrequent-communication baselines, reduces iso-token wall-clock time by 6–27% on Ethernet interconnects, and enables effective cross-datacenter training and training over wide geographic areas (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&).

^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^. Problem setting and the time-scale mismatch

MT-DAO is formulated in the standard distributed data-parallel setting with infrequent communication, often referred to as Local SGD. There are PRESERVED_PLACEHOLDER_^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ workers, each maintaining its own copy of the parameter vector PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^. Starting from a common $\theta_t$, each worker $m$ performs $\tau$ local SGD steps on local data before synchronization. With global rounds indexed by $r$, the within-round update is

$$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$

where $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$. After $\tau$ steps, workers synchronize through

$$\theta_{r+1}\;=\;\frac1M\sum_{m=1}^M\theta_{r,\;\tau}^m.$$

The communication advantage is immediate: Local SGD reduces communication by a factor of PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^. The difficulty arises when adaptive optimizers such as Adam are used in this regime. A single-timescale exponential moving average (EMA) momentum,

PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2(Iacob et al., 6 Oct 2025) OR \2^^^^

is tuned for frequent updates. The MT-DAO analysis identifies the resulting failure mode as a mismatch between the optimizer’s internal time scale and the synchronization interval. When PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \22^^^^, the momentum effectively forgets information accumulated before the most recent communication boundary. For typical PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^3 and PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^4, the paper states that PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^5, so the local momentum after PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 steps carries almost no information from the last synchronization. This produces high variance and drift and degrades both convergence and final model quality (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&).

A common misconception is that the degradation of local adaptive methods is explained solely by stale parameters. MT-DAO instead attributes the main gap, in this formulation, to the optimizer state itself: the fast-moving momentum ceases to provide effective smoothing over long local intervals. This suggests that communication-efficient distributed optimization depends not only on how often parameters are averaged, but also on whether optimizer statistics evolve on compatible time scales.

2. Multi-timescale optimizer construction

The MT-DAO family resolves the mismatch by maintaining PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^7 parallel first-momentum traces, each with a different decay factor, and optionally a second EMA of squared gradients. Denoting the first-momenta by PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^8 with decays PRESERVED_PLACEHOLDER_^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^9, worker $\theta_t$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ computes a clipped stochastic gradient $\theta_t$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^ at each local step and updates

$\theta_t$2

Here $\theta_t$3 is the most recent synchronized value of the $\theta_t$4-th momentum. If $\theta_t$5 is not a multiple of its synchronization period $\theta_t$6, then $\theta_t$7. A second-moment trace $\theta_t$8 with decay $\theta_t$9 is maintained and synchronized every $m$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ steps.

The update direction is formed by convexly combining the $m$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^ first-momenta with the current gradient, which functions as a “zero-decay” momentum. With weights $m$2 satisfying $m$3,

$m$4

The preconditioned parameter update is then

$m$5

Parameter synchronization is decoupled from optimizer-state synchronization: $m$6 This decoupling is structurally important. MT-DAO does not merely delay all communication uniformly; it assigns different synchronization periods to parameters, first moments, and second moments. In the paper’s framing, this allows slow-moving statistics to preserve long-horizon information while retaining a fast component for responsiveness to current gradients.

The reported implementation is an Adam-style MT-DAO algorithm in which workers compute clipped gradients, update second moments, update each first momentum, combine them into $m$7, apply the preconditioned step, and synchronize each state according to its own schedule. The authors emphasize the practically important case $m$8, corresponding to a quasi-hyperbolic form in which a slow EMA is combined with the current gradient.

3. Convergence guarantees

Under standard nonconvex assumptions, the paper gives convergence guarantees for an SGDM-style MT-DAO variant and states that these are the first guarantees for the proposed multi-timescale local adaptive setting (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&). The assumptions are:

^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^. each $m$9 is $\tau$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^-smooth;

1. stochastic gradients satisfy $\tau$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^ and $\tau$2;
2. heterogeneity obeys

$\tau$3

For averaged iterates $\tau$4, the theorem uses a stepsize bound

$\tau$5

where

$\tau$6

Under these conditions,

$\tau$7

The stated consequence is the optimal $\tau$8 rate.

The proof sketch proceeds through “virtual iterates” $\tau$9 that linearize the momentum recurrences so that $r$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^. Smoothness of $r$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^ is then used to relate function decrease to $r$2, plus drift terms induced by asynchrony and variance. The analysis controls three deviations: between local worker parameters $r$3 and the average $r$4; between virtual iterates $r$5 and actual iterates $r$6, controlled by $r$7; and gradient heterogeneity, controlled by $r$8. The theorem thereby formalizes the role of both momentum time scales and synchronization schedules in the convergence rate.

A plausible implication is that MT-DAO’s benefit is not reducible to a heuristic choice of large $r$9. In the theorem, the synchronization probabilities $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ and $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^, the decay factors $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$2, and the mixture weights $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$3 appear jointly in the complexity terms, indicating that optimizer design and communication policy are analytically coupled.

4. Empirical evaluation and operating regimes

The empirical study uses GPT-style transformers with ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 M, ^^^^2(Iacob et al., 6 Oct 2025) OR \22^^^^5 M, and 72^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ M parameters, incorporating Peri-LayerNorm, RoPE, and SiLU activations. Training uses the SmolLM2 mixture with sequence length 2^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^48, with ^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^% held out for validation. Experiments run on a cluster of 4× NVIDIA H^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ GPUs connected over 5^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^–^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ Gb/s Ethernet, with additional cross-datacenter scenarios evaluated through a latency/bandwidth model. Baselines are DDP with ADOPT and QHADOPT (quasi-hyperbolic Adam), Local Adam (synchronizing all states every $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$4), and Federated outer-momentum (FedOpt/Nesterov). Batch sizes are shared across methods: 64 K tokens for ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 M, 256 K for ^^^^2(Iacob et al., 6 Oct 2025) OR \22^^^^5 M, and 5^^^^2(Iacob et al., 6 Oct 2025) OR \22^^^^ K for 72^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ M, with training steps chosen per compute-optimal budgets (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&).

The main reported result is that MT-DAO closes the gap to fully synchronous DDP at all scales. In the wall-clock versus perplexity curves, for the 72^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ M model with $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$5, MT-DAO reaches the DDP reference perplexity in 24% fewer tokens and 35% less wall time. In an iso-step comparison at $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$6, Local Adam degrades by up to +5% perplexity, whereas MT-DAO holds within ^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^.^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^% of DDP. The abstract also states that, at the 72^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^M scale, MT-DAO reaches a target perplexity in 24% fewer steps and 35% less time than the single-momentum DDP baseline.

The paper also reports communication-sensitive behavior in cross-datacenter settings. In Figures ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^8–^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^9, when bandwidth drops below ^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ Gb/s, the half-life synchronization variant—which synchronizes slow states infrequently—cuts communication time by up to 8^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^%, halving total runtime. This suggests that the benefit of multi-timescale synchronization increases as network bandwidth becomes more restrictive.

These results are framed specifically around language-model pre-training and perplexity. They do not claim universal superiority across all tasks or all model families. The empirical significance lies in showing that infrequent communication need not entail the usual quality gap relative to synchronous DDP, provided the optimizer state is restructured to match the communication regime.

5. Ablations and mechanistic interpretation

The ablation study focuses on three dimensions: momentum time scales, communication interval, and alignment across workers. For momentum timescales, tuning $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$7 and $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$8 on the ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 M model shows best performance for slow decays $$\theta_{r,\;k+1}^m \;=\;\theta_{r,\;k}^m\;-\;\eta\,g_{r,k}^m, \quad k=0,\dots,\tau-1,$$9 and large weight $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^. For communication interval, Table 2 shows that larger $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^ reduces performance degradation as $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$2 increases from ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 to ^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates22^^^^4 steps, with an example of +6.2% vs +3.7% perplexity. These observations support the paper’s central thesis that long-half-life momentum is the relevant corrective mechanism.

The paper further examines the rate of parameter change and cross-worker alignment. It reports that slow momentum reduces

$g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$3

as shown in Figure 1 (left), thereby minimizing drift. It also reports that MT-DAO raises the cross-worker cosine similarity of “pseudo-gradients” $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$4 and momenta to $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$5, compared with $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$6 for Local Adam, as shown in Figure 2. The intended interpretation is that slow momentum maintains alignment among workers even when local trajectories diverge between synchronization points (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&).

A misconception that these ablations help dispel is that any increase in momentum persistence should suffice. The reported optima are not arbitrary large-$g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$7 settings; they emerge in conjunction with large $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$8, specific synchronization intervals, and a mixed fast/slow update rule. This suggests that MT-DAO’s effect depends on combining long-term memory with a residual fast component rather than replacing one with the other.

6. Practical recommendations, limits, and extensions

The paper gives explicit practical recommendations. For the slow first-momentum decay, it recommends $g_{r,k}^m=\nabla F_m(\theta_{r,k}^m;\xi_{r,k}^m)$9. Higher $\tau$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ corresponds to a longer half-life, allowing first-momentum synchronization every

$\tau$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^

steps, with the example $\tau$2 for $\tau$3. For the mixture coefficient, it recommends $\tau$4 so that slow momentum dominates while preserving some responsiveness to the current gradient. For parameter synchronization, the recommendation is $\tau$5–64 on 5^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^–^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ Gb/s Ethernet, and $\tau$6–5^^^^2(Iacob et al., 6 Oct 2025) OR \22^^^^ in cross-datacenter (^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^–^^^^2(Iacob et al., 6 Oct 2025) OR \2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ Gb/s) settings. Learning-rate tuning is recommended on a ^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^6 M “proxy” with CompleteP transfer; the paper notes typical $\tau$7 for ADOPT, then scaling by a quasi-hyperbolic multiplier (^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^.5–2×) when using slow momentum. The synchronization schedule recommendation is to use half-life-based sync for momentum, $\tau$8, and $\tau$9 for parameters, with the claim that this decoupling maximizes communication savings.

The stated limitations are equally explicit. Experiments run only up to 72^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ M parameters, so behavior at multi-billion scale remains to be fully validated. The study focuses on $$\theta_{r+1}\;=\;\frac1M\sum_{m=1}^M\theta_{r,\;\tau}^m.$$^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^ for ease of tuning and minimal memory overhead; adding more momenta, $$\theta_{r+1}\;=\;\frac1M\sum_{m=1}^M\theta_{r,\;\tau}^m.$$^^^^2(Iacob et al., 6 Oct 2025) OR \2^^^^, can yield further but diminishing returns. The paper lists several future directions: dynamic adaptation of $$\theta_{r+1}\;=\;\frac1M\sum_{m=1}^M\theta_{r,\;\tau}^m.$$2 or $$\theta_{r+1}\;=\;\frac1M\sum_{m=1}^M\theta_{r,\;\tau}^m.$$3 during training, integration with advanced compression primitives such as quantization and sparsification, and application to heterogeneous or fault-tolerant federated settings (&&&^^^^2MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates2^^^^&&&).

These limitations matter for interpretation. MT-DAO is not presented as a universal replacement for synchronous training or as a completed solution for all distributed adaptive optimization regimes. Rather, the evidence supports a more specific conclusion: under infrequent communication, preserving long-term memory in optimizer state can close the performance gap that otherwise appears when standard single-timescale adaptive methods are deployed unchanged.