Adaptive Federated SGD: Advances and Methods

• Adaptive federated SGD is a framework that enhances classic federated averaging by integrating adaptive momentum, per-coordinate preconditioning, and online stepsize adaptations to tackle data heterogeneity and client drift.
• The methodology leverages techniques like directional and per-coordinate adaptivity, error-based learning rate adjustment, and adaptive participation scheduling to improve convergence and stability.
• Empirical results demonstrate that adaptive variants achieve higher accuracy and communication efficiency compared to vanilla FedAvg, especially in non-IID data environments and sparse participation scenarios.

Adaptive federated stochastic gradient descent (SGD) denotes federated optimization schemes that preserve the client–server structure of Federated SGD or FedAvg—broadcast of a global model, local stochastic updates, and periodic aggregation—while adapting one or more components of the descent process to data heterogeneity, staleness, curvature, communication constraints, or personalization. A standard formulation is

$$\min_{\boldsymbol{\theta}\in\mathbb{R}^{d}} f(\boldsymbol{\theta}) = \frac{1}{N}\sum_{i=1}^{N} f_i(\boldsymbol{\theta}), \qquad f_i(\boldsymbol{\theta}) = \mathbb{E}_{\zeta \sim \mathcal{D}_i} F(\boldsymbol{\theta},\zeta),$$

or, equivalently, $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$ under client weights $p_k$. Within this template, adaptive variants include momentum injected into local and server updates, per-coordinate preconditioning, extrapolation-based learning-rate control, adaptive participation schedules, sparse transmission of adaptive states, and exact stochastic-gradient constructions for personalized models (Ozfatura et al., 2020, Yang et al., 2022, Koskela et al., 2018, Nikoloutsopoulos et al., 2022).

1. Problem formulation and meanings of adaptivity

In its basic form, federated SGD proceeds in communication rounds. The server broadcasts a global model, a subset of clients performs $H$ local SGD steps on private data, and the server aggregates the resulting client states or updates. FedAvg is the canonical instance: client $i$ initializes $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$, performs local steps

$$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$

and the server averages the returned models to obtain $\boldsymbol{\theta}_{t+1}$ (Ozfatura et al., 2020).

The literature uses “adaptive” in several technically distinct senses. One sense is directional adaptivity, where a global momentum vector or aggregated-gradient surrogate changes the local search direction as training history evolves. Another is per-coordinate adaptivity, where an AdaGrad-, AMSGrad-, Adam-, or Yogi-like preconditioner rescales coordinates according to accumulated gradient statistics. A third is online scalar stepsize adaptation, where each client updates a local learning rate from an error estimator rather than from a fixed schedule. A fourth is system-level adaptivity, where the number of participating workers or the effective synchronization pattern is varied over time to optimize an error–runtime or error–communication trade-off. These meanings are not interchangeable: FedADC explicitly states that its adaptivity is not Adam-like per-coordinate learning rates, whereas Local AMSGrad is precisely a per-coordinate adaptive method, and ADADP adapts a single scalar learning rate per client (Ozfatura et al., 2020, Chen et al., 2021, Koskela et al., 2018).

A further complication is that standard federated implementations may already exhibit implicit adaptive behavior. In the cached-gradient FedSGD model analyzed by “Federated Stochastic Gradient Descent Begets Self-Induced Momentum,” stale client gradients induce an expectation-level momentum term whose coefficient depends on the participation ratio, even when no explicit momentum buffer is implemented (Yang et al., 2022). This result makes “plain” federated SGD less primitive than its update equations alone suggest.

2. Momentum, acceleration, and drift-controlled federated SGD

A central family of adaptive federated SGD methods augments the FedAvg outer loop with momentum or acceleration, and in some cases reuses the same state to mitigate client drift. FedADC is the most explicit realization of this idea. It keeps a single global momentum vector $\mathbf{m}_t$, interprets averaged client differences as pseudo-gradients, and updates the server state via

$$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$

Its distinctive step is to inject a normalized version of this momentum into each local step,

$f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$0

or, in the paper’s Nesterov-like variant, to evaluate the gradient after a momentum look-ahead step. The same vector $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$1 therefore acts both as a server-side acceleration direction and as a client-side drift-control term, pulling local trajectories toward a shared direction under non-IID data (Ozfatura et al., 2020).

The drift problem addressed by FedADC is the deviation of local iterates from the broadcast global model after multiple local steps on heterogeneous data. In FedAvg, client drift is entirely driven by the sum of client-specific local gradients. In FedADC, the total model change becomes

$f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$2

so every client receives an identical global component in addition to its local gradients. This makes the method an accelerated and drift-aware variant of federated SGD without adding per-client control variates or changing the basic communication pattern (Ozfatura et al., 2020).

A complementary line studies acceleration from first principles. FedAc replaces local SGD with a generalized accelerated SGD involving three sequences $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$3 and periodic averaging of both the main and accelerated iterates. Its key stability parameter is

$f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$4

where $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$5 is the synchronization interval. This choice implements a strategic tradeoff between acceleration and stability: standard Nesterov-style acceleration is shown to be unstable under infrequent synchronization, whereas FedAc tunes the acceleration strength to the communication regime. For strongly convex and smooth objectives, FedAc improves the communication requirement for linear speedup from $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$6 synchronization rounds for the previous state of the art to $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$7 rounds, and under third-order smoothness it yields stronger guarantees still (Yuan et al., 2020).

Momentum can also arise without being programmed explicitly. In the cached-gradient FedSGD model with $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$8 total clients and $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$9 selected per round, the staleness parameter is $p_k$0, and the expected parameter recursion becomes

$p_k$1

This matches classical momentum SGD in expectation, with effective momentum coefficient $p_k$2 and effective gradient step $p_k$3. In that sense, partial participation and stale cached gradients already induce an adaptive memory term tied directly to systems parameters rather than to an optimizer hyperparameter (Yang et al., 2022).

3. Learning-rate and preconditioner adaptation

Per-coordinate adaptive federated SGD is exemplified by Local AMSGrad. Each client maintains local first- and second-moment states $p_k$4 and $p_k$5, but the AMSGrad denominator is synchronized globally: $p_k$6 with $p_k$7 updated only at communication rounds. The paper’s central claim is negative as well as positive: the naive alternative, in which each client maintains its own $p_k$8 and only parameters are averaged, can diverge even on a simple one-dimensional problem because the same coordinate is updated with inconsistent effective learning rates across clients. By contrast, shared adaptive denominators preserve consensus and yield a nonconvex convergence rate of $p_k$9 when the communication period satisfies $H$0 (Chen et al., 2021).

A different notion of adaptivity is used in the extrapolation-based method of “Learning Rate Adaptation for Federated and Differentially Private Learning.” Here the optimizer remains SGD, but each client updates a scalar learning rate online by comparing one full step of size $H$1 with two half-steps of size $H$2. With error function

$H$3

the stepsize controller is

$H$4

In the federated specialization, all clients participate in each round and run this adaptive SGD locally inside FedAvg. The method uses no validation set, which is important in both federated and differentially private learning, and the federated experiments report robustness under highly non-IID client partitions where fixed-step SGD and Adam degrade (Koskela et al., 2018).

FedAgg derives adaptive local learning rates from a discrete-time linear–quadratic control formulation. It introduces mean-field terms

$H$5

and penalizes both deviation from $H$6 and large learning rates via

$H$7

The adaptive stepsize is obtained from the Hamiltonian optimality condition,

$H$8

with a backward recursion for the costate $H$9. The resulting $i$0 is solvable by backward induction and is bounded in $i$1 with $i$2; the paper further proves bounded client drift and convergence bounds in both convex and non-convex settings (Yuan et al., 2023).

4. Consistency, bias, and exactness

A major theoretical issue is that local adaptivity can improve the rate at which federated training converges while changing the point to which it converges. “Local Adaptivity in Federated Learning: Convergence and Consistency” formalizes this via a client-optimizer operator $i$3 and shows that federated training with restarted client states converges to the unique fixed point $i$4 of $i$5, not necessarily to the minimizer $i$6 of the global objective. The paper proves convergence to $i$7 under contractive local operators, but then demonstrates a non-vanishing solution bias: even as learning rates go to zero, $i$8 in general when local preconditioners, local learning rates, or local step counts differ across clients. In its quadratic example, the limiting fixed point depends on client-specific matrices $i$9, making heterogeneous adaptive preconditioning itself a source of inconsistency (Wang et al., 2021).

The same work proposes two correction mechanisms. Local correction normalizes the client model change by an accumulated preconditioning matrix $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$0, sending $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$1 rather than the raw update; global correction rescales the aggregated normalized update by an average inverse correction matrix at the server. The local correction is designed so that the leading skew term at $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$2 cancels, leaving only higher-order error, and the paper proves a non-convex convergence guarantee in terms of the true global gradient norm rather than the fixed point of a biased operator (Wang et al., 2021).

A different route to exactness is taken in personalized federated learning. “Personalized Federated Learning with Exact Stochastic Gradient Descent” considers a shared backbone $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$3 and client-specific heads $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$4, with objective

$$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$5

Selected clients first update only $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$6 locally, then compute the joint gradient of $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$7 with respect to $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$8 and $$\boldsymbol{\theta}^{0}_{i,t}=\boldsymbol{\theta}_t$$9, keep the personalized update locally, and return only the shared-gradient component to the server. The server update

$$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$0

is an unbiased SGD step on the full personalized objective, and the paper proves $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$1. This construction shows that exact stochastic-gradient aggregation remains possible even when personalization breaks the single-global-model assumption (Nikoloutsopoulos et al., 2022).

The contrast between these two strands is instructive. Local adaptive preconditioning can distort the effective global operator unless corrected, whereas explicit gradient-based decomposition of shared and personalized parameters can preserve unbiasedness exactly. A plausible implication is that future adaptive federated SGD designs can benefit from combining exact gradient constructions with adaptive server or client optimizers, rather than relying only on parameter averaging.

5. Communication-efficient adaptive optimization

Communication efficiency is a defining constraint of federated SGD, and adaptive methods introduce additional state that can aggravate it. Local AMSGrad addresses communication frequency by using periodic averaging: with total iterations $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$2 and communication period $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$3, the number of communication rounds is roughly $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$4, and the theory allows $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$5 up to $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$6 while retaining the $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$7 nonconvex rate. In this sense, adaptive per-coordinate preconditioning can be made communication-efficient without abandoning the local-SGD template (Chen et al., 2021).

FedAdam-style methods create a sharper payload problem because the server must aggregate not only model updates but also first and second moments. “Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization” states that standard FedAdam incurs a threefold increase in uplink communication overhead relative to FedSGD because each client transmits the model, first moment, and second moment. FedAdam-SSM sparsifies the updates

$$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$8

with a shared sparse mask, reducing the per-client uplink asymptotically from $$\boldsymbol{\theta}^{\tau}_{i,t} = \boldsymbol{\theta}^{\tau-1}_{i,t}-\eta_t \mathbf{g}^{\tau}_{i,t},$$9 to $\boldsymbol{\theta}_{t+1}$0. The mask is optimized by minimizing an upper bound on the divergence from centralized Adam; under the paper’s weighting comparison $\boldsymbol{\theta}_{t+1}$1 and the empirical magnitude ordering $\boldsymbol{\theta}_{t+1}$2, the optimal shared sparse mask is the top-$\boldsymbol{\theta}_{t+1}$3 mask of the model update $\boldsymbol{\theta}_{t+1}$4. The method comes with convergence bounds in both non-convex and Polyak–Łojasiewicz settings, and the abstract reports convergence over $\boldsymbol{\theta}_{t+1}$5 faster than the sparse FedAdam baselines and test accuracy over $\boldsymbol{\theta}_{t+1}$6 ahead of the quantized FedAdam baselines (Deng et al., 2024).

Communication efficiency can also be pursued by adapting participation rather than optimizer state. In “Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning,” the master waits for the fastest $\boldsymbol{\theta}_{t+1}$7 workers at each iteration and adapts $\boldsymbol{\theta}_{t+1}$8 over time. The fixed-$\boldsymbol{\theta}_{t+1}$9 error bound exhibits the classic trade-off: small $\mathbf{m}_t$0 shortens iteration time $\mathbf{m}_t$1 but raises the stochastic floor proportional to $\mathbf{m}_t$2, while large $\mathbf{m}_t$3 slows rounds but lowers variance. The paper derives a bound-optimal increasing schedule for $\mathbf{m}_t$4 and a practical heuristic based on sign changes in consecutive gradient inner products. Interpreted federatively, this is an adaptive client-participation scheduler: start with a small target number of responses, then increase participation when the transient phase is exhausted (Hanna et al., 2022).

These communication-oriented methods broaden the meaning of adaptive federated SGD. Adaptivity may target step sizes or momentum, but it may also target what is transmitted, how much optimizer state is shared, or how many client updates are waited for in each round.

6. Empirical landscape, misconceptions, and limitations

Across the cited literature, adaptive federated SGD consistently shows its strongest advantages under non-IID data, sparse participation, or constrained communication. FedADC reports that on CIFAR-10 with $\mathbf{m}_t$5, $\mathbf{m}_t$6, 100 clients, 500 rounds, and 2 local epochs, FedAvg reaches $\mathbf{m}_t$7, FedDyn $\mathbf{m}_t$8, FedADC $\mathbf{m}_t$9, and FedADC+ $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$0; on CIFAR-100 with $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$1, $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$2, the same table gives FedAvg $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$3, FedDyn $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$4, FedADC $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$5, and FedADC+ $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$6 (Ozfatura et al., 2020). FedAgg reports the same general pattern across MNIST, EMNIST-L, CIFAR-10, and CIFAR-100, including a marked gain on MNIST-CNN under Non-IID 20% participation, where FedAvg attains $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$7 and FedAgg $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$8 (Yuan et al., 2023). For communication-limited adaptive moments, FedAdam-SSM reaches a target $$\mathbf{m}_{t+1} = \bar{\boldsymbol{\Delta}}_t + (\beta_{\text{global}}-\beta_{\text{local}})\mathbf{m}_t, \qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \eta_t \mathbf{m}_{t+1}.$$9 test accuracy on Fashion-MNIST IID with 8.15 Mbit, compared with 11.3 Mbit for FedAdam-Top, 17.1 Mbit for Fairness-Top, and 24.0 Mbit for full FedAdam (Deng et al., 2024).

Several recurring misconceptions are explicitly contradicted by the literature. The first is that “adaptive federated SGD” is synonymous with Adam-like coordinate-wise learning rates. FedADC uses adaptivity in the form of global momentum and drift control rather than per-coordinate scaling, and the staleness analysis of cached-gradient FedSGD shows that implicit momentum may already be present even without explicit adaptive states (Ozfatura et al., 2020, Yang et al., 2022). The second is that any local adaptive optimizer should improve federated learning. Local AMSGrad shows that unsynchronized adaptive denominators can diverge, and the consistency analysis of local adaptivity proves that heterogeneous local preconditioning can create a non-vanishing solution bias even when the training iterates converge rapidly (Chen et al., 2021, Wang et al., 2021). The third is that more aggressive acceleration or more local work is automatically beneficial. FedAc proves that naive acceleration can be unstable under infrequent synchronization, and both FedADC and FedAdam-SSM report a trade-off between larger local epochs and increased client drift or sparsification bias (Yuan et al., 2020, Deng et al., 2024).

The current theory remains fragmented across assumptions and regimes. FedAc assumes homogeneous workers with i.i.d. samples from the same distribution, Local AMSGrad assumes synchronous participation of all $f(\mathbf{w})=\sum_{k=1}^{K} p_k f_k(\mathbf{w})$00 nodes in its formal analysis, and the self-induced momentum analysis assumes a geometric staleness law induced by uniform random client selection (Yuan et al., 2020, Chen et al., 2021, Yang et al., 2022). ADADP does not provide formal stochastic or federated convergence theorems, focusing instead on privacy accounting and empirical robustness; FedAgg requires estimation of mean-field terms through an auxiliary fixed-point procedure; and several methods note that partial participation, asynchronous communication, or realistic dropout would require additional theory (Koskela et al., 2018, Yuan et al., 2023, Chen et al., 2021).

Taken together, these results define adaptive federated SGD not as a single algorithmic template but as a research area centered on modifying local or server SGD so that the descent process responds to federated pathologies rather than ignoring them. The field’s core technical questions concern which statistics should be shared, which should remain local, how adaptive states interact with heterogeneity and synchronization, and when an apparent improvement in rate preserves the correct optimization target rather than merely converging faster to a biased fixed point.