Federated Stochastic Gradient Descent

• Federated Stochastic Gradient Descent (FedSGD) is a distributed optimization algorithm that aggregates single-step stochastic gradients from multiple clients to update a shared global model.
• It emphasizes frequent communication and synchronous updates, managing gradient staleness and statistical heterogeneity inherent in non-IID data settings.
• Practical implementations require careful tuning of client participation, learning rates, and communication strategies to balance convergence speed with resource constraints.

Federated Stochastic Gradient Descent (FedSGD) is a foundational algorithm in federated learning frameworks, designed to solve distributed optimization problems in settings where data resides on multiple clients and direct data sharing is precluded by privacy, regulatory, or practical communication constraints. FedSGD operates by orchestrating synchronous computation of stochastic gradients on privileged local data subsets and then aggregating these gradients on a central server to update a shared global model. The method is characterized by its extremal communication regime: each selected client performs exactly one stochastic gradient update per communication round, resulting in frequent model synchronization and minimal local computation. While conceptually simple, the interplay between gradient staleness, partial participation, statistical heterogeneity, and communication constraints introduces nuanced statistical and system-level considerations that differentiate FedSGD from classical SGD, as well as from alternative federated algorithms such as FedAvg and FedProx.

1. Canonical FedSGD Algorithm and Variants

The core FedSGD protocol encompasses the following stages per communication round (Konečný, 2017, &&&1&&&, Le et al., 2024):

1. Broadcast: The server distributes the current model $w^t \in \mathbb{R}^d$ to a selected subset $S_t \subset \{1,\ldots,K\}$ of $N$ clients.
2. Local Stochastic Gradient Computation: Each client $k \in S_t$ computes

$$g_k^t = \frac{1}{H} \sum_{s=0}^{H-1} \nabla \ell(w^t; x_{k,s}, y_{k,s}),$$

using $H$ local samples, and returns $g_k^t$ to the server.

1. Aggregation: The server aggregates received gradients, reuses stale gradients $g_j^{t-1}$ for $j \notin S_t$, and computes the weighted sum

$$g^t = \sum_{k=1}^K p_k g_k^t, \quad p_k = \frac{n_k}{n}.$$

2. Global Update: The model is updated via

$w^{t+1} = w^t - \eta g^t,$

where $\eta > 0$ is the learning rate.

Variants incorporate full-batch versus mini-batch gradients, synchronous versus asynchronous aggregation, and diverse sampling or communication topologies including graph-based message passing (Balik, 2024).

2. Theoretical Analysis and Emergent Dynamics

A critical property of FedSGD in the federated context is the emergence of "self-induced momentum" arising from partial participation and stale gradient reuse. Specifically, if only $N < K$ clients contribute fresh gradients per round, the update satisfies (Yang et al., 2022):

$$\mathbb{E}[w^{t+1} - w^t] = \beta \mathbb{E}[w^t - w^{t-1}] - (1-\beta)\eta \mathbb{E}[\nabla f(w^t)],$$

where $\beta = 1 - N/K$. This recapitulates the form of a momentum SGD update with implicit coefficient $\beta$, even when no explicit momentum is added. The expected convergence rate to stationary points for nonconvex $L$-smooth loss, with Gaussian or bounded-variance stochastic gradients, is $O(1/\sqrt{T})$, where the prefactor deteriorates as $\beta \uparrow 1$ (i.e., as participation per round decreases) (Yang et al., 2022). Thus, there exists a trade-off between communication efficiency and convergence speed mediated by the client participation rate.

The classical convergence guarantees for FedSGD under convexity and strong convexity mirror those of centralized SGD, with linear convergence achievable under strong convexity and a diminishing learning rate for the non-strongly convex case (Konečný, 2017, V et al., 2024).

3. Extensions in Heterogeneous and Personalized Settings

In the presence of statistical heterogeneity (non-IID data) or when personalization is required, standard FedSGD suffers from client-drift and instability. Several approaches have adapted the FedSGD paradigm:

• Personalized Federated SGD: PFLEGO (Nikoloutsopoulos et al., 2022) augments the model with client-specific "heads", enabling each client to update local parameters and return exact, unbiased gradients for the global (shared) parameters. This decoupling facilitates unbiased SGD steps and shows empirically improved performance in high-personalization regimes.
• Depersonalized FedSGD: Alternating or "depersonalized" SGD techniques (Zhou et al., 2022) iteratively decouple personalized and shared objectives, introducing penalization and mixing-rate hyperparameters in local updates to mitigate drift and reduce variance. These approaches achieve sublinear convergence rates with significantly reduced dependency on data heterogeneity.
• Age-Weighted FedSGD: To counteract sampling bias from underparticipated or "stale" clients, age-of-information weighting schemes scale the contributions of each device in proportion to their staleness, thereby improving convergence and mitigating weight divergence (Wang et al., 2024).

4. Communication, Privacy, and Systems-Level Trade-offs

FedSGD is characterized by high communication intensity: each participating client communicates gradients every round. Communication cost per round is $O(d|S_t|)$ where $d$ is model dimensionality, and scales linearly with the number of communication rounds required for global optimality. This is often prohibitive under realistic constraints. Strategies to mitigate these costs include:

• Partial Participation and Staleness Control: Modulating $N$ enables balancing between communication overhead and convergence; increasing participation reduces staleness, improving convergence at the expense of increased traffic (Yang et al., 2022).
• Energy and Resource Constraints: In wireless and resource-constrained networks, joint optimization of device selection, resource allocation, and sub-channel assignment (using, for example, KKT-based solutions and matching algorithms) enables efficient device participation and improved trade-offs across latency, energy, and learning speed (Wang et al., 2024).
• Graph-Based Communication: Deployments with empirical graph structures (e.g., hospitals connected by patient similarity) utilize FedSGD-style peer-to-peer updates rather than star-topology aggregations, achieving improved scalability and localized communication (Balik, 2024).

From a privacy perspective, FedSGD only shares model updates (gradients) between clients and aggregators or neighbors; raw data remains local at all times, offering inherent regulatory compliance for sensitive domains (Balik, 2024).

5. Comparative Evaluation and Hybrid Algorithms

FedSGD serves as a baseline for federated optimization. However, alternative methods such as FedAvg (multiple local steps per aggregation), FedProx (proximal regularization), and variance-reduced methods typically achieve superior communication efficiency for non-IID data at the cost of local computation or more complex coordination (Konečný, 2017, V et al., 2024, Le et al., 2024).

DynamicFL (Le et al., 2024) provides a flexible framework for communication resource allocation that interpolates between FedSGD (high-frequency, low-local-computation) and FedAvg (low-frequency, high-local-computation). By selectively allocating higher communication frequencies to statistically critical clients under system-wide resource constraints, DynamicFL bridges the statistical and heterogeneity gaps, improving model performance by 10–30 absolute points over uniform selection at a fraction (<60%) of FedSGD's communication cost.

Comparison Table: Communication and Convergence Properties

Algorithm	Local Steps/Round	Communication Cost	Convergence Rate
FedSGD	1	High	$O(1/\sqrt{T})$
FedAvg	$E>1$	Lower	$O(1/(\sqrt{mET}))$
FedProx	$E>1$, Proximal	Similar to FedAvg	Improved on non-IID
PFLEGO	1 (shared), multi (personalized)	Lower (due to gradient transmission)	SGD-like, improved in heterogeneity
DynamicFL	Mixed	Tunable	FedSGD–FedAvg bridge

6. Practical Guidelines, Open Questions, and Limitations

The practical deployment of FedSGD requires careful tuning of:

• Learning rate schedules (constant or diminishing as $1/\sqrt{T}$),
• Participation fraction $N/K$,
• Local batch size $H$ to trade off between variance and cost.

Staleness must be controlled, for example by deadline-driven client selection. Where communication is a bottleneck, increasing local batch size or incorporating explicit momentum can compensate for the side effects of sparse or infrequent participation (Yang et al., 2022). For highly heterogeneous or personalized settings, splitting parameter updates or weighting by staleness improves stability and convergence (Nikoloutsopoulos et al., 2022, Wang et al., 2024).

Limitations of FedSGD include sensitivity to statistical heterogeneity (bias from non-IID data), high communication requirements, and relatively slow convergence when contrasted with multi-step local solvers. FedSGD's benefits emerge primarily in scenarios with abundant communication bandwidth, highly unbalanced local data, or when strict synchronization and minimal local compute are operational requirements (Konečný, 2017).

7. Empirical Results and Impact in Applications

Empirical studies have shown that FedSGD delivers competitive or superior performance in highly decentralized or privacy-critical settings:

• Superior MSE compared to FedAvg in graph-based hospital networks for predicting hospital length of stay, exploiting local regularization and scalable, peer-to-peer communication (Balik, 2024).
• Robustness and stable loss minimization under non-IID splits demonstrated by SA-FedSGD as compared to FedAvg and FedProx (V et al., 2024).
• Substantial improvements in personalized, heterogeneous regimes using modified FedSGD variants (PFLEGO, depersonalized SGD, AoI-weighted) in image and text classification with gains up to 10 percentage points in test accuracy (Nikoloutsopoulos et al., 2022, Zhou et al., 2022, Wang et al., 2024).
• In DynamicFL, communication-aware variants attain near-FedSGD accuracy at half the per-round communication cost on large-scale heterogeneity benchmarks (Le et al., 2024).

These empirical validations, combined with rigorously characterized convergence properties, establish FedSGD as a critical reference point in federated optimization research, as well as a practical tool for distributed learning under stringent privacy and system constraints.