SCAFFOLD: Controlled Averaging in FL

• Stochastic Controlled Averaging (SCAFFOLD) is a variance reduction method that uses control variates to correct client drift in distributed stochastic optimization.
• It enhances convergence rates and communication efficiency in federated learning, particularly under data heterogeneity and partial client participation.
• The algorithm achieves strong theoretical guarantees, including robustness to compression and improved performance over FedAvg and LocalSGD.

Stochastic Controlled Averaging (SCAFFOLD) is a variance reduction method for distributed stochastic optimization, with a primary application in federated learning (FL) regimes characterized by data heterogeneity and partial client participation. SCAFFOLD addresses the “client-drift” phenomenon by equipping each client and the server with control variates that correct the drift induced by non-IID local objectives. The algorithm achieves provable improvements in communication efficiency and convergence rates over classical schemes such as Federated Averaging (FedAvg) and LocalSGD, under notably weaker assumptions. Recent developments also demonstrate SCAFFOLD’s compatibility with communication compression and its rigorous convergence under general stochastic regimes.

1. Problem Formulation and Background

The foundational optimization problem is the distributed nonconvex finite-sum objective:

$$\min_{x\in\mathbb{R}^d}\ f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x),$$

where $f_i:\mathbb{R}^d \to \mathbb{R}$ denotes the private objective of client (or worker) $i$ and where $f_i$ need not equal $f_j$. In the federated setting, each client can query unbiased stochastic gradients $G_i(x,\xi^i)$, satisfying $\mathbb{E}_\xi[G_i(x,\xi)] = \nabla f_i(x)$ and finite variance.

Classical local-update protocols such as FedAvg perform several stochastic steps locally and aggregate parameters across clients periodically. For non-IID data, each local model drifts towards its local minimizer, resulting in biased global updates—termed “client drift”—that renders convergence slow or unstable (Karimireddy et al., 2019).

2. SCAFFOLD Algorithm: Structure and Control Variates

SCAFFOLD enhances the robustness of federated optimization to heterogeneity by employing “control variates,” per-client and global, to correct the expected drift in local gradients.

Core Protocol

Let $x^t$ be the global model at round $t$, $c^t$ the server-side control variate ($f_i:\mathbb{R}^d \to \mathbb{R}$0), and $f_i:\mathbb{R}^d \to \mathbb{R}$1 the client-side control variate ($f_i:\mathbb{R}^d \to \mathbb{R}$2). In each communication round:

• Each participating client $f_i:\mathbb{R}^d \to \mathbb{R}$3 performs $f_i:\mathbb{R}^d \to \mathbb{R}$4 local stochastic steps:

$f_i:\mathbb{R}^d \to \mathbb{R}$5

• After $f_i:\mathbb{R}^d \to \mathbb{R}$6 steps, the client updates its control variate:

$f_i:\mathbb{R}^d \to \mathbb{R}$7

• The server aggregates the updates:

$f_i:\mathbb{R}^d \to \mathbb{R}$8

$f_i:\mathbb{R}^d \to \mathbb{R}$9

The client’s local step thus becomes an (approximately) unbiased estimator for $i$0, synchronized across the population via the control variates (Karimireddy et al., 2019, Luo et al., 8 Jan 2025).

Algorithmic Simplifications

An operationally efficient variant reduces uplink communication by using a single “increment” $i$1 per round. Communication-efficient SCAFFOLD relies on this increment and is amenable to both unbiased and biased message compression (Huang et al., 2023).

3. Theoretical Foundations: Assumptions and Convergence Guarantees

SCAFFOLD’s theoretical analysis in recent work establishes rigorous performance bounds under general and practical assumptions:

Key Assumptions

• Smoothness: Each $i$2 is $i$3-smooth.
• Gradient Similarity: $i$4.
• Hessian Similarity: $i$5.
• Variance Boundedness: For all $i$6, $i$7.
• Weak Convexity or Strong Convexity: As applicable to the objective class (Luo et al., 8 Jan 2025, Karimireddy et al., 2019).

Main Results

Non-Convex Setting

Under Hessian similarity and weak convexity, for stepsize $i$8 appropriately chosen,

$i$9

(Luo et al., 8 Jan 2025). The terms correspond to communication contraction, weak convexity correction, variance, and residual client drift.

Convex & Strongly Convex Cases

In the strongly convex regime, with full participation and under smoothness,

$f_i$0

communication rounds suffice; this matches large-batch SGD, independent of heterogeneity (Karimireddy et al., 2019).

General Variance Condition

Under the relaxed “ABC” condition, the complexity in the nonconvex smooth case is

$f_i$1

with $f_i$2 quantifying the heterogeneity-induced gap (Huang et al., 2023).

Stochastic Regime: Markov Chain Analysis

Modeling the SCAFFOLD iterates as a Markov chain reveals geometric ergodicity in Wasserstein-2 distance and yields:

$f_i$3

Linear speedup in $f_i$4 is achievable up to higher-order bias terms (Mangold et al., 10 Mar 2025).

4. Comparative Analysis: SCAFFOLD vs. FedAvg and LocalSGD

A central result is the ability of SCAFFOLD to remove the explicit dependence on heterogeneity that afflicts FedAvg and standard LocalSGD.

Method	Heterogeneity Complexity Term	Leading Term with Full Participation
FedAvg	$f_i$5	Slower when $f_i$6
LocalSGD	Similar structure, extra “drift” term	Matches MbSGD only for low drift
SCAFFOLD	No leading heterogeneity-induced penalty	$f_i$7

SCAFFOLD attains its rate without requiring strong uniformity or bounded similarity. In special cases (e.g., quadratic objectives with matched Hessians), SCAFFOLD achieves additional “interpolation” acceleration (Karimireddy et al., 2019, Luo et al., 8 Jan 2025).

5. Extensions: Communication Compression and Partial Participation

SCAFFOLD’s structure is conducive to communication-efficient federated learning:

• SCALLION: Supports unbiased message compression. By communicating compressed increments, the communication cost is halved, and convergence matches full-precision SCAFFOLD asymptotically under standard unbiased compressors.
• SCAFCOM: Admits biased (contractive) compression, using momentum to neutralize systematic bias. Complexity results maintain robustness to heterogeneity and partial participation (Huang et al., 2023).

SCAFFOLD’s compressed variants preserve performance even under drastic uplink reduction (e.g., Top-$f_i$8 sparsification, low-bit random dithering).

6. Proof Techniques and Analytical Innovations

Rigorous analysis of SCAFFOLD relies on several advanced methods:

• Refined Consensus Error Bounds: Tight decomposition of client–server gradient discrepancies using Hessian similarity and Lipschitz continuity, yielding sharper contraction rates for client drift (Luo et al., 8 Jan 2025).
• Variance–Drift Decoupling: Use of a “noiseless sequence” to remove coupling between stochastic gradient noise and drift recursion.
• Markov Chain Contraction: Wasserstein-metric coupling arguments underpin geometric convergence under stochastic gradients (Mangold et al., 10 Mar 2025).
• ABC Condition: General variance frameworks clarify the effect of heterogeneity, with direct control over the heterogeneity gap $f_i$9 (Huang et al., 2023).

7. Practical Considerations and Tuning Guidelines

• Stepsizes: Step-size parameters should satisfy $f_j$0 and scale with $f_j$1, $f_j$2, $f_j$3, and $f_j$4 as dictated in (Luo et al., 8 Jan 2025).
• Communication Interval ($f_j$5, $f_j$6, $f_j$7, $f_j$8): Longer intervals reduce communication but increase drift error ($f_j$9); SCAFFOLD endures larger intervals with mild heterogeneity.
• Control Variate Storage: Only the averaged gradients or increments need to be stored, so memory cost is negligible.
• Heterogeneity Estimation: The heterogeneity gap $G_i(x,\xi^i)$0 is operationally observable and diagnostic for variance-reduction needs (Huang et al., 2023).
• Compression Regimes: Compression schemes (SCALLION, SCAFCOM) are optimal under unbiased or contractive compressor designs, maintaining the $G_i(x,\xi^i)$1 gradient complexity when matched to problem parameters (Huang et al., 2023).

SCAFFOLD, alongside its communication-efficient variants, currently represents the most theoretically robust approach to heterogeneity- and communication-resilient federated optimization under standard smoothness and stochastic assumptions. Its regime of authentication-free superiority over both FedAvg and LocalSGD is precisely characterized in terms of Hessian similarity, weak convexity, and variance parameters, with rigorous nonasymptotic rates and stability assurances (Luo et al., 8 Jan 2025, Huang et al., 2023, Karimireddy et al., 2019, Mangold et al., 10 Mar 2025, Huang et al., 2023).