---
title: 'SCAFFOLD: Controlled Averaging in FL'
url: https://www.emergentmind.com/topics/stochastic-controlled-averaging-scaffold
type: topic
---

# SCAFFOLD: Controlled Averaging in FL

Stochastic Controlled Averaging (SCAFFOLD) is a variance reduction method for distributed stochastic optimization, with a primary application in federated learning (FL) regimes characterized by data heterogeneity and partial client participation. SCAFFOLD addresses the “client-drift” phenomenon by equipping each client and the server with control variates that correct the drift induced by non-IID local objectives. The algorithm achieves provable improvements in communication efficiency and convergence rates over classical schemes such as Federated Averaging (FedAvg) and LocalSGD, under notably weaker assumptions. Recent developments also demonstrate SCAFFOLD’s compatibility with communication compression and its rigorous convergence under general stochastic regimes.

## 1. Problem Formulation and Background

The foundational optimization problem is the distributed nonconvex finite-sum objective:
$$
\min_{x\in\mathbb{R}^d}\ f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x),
$$
where $f_i:\mathbb{R}^d \to \mathbb{R}$ denotes the private objective of client (or worker) $i$ and where $f_i$ need not equal $f_j$. In the federated setting, each client can query unbiased stochastic gradients $G_i(x,\xi^i)$, satisfying $\mathbb{E}_\xi[G_i(x,\xi)] = \nabla f_i(x)$ and finite variance.

Classical local-update protocols such as FedAvg perform several stochastic steps locally and aggregate parameters across clients periodically. For non-IID data, each local model drifts towards its local minimizer, resulting in biased global updates—termed “client drift”—that renders convergence slow or unstable [1910.06378].

## 2. SCAFFOLD Algorithm: Structure and Control Variates

SCAFFOLD enhances the robustness of federated optimization to heterogeneity by employing “control variates,” per-client and global, to correct the expected drift in local gradients.

### Core Protocol

Let $x^t$ be the global model at round $t$, $c^t$ the server-side control variate ($\approx \nabla f(x^t)$), and $c_i^t$ the client-side control variate ($\approx \nabla f_i(x^t)$). In each communication round:

- Each participating client $i$ performs $K$ local stochastic steps:
  $$
  y_{i}^{t,k+1} = y_{i}^{t,k} - \eta_{\ell} \left( \nabla F_i\left(y_{i}^{t,k}; \xi_i^{t,k}\right) - c_i^t + c^t \right), \quad y_{i}^{t,0} = x^t
  $$
- After $K$ steps, the client updates its control variate:
  $$
  c_i^{t+1} = c_i^t + \frac{1}{K} \sum_{k=0}^{K-1} \nabla F_i(y_{i}^{t,k};\xi_i^{t,k}) - c_i^t
  $$
- The server aggregates the updates:
  $$
  x^{t+1} = x^t + \frac{\eta_g}{S} \sum_{i\in S^t} \left( y_i^{t,K} - x^t \right)
  $$
  $$
  c^{t+1} = c^t + \frac{1}{N} \sum_{i\in S^t} \left( c_i^{t+1} - c_i^t \right)
  $$

The client’s local step thus becomes an (approximately) unbiased estimator for $\nabla f(y_{i}^{t,k})$, synchronized across the population via the control variates [1910.06378, 2501.04443].

### Algorithmic Simplifications

An operationally efficient variant reduces uplink communication by using a single “increment” $\Delta_i^t$ per round. Communication-efficient SCAFFOLD relies on this increment and is amenable to both unbiased and biased message compression [2308.08165].

## 3. Theoretical Foundations: Assumptions and Convergence Guarantees

SCAFFOLD’s theoretical analysis in recent work establishes rigorous performance bounds under general and practical assumptions:

### Key Assumptions

- **Smoothness:** Each $f_i$ is $L$-smooth.
- **Gradient Similarity:** $\sup_{x} \frac{1}{n} \sum_{i=1}^n \| \nabla f_i(x) - \nabla f(x) \|^2 \le \zeta^2$.
- **Hessian Similarity:** $\frac{1}{n} \sum_{i=1}^n \| \nabla f_i(x) - \nabla f(x) - (\nabla f_i(y) - \nabla f(y)) \|^2 \le \delta^2 \| x-y \|^2$.
- **Variance Boundedness:** For all $x$, $\mathbb{E}_\xi \| G_i(x, \xi) - \nabla f_i(x) \|^2 \le \sigma^2$.
- **Weak Convexity or Strong Convexity:** As applicable to the objective class [2501.04443, 1910.06378].

### Main Results

#### Non-Convex Setting
Under Hessian similarity and weak convexity, for stepsize $\eta$ appropriately chosen,
$$
G_{\mathrm{SCAF}} = \frac{2}{T} \sum_{r=0}^{R-1} \sum_{k=0}^{\tau-1} \mathbb{E} \| \nabla f(\bar{x}_{2r\tau+\tau+k}) \|^2
= \mathcal{O} \left(
\frac{L/\tau + \sqrt{L\mu} + \mu}{R}
+ \sqrt{ \frac{L \sigma^2}{n \tau R} }
+ (L\sigma)^{2/3} \tau^{1/3} R^{-2/3}
\right)
$$
[2501.04443]. The terms correspond to communication contraction, weak convexity correction, variance, and residual client drift.

#### Convex & Strongly Convex Cases
In the strongly convex regime, with full participation and under smoothness,
$$
R = \tilde{O} \left( \frac{\sigma^2}{\mu N K} + \frac{1}{\mu} \right)
$$
communication rounds suffice; this matches large-batch SGD, independent of heterogeneity [1910.06378].

#### General Variance Condition
Under the relaxed “ABC” condition, the complexity in the nonconvex smooth case is
$$
O\left( \frac{L_0}{T} + \frac{D + C \Delta_{\text{het}}}{\sqrt{n Q T}} + \cdots \right)
$$
with $\Delta_{\text{het}} = f^* - \frac{1}{n} \sum_{i=1}^n f_i^*$ quantifying the heterogeneity-induced gap [2301.12677].

#### Stochastic Regime: Markov Chain Analysis
Modeling the SCAFFOLD iterates as a Markov chain reveals geometric ergodicity in Wasserstein-2 distance and yields:
$$
\mathbb{E}[ \| \theta^T - \theta_\star \|^2 ] \leq 2 \left( 1 - \frac{\mu}{4} \right)^{H T} \{ \cdots \} + \frac{20 d \sigma^2}{N} + O( \gamma^{3/2} + \gamma^3 H )
$$
Linear speedup in $N$ is achievable up to higher-order bias terms [2503.07594].

## 4. Comparative Analysis: SCAFFOLD vs. FedAvg and LocalSGD

A central result is the ability of SCAFFOLD to remove the explicit dependence on heterogeneity that afflicts FedAvg and standard LocalSGD.

| Method      | Heterogeneity Complexity Term              | Leading Term with Full Participation               |
|-------------|-------------------------------------------|---------------------------------------------------|
| FedAvg      | $\mathcal{O}\left( \frac{\sigma^2}{\mu N K} + \frac{G}{\mu \sqrt{R}} \right)$  | Slower when $G > 0$                               |
| LocalSGD    | Similar structure, extra “drift” term     | Matches MbSGD only for low drift                   |
| SCAFFOLD    | No leading heterogeneity-induced penalty  | $\mathcal{O}\left( \frac{\sigma^2}{\mu N K} + \frac{1}{\mu} \right)$  |

SCAFFOLD attains its rate without requiring strong uniformity or bounded similarity. In special cases (e.g., quadratic objectives with matched Hessians), SCAFFOLD achieves additional “interpolation” acceleration [1910.06378, 2501.04443].

## 5. Extensions: Communication Compression and Partial Participation

SCAFFOLD’s structure is conducive to communication-efficient federated learning:

- **SCALLION:** Supports unbiased message compression. By communicating compressed increments, the communication cost is halved, and convergence matches full-precision SCAFFOLD asymptotically under standard unbiased compressors.
- **SCAFCOM:** Admits biased (contractive) compression, using momentum to neutralize systematic bias. Complexity results maintain robustness to heterogeneity and partial participation [2308.08165].

SCAFFOLD’s compressed variants preserve performance even under drastic uplink reduction (e.g., Top-$r$ sparsification, low-bit random dithering).

## 6. Proof Techniques and Analytical Innovations

Rigorous analysis of SCAFFOLD relies on several advanced methods:

- **Refined Consensus Error Bounds:** Tight decomposition of client–server gradient discrepancies using Hessian similarity and Lipschitz continuity, yielding sharper contraction rates for client drift [2501.04443].
- **Variance–Drift Decoupling:** Use of a “noiseless sequence” to remove coupling between stochastic gradient noise and drift recursion.
- **Markov Chain Contraction:** Wasserstein-metric coupling arguments underpin geometric convergence under stochastic gradients [2503.07594].
- **ABC Condition:** General variance frameworks clarify the effect of heterogeneity, with direct control over the heterogeneity gap $\Delta_{\rm het}$ [2301.12677].

## 7. Practical Considerations and Tuning Guidelines

- **Stepsizes:** Step-size parameters should satisfy $\eta \lesssim 1/L$ and scale with $\tau$, $n$, $R$, and $\sigma^2$ as dictated in [2501.04443].
- **Communication Interval ($\tau$, $K$, $Q$, $H$):** Longer intervals reduce communication but increase drift error ($\sim \tau^{1/3} R^{-2/3}$); SCAFFOLD endures larger intervals with mild heterogeneity.
- **Control Variate Storage:** Only the averaged gradients or increments need to be stored, so memory cost is negligible.
- **Heterogeneity Estimation:** The heterogeneity gap $\Delta_{\rm het}$ is operationally observable and diagnostic for variance-reduction needs [2301.12677].
- **Compression Regimes:** Compression schemes (SCALLION, SCAFCOM) are optimal under unbiased or contractive compressor designs, maintaining the $\mathcal{O}(N^{-1} \epsilon^{-2})$ gradient complexity when matched to problem parameters [2308.08165].

SCAFFOLD, alongside its communication-efficient variants, currently represents the most theoretically robust approach to heterogeneity- and communication-resilient federated optimization under standard smoothness and stochastic assumptions. Its regime of authentication-free superiority over both FedAvg and LocalSGD is precisely characterized in terms of Hessian similarity, weak convexity, and variance parameters, with rigorous nonasymptotic rates and stability assurances [2501.04443, 2308.08165, 1910.06378, 2503.07594, 2301.12677].

Source: https://www.emergentmind.com/topics/stochastic-controlled-averaging-scaffold