---
title: 'Kuramoto-FedAvg: Sync in Federated Learning'
url: https://www.emergentmind.com/topics/kuramoto-fedavg
type: topic
---

# Kuramoto-FedAvg: Sync in Federated Learning

Kuramoto-FedAvg is a federated optimization algorithm that addresses the challenge of slow convergence of federated learning under statistical heterogeneity due to client drift. The method reconceptualizes the server-side aggregation step as a synchronization problem inspired by the Kuramoto model of coupled oscillators. By dynamically weighting each client's update according to its phase alignment with the global update direction, Kuramoto-FedAvg suppresses misaligned ("out-of-phase") updates and amplifies those that are well-aligned, resulting in faster convergence and improved performance under non-IID data distributions [2505.19605].

## 1. Phase-Based Synchronization: Theoretical Underpinnings

In Kuramoto-FedAvg, each client's local update vector is interpreted as an oscillator whose phase encodes its update direction. Let the global model at communication round $t$ be $w^t\in\mathbb{R}^d$, and let client $k$ return update $\Delta w_k^t=w_k^t-w^t$ after performing $E$ local SGD steps. The weighted mean update is

$$
\bar\Delta^t = \sum_{k=1}^K p_k\,\Delta w_k^t, \qquad \sum_{k=1}^K p_k =1,
$$

where $p_k$ typically reflects client data proportions. The phase of each client's update is

$$
\theta_k^t = \arccos\left(
    \frac{\langle \Delta w_k^t, \bar\Delta^t\rangle}
         {\|\Delta w_k^t\|\;\|\bar\Delta^t\|}
  \right)\in[0,\pi],
$$

measuring its directional alignment with $\bar\Delta^t$. This phase-based representation borrows directly from the Kuramoto model, which studies synchronization between coupled oscillators using the dynamics

$$
\dot\theta_i = \omega_i + \frac{K}{N}\sum_{j=1}^N \sin(\theta_j-\theta_i).
$$

Kuramoto-FedAvg does not use continuous ODE evolution but instead employs a coupling force via the sine of phase differences in a single-step aggregation.

## 2. Dynamic Aggregation Mechanism

Aggregation weights are constructed from the phase alignment of client updates. The mean phase is

$$
\bar\theta^t = \frac1K\sum_{k=1}^K \theta_k^t,
$$

and each client's synchronization weight is

$$
\rho_k^t = \frac{\sin(\bar\theta^t - \theta_k^t)}{\sum_{j=1}^K \sin(\bar\theta^t - \theta_j^t)}.
$$

Intuitively, clients whose update directions are closely aligned with the global mean receive higher weights, while those out of phase are down-weighted. The server's final update incorporates these dynamic weights with an optional coupling strength schedule,

$$
w^{t+1} = w^t + \kappa_t \sum_{k=1}^K \rho_k^t\,\Delta w_k^t,
$$

where $\kappa_t$ is decayed over time, e.g., $\kappa_t = \kappa_0/(1+\beta t)$.

## 3. Algorithmic Workflow

The Kuramoto-FedAvg communication round proceeds as follows:

1. **Server broadcasts** $w^t$ to all clients.
2. **Clients initialize** $w_k^{t,0} \leftarrow w^t$ and perform $E$ local SGD steps.
3. **Clients compute updates** $\Delta w_k^t$ and send to server.
4. **Server aggregates** the mean update $\bar\Delta^t$ and computes phases $\theta_k^t$ for all clients.
5. **Synchronization weights** $\rho_k^t$ are calculated using phase differences.
6. **Server updates the model** via the weighted sum above.

This approach introduces negligible overhead: server-side inner products and trigonometric evaluations scale linearly in the number of clients and model dimension ($\mathcal{O}(Kd)$ per round).

## 4. Theoretical Convergence Guarantees

Standard assumptions from the FedAvg literature apply:

- The global objective $F(w) = \sum_{k=1}^K p_k F_k(w)$ is minimized.
- Each local objective $F_k(w)$ is $L$-smooth, stochastic gradients have variance $\sigma_k^2$.
- Gradient diversity (client drift) is quantified as $\Gamma(t) = \sum p_k \|\nabla F_k(w_t)\|^2 - \|\nabla F(w_t)\|^2$.

FedAvg's convergence bound is

$$
\mathbb{E}[F(w_{t+1})] 
\le 
F(w_t) 
- \eta_t\|\nabla F(w_t)\|^2 
+ \frac{L\eta_t^2}{2}\Gamma(t) 
+ \frac{L\eta_t^2}{2}\sigma^2.
$$

For Kuramoto-FedAvg, the induced drift

$$
\Gamma_{\mathrm{Kur}}(t) = \sum \rho_k(t)^2 \|\nabla F_k(w_t)\|^2 - \|\nabla F(w_t)\|^2
$$

is strictly smaller: $\Gamma_{\mathrm{Kur}}(t) < \Gamma(t)$, leading to the improved bound

$$
\mathbb{E}[F(w_{t+1})]
\le
F(w_t)
- \eta_t\|\nabla F(w_t)\|^2 
+ \frac{L\eta_t^2}{2}\Gamma_{\mathrm{Kur}}(t) 
+ \frac{L\eta_t^2}{2}\sigma^2,
$$

implying fewer communication rounds $T$ are needed to reach a target tolerance $\epsilon$ in the non-IID setting. The core mechanism is the suppression of client drift via phase-aligned weighting.

## 5. Empirical Results

Experiments employed MNIST, Fashion-MNIST, and CIFAR-10, with label-shard partitioning to simulate varying degrees of non-IID client data (shards-per-client $s\in\{3,5,10\}$; clients $K\in\{10,20\}$). Training protocol involved:

- Local epochs $E=2$, batch size 64.
- SGD with momentum 0.9, initial learning rate 0.01 (cosine decayed).
- Coupling strengths $\kappa_0\in\{0.005,0.1,0.3\}$.

Key metrics were mean test accuracy and variance of client train losses ($\mathrm{Var}_k\,\mathcal{L}_k(w^t)$), the latter serving as a proxy for synchronization. Baselines included FedAvg and SCAFFOLD under identical regimes.

### Main Empirical Findings

- Kuramoto-FedAvg reduced loss variance ("synchronization") 20–40 rounds earlier than baselines.
- Under strong non-IID (e.g., $s=3$), Kuramoto-FedAvg outperformed FedAvg by 10–15% on CIFAR-10 and required fewer rounds to convergence.
- Ablation of $\kappa_0$ demonstrated that small values ($\approx 0.005$) suffice, while excessive coupling ($\kappa_0=0.3$) degrades test accuracy, suggesting over-coupling impedes local exploration.

| Dataset     | No-sync | $\kappa_0$=0.005 | 0.1   | 0.3   |
|-------------|---------|------------------|-------|-------|
| CIFAR-10    | 61.42   | 73.30            | 72.52 | 67.88 |
| MNIST       | 97.67   | 98.14            | 98.10 | 98.04 |
| FMNIST      | 83.31   | 85.31            | 84.35 | 75.53 |

These results consistently establish the benefits of synchronization-based aggregation in the presence of client drift and statistical heterogeneity.

## 6. Practical Considerations and Extensions

Kuramoto-FedAvg introduces limited overhead: server-side computation per round is $\mathcal{O}(Kd)$. No inter-client communication is necessary beyond standard FL primitives. The major hyperparameter requiring tuning is the initial coupling strength $\kappa_0$: insufficient coupling provides minimal acceleration, whereas excessive coupling can suppress valuable local exploration, reducing performance.

Potential future extensions include:

- Adaptive or per-client coupling schedules.
- Topology-aware or graph-based synchronization for partial client connectivity.
- Integration with momentum or advanced variance-reduction methods.
- Theoretical analysis for non-convex losses or asynchronous scenarios.

A plausible implication is that synchronization-based aggregation mechanisms, of which Kuramoto-FedAvg is a minimal instantiation, can be generalized and combined with other control-variates or regularization schemes for further advances in federated optimization under heterogeneity.

## 7. Summary and Significance

Kuramoto-FedAvg offers a theoretically justified, empirically substantiated, and lightweight strategy for improving the optimization dynamics of federated learning in non-IID settings. By mapping client updates to oscillator phases and aggregating via phase-sensitive synchronization, it directly mitigates client drift at the aggregation step rather than by modifying local optimization or exchanging additional control variates. This approach provides a strictly tighter convergence bound and improves mean accuracy and synchronization quality across multiple benchmarks, supporting the broader paradigm of coordination and synchronization-based architectures in federated optimization [2505.19605].

Source: https://www.emergentmind.com/topics/kuramoto-fedavg