Kuramoto-FedAvg: Sync in Federated Learning

• Kuramoto-FedAvg is a federated optimization algorithm that addresses client drift in non-IID settings by weighting client updates based on phase alignment.
• It reinterprets client updates as oscillators and employs dynamic aggregation weights derived from the sine of phase differences to emphasize well-aligned updates.
• Empirical evaluations on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate faster convergence and higher accuracy compared to baseline methods with minimal computational overhead.

Kuramoto-FedAvg is a federated optimization algorithm that addresses the challenge of slow convergence of federated learning under statistical heterogeneity due to client drift. The method reconceptualizes the server-side aggregation step as a synchronization problem inspired by the Kuramoto model of coupled oscillators. By dynamically weighting each client's update according to its phase alignment with the global update direction, Kuramoto-FedAvg suppresses misaligned ("out-of-phase") updates and amplifies those that are well-aligned, resulting in faster convergence and improved performance under non-IID data distributions (Muhebwa et al., 26 May 2025).

1. Phase-Based Synchronization: Theoretical Underpinnings

In Kuramoto-FedAvg, each client's local update vector is interpreted as an oscillator whose phase encodes its update direction. Let the global model at communication round $t$ be $w^t\in\mathbb{R}^d$, and let client $k$ return update $\Delta w_k^t=w_k^t-w^t$ after performing $E$ local SGD steps. The weighted mean update is

$$\bar\Delta^t = \sum_{k=1}^K p_k\,\Delta w_k^t, \qquad \sum_{k=1}^K p_k =1,$$

where $p_k$ typically reflects client data proportions. The phase of each client's update is

$$\theta_k^t = \arccos\left( \frac{\langle \Delta w_k^t, \bar\Delta^t\rangle} {\|\Delta w_k^t\|\;\|\bar\Delta^t\|} \right)\in[0,\pi],$$

measuring its directional alignment with $\bar\Delta^t$. This phase-based representation borrows directly from the Kuramoto model, which studies synchronization between coupled oscillators using the dynamics

$$\dot\theta_i = \omega_i + \frac{K}{N}\sum_{j=1}^N \sin(\theta_j-\theta_i).$$

Kuramoto-FedAvg does not use continuous ODE evolution but instead employs a coupling force via the sine of phase differences in a single-step aggregation.

2. Dynamic Aggregation Mechanism

Aggregation weights are constructed from the phase alignment of client updates. The mean phase is

$w^t\in\mathbb{R}^d$0

and each client's synchronization weight is

$w^t\in\mathbb{R}^d$1

Intuitively, clients whose update directions are closely aligned with the global mean receive higher weights, while those out of phase are down-weighted. The server's final update incorporates these dynamic weights with an optional coupling strength schedule,

$w^t\in\mathbb{R}^d$2

where $w^t\in\mathbb{R}^d$3 is decayed over time, e.g., $w^t\in\mathbb{R}^d$4.

3. Algorithmic Workflow

The Kuramoto-FedAvg communication round proceeds as follows:

1. Server broadcasts $w^t\in\mathbb{R}^d$5 to all clients.
2. Clients initialize $w^t\in\mathbb{R}^d$6 and perform $w^t\in\mathbb{R}^d$7 local SGD steps.
3. Clients compute updates $w^t\in\mathbb{R}^d$8 and send to server.
4. Server aggregates the mean update $w^t\in\mathbb{R}^d$9 and computes phases $k$0 for all clients.
5. Synchronization weights $k$1 are calculated using phase differences.
6. Server updates the model via the weighted sum above.

This approach introduces negligible overhead: server-side inner products and trigonometric evaluations scale linearly in the number of clients and model dimension ($k$2 per round).

4. Theoretical Convergence Guarantees

Standard assumptions from the FedAvg literature apply:

• The global objective $k$3 is minimized.
• Each local objective $k$4 is $k$5-smooth, stochastic gradients have variance $k$6.
• Gradient diversity (client drift) is quantified as $k$7.

FedAvg's convergence bound is

$k$8

For Kuramoto-FedAvg, the induced drift

$k$9

is strictly smaller: $\Delta w_k^t=w_k^t-w^t$0, leading to the improved bound

$\Delta w_k^t=w_k^t-w^t$1

implying fewer communication rounds $\Delta w_k^t=w_k^t-w^t$2 are needed to reach a target tolerance $\Delta w_k^t=w_k^t-w^t$3 in the non-IID setting. The core mechanism is the suppression of client drift via phase-aligned weighting.

5. Empirical Results

Experiments employed MNIST, Fashion-MNIST, and CIFAR-10, with label-shard partitioning to simulate varying degrees of non-IID client data (shards-per-client $\Delta w_k^t=w_k^t-w^t$4; clients $\Delta w_k^t=w_k^t-w^t$5). Training protocol involved:

• Local epochs $\Delta w_k^t=w_k^t-w^t$6, batch size 64.
• SGD with momentum 0.9, initial learning rate 0.01 (cosine decayed).
• Coupling strengths $\Delta w_k^t=w_k^t-w^t$7.

Key metrics were mean test accuracy and variance of client train losses ($\Delta w_k^t=w_k^t-w^t$8), the latter serving as a proxy for synchronization. Baselines included FedAvg and SCAFFOLD under identical regimes.

Main Empirical Findings

• Kuramoto-FedAvg reduced loss variance ("synchronization") 20–40 rounds earlier than baselines.
• Under strong non-IID (e.g., $\Delta w_k^t=w_k^t-w^t$9), Kuramoto-FedAvg outperformed FedAvg by 10–15% on CIFAR-10 and required fewer rounds to convergence.
• Ablation of $E$0 demonstrated that small values ($E$1) suffice, while excessive coupling ($E$2) degrades test accuracy, suggesting over-coupling impedes local exploration.

Dataset	No-sync	$E$3=0.005	0.1	0.3
CIFAR-10	61.42	73.30	72.52	67.88
MNIST	97.67	98.14	98.10	98.04
FMNIST	83.31	85.31	84.35	75.53

These results consistently establish the benefits of synchronization-based aggregation in the presence of client drift and statistical heterogeneity.

6. Practical Considerations and Extensions

Kuramoto-FedAvg introduces limited overhead: server-side computation per round is $E$4. No inter-client communication is necessary beyond standard FL primitives. The major hyperparameter requiring tuning is the initial coupling strength $E$5: insufficient coupling provides minimal acceleration, whereas excessive coupling can suppress valuable local exploration, reducing performance.

Potential future extensions include:

• Adaptive or per-client coupling schedules.
• Topology-aware or graph-based synchronization for partial client connectivity.
• Integration with momentum or advanced variance-reduction methods.
• Theoretical analysis for non-convex losses or asynchronous scenarios.

A plausible implication is that synchronization-based aggregation mechanisms, of which Kuramoto-FedAvg is a minimal instantiation, can be generalized and combined with other control-variates or regularization schemes for further advances in federated optimization under heterogeneity.

7. Summary and Significance

Kuramoto-FedAvg offers a theoretically justified, empirically substantiated, and lightweight strategy for improving the optimization dynamics of federated learning in non-IID settings. By mapping client updates to oscillator phases and aggregating via phase-sensitive synchronization, it directly mitigates client drift at the aggregation step rather than by modifying local optimization or exchanging additional control variates. This approach provides a strictly tighter convergence bound and improves mean accuracy and synchronization quality across multiple benchmarks, supporting the broader paradigm of coordination and synchronization-based architectures in federated optimization (Muhebwa et al., 26 May 2025).