---
title: FedProx Algorithm Overview
url: https://www.emergentmind.com/topics/fedprox-algorithm
type: topic
---

# FedProx Algorithm Overview

FedProx is a federated optimization algorithm that generalizes and re-parameterizes Federated Averaging (FedAvg) by replacing each client’s unconstrained local objective with a proximal-regularized subproblem centered at the current global model. Its defining purpose is to stabilize federated training under the two heterogeneities emphasized in federated learning: statistical heterogeneity, in which client data are non-IID, and systems heterogeneity, in which clients perform variable amounts of local work or exhibit straggling behavior. In the canonical formulation, these modifications are intentionally minimal at the server side—aggregation remains averaging—while materially changing the client-side optimization geometry [1812.06127].

## 1. Formal definition and relation to FedAvg

Let \(K\) denote the number of clients, \(F_k(w)\) the local objective on client \(k\), and \(p_k \ge 0\) aggregation weights with \(\sum_k p_k = 1\). The standard federated objective is

\[
F(w) = \sum_{k=1}^{K} p_k F_k(w).
\]

FedProx modifies the local problem solved at round \(t\) by client \(k\) to

\[
\min_{w}\; h_k(w; w^t) = F_k(w) + \frac{\mu}{2}\,\|w - w^{t}\|^2,
\]

where \(w^t\) is the global model broadcast by the server and \(\mu \ge 0\) is the proximal coefficient. A common first-order instantiation is

\[
w_{k}^{t+1} = w_{k}^{t} - \eta\big(\nabla F_k(w_{k}^{t}) + \mu (w_{k}^{t} - w^{t})\big),
\]

followed by server aggregation

\[
w^{t+1} = \sum_{k \in \mathcal{S}^{t}} p_k\, w_k^{t+1},
\]

with \(\mathcal{S}^{t}\) the participating clients in round \(t\) [2501.15288].

The critical algorithmic distinction from FedAvg is therefore local, not global. FedAvg and FedProx share the same aggregation rule; they differ in the client-side objective and update. FedAvg is recovered exactly when \(\mu = 0\) [2402.10035]. This equivalence is important both conceptually and practically: FedProx is not a different federation protocol so much as a proximal perturbation of standard local training.

In implementations, the proximal term is added to whatever local task loss is already used. The surveyed applications instantiate this with cross-entropy, binary cross-entropy, Dice-loss-based objectives, or logistic loss, and optimize locally with SGD or Adam, but the canonical abstraction is independent of the specific solver [2501.15288].

## 2. Heterogeneity, client drift, and the original convergence framework

FedProx was introduced to address a failure mode of local training under heterogeneity: client drift. When clients minimize only \(F_k(w)\), their local iterates can move toward client-specific optima that are poorly aligned with the global objective, especially when local datasets are non-IID or when clients complete different amounts of work before synchronization. The proximal term constrains this movement by tethering local optimization to the current global iterate [1812.06127].

The original framework explicitly models both statistical and systems heterogeneity. Statistical heterogeneity is captured through non-identical local objectives and a bounded dissimilarity condition. Systems heterogeneity is handled by allowing variable inexactness in local solves. In the original analysis, client \(k\) returns a \(\gamma_k^t\)-inexact solution \(w_k^{t+1}\) satisfying

\[
\|\nabla h_k(w_k^{t+1}; w^t)\| \le \gamma_k^t\, \|\nabla h_k(w^t; w^t)\|,
\quad \gamma_k^t \in [0,1].
\]

A smaller \(\gamma_k^t\) corresponds to more accurate local optimization and, operationally, to more local work. This formulation makes partial work a first-class object rather than an implementation nuisance [1812.06127].

The same work formalizes heterogeneity through \(B\)-local dissimilarity, with

\[
E_k\big[\|\nabla F_k(w)\|^2\big] \le \|\nabla f(w)\|^2 B^2,
\]

outside a small-gradient region. Under \(L\)-smoothness, a lower Hessian bound \(\nabla^2 F_k \succeq -L_- I\), and \(\bar{\mu} = \mu - L_- > 0\), the analysis derives an expected decrease bound for the global objective. Sufficient—though not necessary—conditions include \(\gamma B < 1\) and \(B/\sqrt{K} < 1\), making explicit the interaction among local inexactness, heterogeneity, and participation level [1812.06127].

This construction yields a particular interpretation of FedProx: it is neither merely a regularizer nor merely a robustness heuristic. It is a proximal-point style federated method whose local subproblem is designed to remain well behaved even when local computation budgets differ and local objectives are misaligned.

## 3. Later theory: beyond bounded dissimilarity and beyond stationarity

Subsequent theory substantially broadened the analytical picture. One line of work shows that favorable FedProx complexity bounds do not fundamentally require the local gradient dissimilarity assumptions used in early analyses. Through an algorithmic-stability argument, later results establish convergence guarantees for smooth and non-smooth federated problems without assuming local dissimilarity conditions, and derive linear speedup in the smooth minibatch setting with respect to minibatch size and the number of sampled devices [2206.05187].

In the smooth case, the minibatch stochastic proximal extension studied there attains rates of the form

\[
\max\!\Big\{T^{-2/3},\;(T b I)^{-1/2}\Big\},
\]

where \(T\) is the number of rounds, \(b\) the local minibatch size, and \(I\) the number of sampled devices. In the non-smooth weakly convex setting, the analysis proceeds through Moreau-envelope stationarity rather than ordinary gradient stationarity [2206.05187]. A central consequence is that local dissimilarity and smoothness are not treated as indispensable prerequisites for meaningful FedProx guarantees.

A different theoretical development, cast in RKHS non-parametric regression, reaches an apparently paradoxical conclusion: FedAvg and FedProx may fail to reach stationary points of the global empirical objective even for homogeneous linear regression, yet still achieve strong statistical rates. In that analysis, the empirical and RKHS estimation errors decay as \(1/t\) in general and exponentially for finite-rank kernels, and the authors introduce “federation gain” as the reduction in estimation error a client obtains by participating in federated learning rather than training locally [2106.15216]. This sharply separates optimization stationarity from statistical efficiency.

Taken together, these later analyses reposition FedProx from a method justified only by bounded-dissimilarity non-IID theory to one understood through multiple lenses: proximal-point stability, weakly convex analysis, statistical efficiency, and client-level benefit under heterogeneity.

## 4. Variants, extrapolation, and adaptive proximal weighting

FedProx has also served as the base algorithm for several extensions. The most mathematically explicit is server-side extrapolation. FedExProx updates

\[
x_{k+1} = x_k + \alpha \Big[\frac{1}{n}\sum_{i=1}^n \operatorname{prox}_{\gamma f_i}(x_k) - x_k\Big],
\]

which is equivalent to gradient descent on a Moreau-envelope surrogate with stepsize \(\alpha \gamma\). In the exact-prox, smooth convex interpolation regime, the optimal extrapolation parameter is reported as \(\alpha^* = 1/(\gamma L_\gamma)\), and the theory yields improved constants over vanilla FedProx [2405.13766].

A later analysis studies FedExProx with inexact proximal solutions. Under an absolute inexactness model, the method converges linearly only to a neighborhood of the optimum. Under a relative inexactness model, it retains exact linear convergence provided the inexactness is controlled; in the sharper biased-compression analysis, this requires \(\epsilon_2 < \mu/(4L_{\max})\) while preserving the optimal extrapolation choice \(\alpha \le 1/(\gamma L_\gamma)\) [2410.01410]. This gives a precise statement of how imperfect local prox computation degrades, but need not destroy, the benefits of extrapolation.

Task-specific variants modify the proximal coefficient itself. In wireless signal enhancement and recognition, FedProx+ replaces the global \(\mu\) with client-specific \(\mu_k\) values and updates them through an unspecified `AdaptiveMu` rule driven by local and global performance metrics; the local objective becomes

\[
h_k(\mathbf{w}; \mathbf{w}_t)
=
F_k(\mathbf{w}) + \frac{\mu_k}{2}\|\mathbf{w}-\mathbf{w}_t\|_2^2.
\]

The exact analytic form of `AdaptiveMu` is not given, but the method is described as increasing or decreasing \(\mu_k\) according to the ratio of local to global performance [2509.15718].

Another extension, G-FedProx, augments FedProx with Gaussian normalization of client updates. It filters or downweights updates outside the interval \([\mu_s - 2\sigma_s,\mu_s + 2\sigma_s]\), where \(\mu_s\) and \(\sigma_s\) are server-side empirical statistics, and reports “approximately 90% better convergence” in proxy experiments on regression statistics rather than end-to-end federated neural training [2406.00748].

These variants are best understood as descendants of the canonical FedProx idea rather than redefinitions of it. What remains invariant is the proximal anchoring of local updates to a server-provided reference model.

## 5. Empirical record across domains

Across the surveyed literature, FedProx is most consistently advantageous in non-IID or partially participating settings; under IID partitions, its effect is often small. The following studies illustrate the range of reported behavior.

| Domain | Setup | Reported FedProx-related outcome |
|---|---|---|
| Foundational heterogeneous benchmarks [1812.06127] | Synthetic, MNIST, FEMNIST, Shakespeare, Sent140; high heterogeneity and stragglers | FedProx improved absolute test accuracy by 22% on average in highly heterogeneous settings |
| 5G jamming detection [2501.15288] | Two-stage framework; supervised stage uses FedProx with \(\mu=0.01\) | With 6 clients: precision 0.94, recall 0.90, F1-score 0.92, accuracy 0.92; robust convergence within 30 rounds |
| OCT image classification [2402.10035] | 10 clients; label-skew via 2SPC and 3SPC | 2SPC: 90.57 ± 3.09 vs 87.22 ± 5.97 for FedAvg; 3SPC: 91.01 ± 2.51 vs 88.32 ± 2.30 |
| Heart disease prediction [2601.17183] | 4 heterogeneous hospital clients; logistic regression | \(\mu=0.05\): 85.00% accuracy vs 84.58% for FedAvg and 83.33% centralized; 18 vs 22 rounds to 95% of final accuracy |
| Mortality prediction on MIMIC-IV [2509.10517] | 5 care-unit clients; SMOTE-Tomek preprocessing | FedProx achieved F1-score 0.8831, accuracy 0.9062, precision 0.8857, recall 0.8812, with stable convergence |
| Ultrasound breast cancer segmentation [2510.19118] | 3 non-IID clients; Attention U-Net with FedProx | Round 6: accuracy 0.9607, IoU 0.5494, F1-score 0.7076, specificity 0.9919 |

The medical and wireless studies exhibit two recurrent patterns. First, under stronger heterogeneity, FedProx frequently improves robustness rather than merely the final mean metric. In the OCT study, this is reflected not only in higher mean accuracy under 2SPC and 3SPC label skew, but also in reduced variance in the more heterogeneous setting [2402.10035]. Second, the strongest results often come from composite pipelines in which FedProx is one stabilizing component among others, such as a frozen pretrained encoder in a two-stage federated classifier or class-balancing via SMOTE-Tomek before federated optimization [2501.15288].

This suggests that FedProx is often most effective when used to control optimization instability created by heterogeneity, rather than as a stand-alone remedy for every pathology in federated learning.

## 6. Hyperparameters, misconceptions, and limitations

The proximal coefficient \(\mu\) is the central FedProx hyperparameter, and the surveyed studies show that its useful range is task dependent. Reported values include \(\mu=0.01\) in 5G jamming detection, \(\mu=0.2\) in OCT classification, and \(\mu=0.05\) as the best value in a heart-disease simulation after a grid search over \(\{0.0, 0.01, 0.05, 0.1, 0.5\}\) [2501.15288][2402.10035][2601.17183]. In the large-scale Flower-based IoT study, \(\mu=1.0\) and \(\mu=2.0\) were explored under extreme straggler rates, with \(\mu=2\) reported to exhibit more stable convergence in those simulations [2312.15375].

A consistent finding is that \(\mu\) mediates a stability–progress trade-off. Too small a value yields behavior close to FedAvg; too large a value over-constrains local updates. The heart-disease study makes this explicit: \(\mu=0.05\) achieved 85.00% accuracy, whereas \(\mu=0.50\) fell to 81.25% despite reaching 95% of final accuracy in fewer rounds [2601.17183]. Several application papers also note that they do not provide a full \(\mu\)-sensitivity sweep, so reported values should be treated as tuned task-specific operating points rather than defaults [2402.10035].

Several misconceptions recur in discussions of FedProx. One is that it changes server aggregation; in fact, canonical FedProx keeps the FedAvg-style averaging step and alters only the local client objective [2402.10035]. Another is that FedProx should always outperform FedAvg. The OCT study reports that under IID partitions both methods are “on par” with centralized training and that the proximal term has negligible effect in that regime [2402.10035]. A third is that FedProx itself provides formal privacy guarantees. It preserves data locality, but differential privacy is a separate mechanism. When DP is added, accuracy can decline materially: one IoT study reports FedProx accuracy up to 0.9146 at round 20 without DP and 0.8353 with DP_FedProx, while a clinical DP pipeline identifies a privacy–utility frontier and shows FedProx outperforming FedAvg at comparable privacy levels rather than eliminating the privacy–utility trade-off [2312.15375][2508.10017].

Finally, FedProx is not a universal cure for federation pathologies. It does not by itself solve extreme label absence, class imbalance, poisoned updates, or the absence of formal secure aggregation. What it does provide is a precise proximal control mechanism for local drift. In the literature surveyed here, that mechanism is most valuable when heterogeneity is genuine, participation is partial or uneven, and local optimization would otherwise diverge, oscillate, or overfit to silo-specific structure.

Source: https://www.emergentmind.com/topics/fedprox-algorithm