---
title: Projection-Based Zeroth-Order Fed SGD
url: https://www.emergentmind.com/topics/projection-based-zeroth-order-federated-sgd
type: topic
---

# Projection-Based Zeroth-Order Fed SGD

Projection-based zeroth-order federated stochastic gradient descent (SGD) refers to a class of optimization algorithms for federated learning that estimate gradients from function values (zeroth-order information) using randomized projections, often leveraging subspace structure to improve efficiency and convergence. These methods are of particular significance when explicit gradients are unavailable, and their development bridges zeroth-order optimization, projection techniques, and distributed stochastic optimization under data and system heterogeneity [2409.15955, 2509.21484, 2507.22855, 2412.16779].

## 1. Problem Formulation and Zeroth-Order Oracle Model

The goal is typically federated minimization of a global objective:
$$
F(x) = \frac{1}{M} \sum_{i=1}^{M} f_i(x), \qquad x \in \mathbb{R}^d
$$
where $f_i$ is client $i$’s (potentially nonconvex) local objective, e.g., $f_i(x) = \mathbb{E}_{\xi \sim \mathcal{D}_i}[F(x; \xi)]$. In zeroth-order federated settings, clients lack access to $\nabla f_i$ and are limited to querying scalar function values, for example:
- **Two-point finite difference:** $f_i(x + h u) - f_i(x - h u)$
- **One-point finite difference:** $f_i(x + h u) - f_i(x)$

The typical gradient surrogate at $x$ using a two-point estimate for a unit vector $u$ (possibly randomized) is:
$$
g_i(x) = \frac{f_i(x + h u) - f_i(x - h u)}{2h}\ u
$$
The overall protocol entails distributed estimation and aggregation of such surrogates to update $x_t$. Projection-based approaches modify the randomization mechanism to emphasize subspaces believed to contain significant descent directions [2409.15955].

## 2. Projection Subspace Construction from Historical Trajectories

A central innovation in recent work is the use of non-isotropic sampling guided by the optimization trajectory history:
- **Trajectory matrix:** At round $t$, form increments $\Delta_{t} = x_{t+1} - x_t$. Collect the last $\tau$ increments into $S_t = [\Delta_{t-1}, \ldots, \Delta_{t-\tau}] \in \mathbb{R}^{d \times \tau}$.
- **Basis via QR decomposition:** Apply thin QR: $S_t = U_t R_t$ with $U_t \in \mathbb{R}^{d \times k}$ ($k \leq \tau$), $U_t^\top U_t = I_k$. The columns of $U_t$ span a "promising" subspace based on recent optimization progress.
- **Projectors:** $P_t = U_t U_t^\top$ projects onto this subspace, $P_t^\perp = I_d - P_t$ onto its orthogonal complement.

This data-driven subspace is leveraged for sampling and regularization in gradient estimation, with the intent to enhance both exploitation of known good directions and exploration of new directions [2409.15955].

## 3. Non-Isotropic Gradient Estimation via Projections

Projections inform the covariance structure for direction sampling in zeroth-order estimation:
- **Sampling covariance:** $C_t = \alpha P_t + (1-\alpha) P_t^\perp / (d - k)$ for trade-off parameter $\alpha \in [0,1]$.
- **Sampling directions:** Sample $v_1 \sim \mathcal{N}(0, I_d)$, $v_2 \sim \mathcal{N}(0, I_k)$,
  $$
  u = \sqrt{\alpha} U_t v_2 + \frac{\sqrt{1-\alpha}}{\sqrt{d-k}} (I_d - U_t U_t^\top) v_1
  $$
  Normalize if desired.

This mechanism allows gradient estimates to concentrate on historically relevant subspaces when $\alpha$ is large, while still enabling search in the full space when $\alpha$ is small. The estimator remains unbiased for the smoothed gradient associated with the resulting Gaussian [2409.15955]. For comparison, other projection-based federated zeroth-order methods exploit different randomness distributions, such as the uniform measure on the $\ell_1$-sphere for refined concentration properties [2509.21484], or tangent-space projections for Riemannian constraints [2507.22855].

## 4. Algorithmic Protocols and Computational Structure

The generic projection-based zeroth-order FedSGD protocol [2409.15955] comprises:
1. **Model broadcast:** Server shares $x_t$ with selected clients.
2. **Subspace refresh:** Every $\tau$ rounds, server computes $U_t$ and shares to clients.
3. **Local updates:** Each client constructs $C_t$, runs $K$ local stochastic steps using projected zeroth-order directions, and returns updated $w_K$.
4. **Aggregation:** Server averages client models, records the increment for future $U_t$ construction.

Key computational overheads include the QR decomposition for the projection subspace ($O(d \tau^2)$ amortized), and $O(dk)$ per-sample cost for subspace-based sampling. This overhead is negligible for practical settings where $k,\tau \ll d$.

Related projection-based zeroth-order federated protocols include:
- **FedZero** [2509.21484]: uses projection onto constraints $\mathcal{X}$ at each server update, with $\ell_1$-sphere-based randomization for improved dimension dependence.
- **Riemannian projection-based ZO-FL** [2507.22855]: projection onto curved feasible sets (e.g., matrix manifolds), with random perturbations in the ambient Euclidean space and corrections for statistical heterogeneity.
- **Fed-ZOE** [2412.16779]: applies random projection compression to local update vectors for communication-efficient over-the-air aggregation but uses first-order local training, offering a contrast to “full” zeroth-order protocols.

## 5. Theoretical Guarantees and Bias-Variance Trade-Offs

Rigorous convergence analyses are available for projection-based zeroth-order federated SGD under various assumptions:
- **Nonconvex setting** [2409.15955]: Under $L$-smooth local objectives and bounded heterogeneity and sampling variance, with sufficiently small $h$ and appropriate stepsizes, the expected squared norm of the global gradient is bounded as
  $$
  \frac{1}{T} \sum_{t=0}^{T-1} \mathbb{E} \lVert \nabla f(x_t) \rVert^2 \leq O\left(\sqrt{\frac{d}{TMK}} + h^2\right)
  $$
  with more precise bounds incorporating $\alpha$ and the subspace dimension. The two-point estimator remains unbiased for the smoothed gradient, with the second moment depending on $h$ and the sampling structure.
- **Convex case and high-probability bounds** [2509.21484]: For constraint sets and $\ell_1$-randomized estimators, the excess loss achieves rates $O(DL\sqrt{d/(nm)})$ up to logarithmic factors, matching information-theoretic minimax lower bounds for federated zeroth-order optimization.
- **Riemannian manifolds** [2507.22855]: For projection-based zeroth-order methods on manifolds, convergence to a stationary point proceeds at $O(1/(\sqrt{n}\tau K) + 1/(n\tau m))$ with query complexity trade-offs dictated by the estimator batch size and geometric constants.

Bias-variance decompositions reveal trade-offs between exploration (full-space search, reducing bias) and exploitation (subspace-centric search, reducing variance). Moderate values of the mixing parameter ($\alpha \in [0.3, 0.6]$) empirically offer the best balance in projection-based finite-difference sampling [2409.15955].

## 6. Applications, Empirical Findings, and Protocol Comparisons

Extensive numerical validation has been conducted:
- **Tabular benchmarks:** Logistic regression, SVM, and MLP on MNIST, Fashion-MNIST, and RCV1 (under IID and non-IID splits) consistently show that projection-based methods ($\alpha$ moderate, small $k$) accelerate convergence (fewer function calls) relative to isotropic ZO-variants [2409.15955].
- **Sparsity effects:** On highly sparse tasks (e.g., RCV1), isotropic sampling can be competitive; on dense tasks, projection-based sampling confers clear advantages.
- **Manifold-constrained FL:** Projection-based zeroth-order estimators accelerate convergence in kPCA and low-rank MLP training on Stiefel and low-rank manifolds, achieving comparable rates to first-order methods while reducing tangent-space computation [2507.22855].
- **Over-the-air FL:** Fed-ZOE and related protocols apply projection-based compression to local model updates for substantial uplink reduction—e.g., $0.07\%$ of symbols relative to full-size communication—while maintaining baseline test accuracy [2412.16779].

A summary table of key empirical results from [2412.16779]:

| Method                        | CIFAR-10 | SVHN   | Tiny-ImageNet | CIFAR-100 | Brain-CT |
|-------------------------------|----------|--------|---------------|-----------|----------|
| Fed–OtA (100% symbols)        | 93.0%    | 95.4%  | 72.1%         | 74.3%     | 85.2%    |
| LoRA-OtA (10%)                | 91.8%    | 94.7%  | 69.0%         | 71.5%     | 83.3%    |
| ZO-OtA (100%)                 | 88.5%    | 92.1%  | 65.2%         | 68.0%     | 80.5%    |
| Fed–ZOE (0.07%)               | 92.6%    | 95.0%  | 71.0%         | 73.5%     | 84.0%    |

*Comm. Load is normalized to Fed–OtA full uplink.*

## 7. Variants and Extensions

Variants of projection-based zeroth-order federated SGD address distinct constraints and system architectures:
- **Manifold constraints:** Riemannian zeroth-order optimization with Euclidean perturbations and projection onto manifolds enables gradient-free FL under non-Euclidean model constraints [2507.22855].
- **High-probability guarantees:** Utilizing $\ell_1$-sphere randomization yields tighter concentration properties and improved high-probability regret bounds in federated convex ZO-SGD [2509.21484].
- **Communication compression:** Over-the-air protocols such as Fed-ZOE leverage projection-based compression, achieving both communication and computational reductions through low-dimensional random sketches of model updates [2412.16779].
- **Hybrid approaches:** Some recent methods perform first-order updates locally and use projection-based zeroth-order compression for transmission, combining the computational advantages of first-order optimization with the bandwidth efficiency of zeroth-order sketching.

A plausible implication is that ongoing research will increasingly hybridize projection-based zeroth-order techniques with first-order methods, especially when communication is the principal bottleneck.

---

References:  
[2409.15955], [2509.21484], [2507.22855], [2412.16779]

Source: https://www.emergentmind.com/topics/projection-based-zeroth-order-federated-sgd