---
title: Simple Averaging (ZO-Avg) in Zeroth-Order Optimization
url: https://www.emergentmind.com/topics/simple-averaging-zo-avg
type: topic
---

# Simple Averaging (ZO-Avg) in Zeroth-Order Optimization

Simple Averaging (ZO-Avg) is a zeroth-order variance-reduction scheme in which multiple independent gradient surrogates, constructed solely from function evaluations, are averaged before or during optimization. In its most explicit Euclidean form, ZO-Avg draws random directions \(u_1,\dots,u_m\) and replaces a single zeroth-order estimator by their arithmetic mean at the current iterate, thereby reducing estimator variance at the cost of additional oracle queries [2405.17761, 2509.15552]. Closely related constructions also appear under other names—notably gradient sample averaging, observation averaging, consensus averaging, moving-average estimators, and transported uniform averages on manifolds—so the literature does not attach a single invariant meaning to the label across all zeroth-order settings.

## 1. Terminological scope and canonical definitions

In the fixed-point Euclidean setting studied most directly, ZO-Avg means averaging \(m\) independent zeroth-order gradient estimates at the same point \(x\). One representative definition is
\[
\hat g_{\mathrm{avg}}(x)=\frac{1}{m}\sum_{j=1}^m \hat \nabla f(x;u_j),
\]
with independent Gaussian directions \(u_j\sim \mathcal N(0,I)\) [2405.17761]. A parallel formulation writes
\[
\hat g_m(x)=\frac{1}{m}\sum_{i=1}^m \hat g(x;u_i)
\approx \left(\frac{1}{m}UU^\top\right)\nabla f(x),
\]
where \(U=[u_1,\dots,u_m]\in\mathbb R^{d\times m}\) [2509.15552]. In that line of work, ZO-Avg is specifically a *multi-query aggregation rule*.

Two estimator families recur in this literature. For a smoothing radius \(\mu_s>0\), the one-point finite-difference estimator is
\[
\hat g(x;u)=\frac{f(x+\mu_s u)-f(x)}{\mu_s}\,u,
\]
while the two-point version is
\[
\hat g^{\mathrm{2p}}(x;u)=\frac{f(x+\mu_s u)-f(x-\mu_s u)}{2\mu_s}\,u.
\]
The fixed-budget conclusions for ZO-Avg are stated to hold for either one- or two-point estimators, because averaging \(m\) samples yields \(1/m\) variance reduction while reducing the iteration count to \(T=Q/m\) when the total query budget is \(Q\) [2509.15552].

The terminology broadens in adjacent literatures. In zeroth-order online ADMM, simple averaging appears as “gradient sample averaging” and “observation averaging,” both of which average multiple independent zeroth-order samples within an iteration [1710.07804]. In distributed stochastic zeroth-order optimization, the closest explicit analogue is a consensus-averaging primal scheme that averages neighboring iterates rather than multiple directions at a single point [2106.02958]. In Riemannian zeroth-order optimization, a baseline called “Simple Averaging (ZO-Avg)” is not introduced explicitly; instead, the paper states that a natural variant is uniform averaging of transported past zeroth-order estimators [2309.14506]. Another recent line of work uses “averaging” both for averaging \(t\) Gaussian oracle calls within each iteration and for averaging the stochastic ZO trajectory in expectation, yielding a perturbed first-order recursion [2604.25372].

| Setting | Averaged object | Representative form |
|---|---|---|
| Euclidean multi-query ZO | \(m\) direction estimators at fixed \(x\) | \(\hat g_m(x)=\frac1m\sum_i \hat g(x;u_i)\) |
| ZOO-ADMM minibatching | directions, observations, or both | \(\frac1q\sum_i \hat g_i\), \(\frac1{q_1q_2}\sum_{i,j}\hat g_{ij}\) |
| Distributed ZO | neighbor iterates | \(x_{i,k+1}=x_{i,k}-\gamma\sum_jL_{ij}x_{j,k}-\eta_k g^e_{i,k}\) |
| Riemannian ZO | transported past estimators | \(\bar g_{k+1}=\frac{k}{k+1}\mathrm{Transport}(\bar g_k)+\frac1{k+1}\mathrm{Transport}(G_\mu^k)\) |

## 2. Statistical mechanism: unbiasedness, smoothing bias, and variance structure

In the idealized Euclidean analysis of multi-query ZO-Avg, the averaged estimator is unbiased:
\[
\mathbb E[\hat g_m(x)]=\nabla f(x),
\]
and its mean-squared error satisfies
\[
\mathrm{MSE}(\hat g_m)=\mathbb E\big[\|\hat g_m(x)-\nabla f(x)\|^2\big]
=\frac{d+1}{m}\|\nabla f(x)\|^2.
\]
This is the basic \(1/m\) variance-reduction law: averaging \(m\) independent queries divides the estimator variance by \(m\) [2509.15552].

A more detailed composite-optimization analysis separates *sampling variance* from *coordinate-wise variance*. In that setting,
\[
F(x)=f(x)+\psi(x),\qquad f(x)=\frac1n\sum_{i=1}^n f_i(x),
\]
with each \(f_i\colon\mathbb R^d\to\mathbb R\) \(L\)-smooth and \(\mu\)-strongly convex, and \(\psi\) convex and proximable [2405.17761]. The paper uses the one-sided Gaussian-smoothing estimator
\[
\hat \nabla f_i(x,u,v)=\frac{f_i(x+vu)-f_i(x)}{v}\,u,\qquad u\sim\mathcal N(0,I_d),
\]
and shows that its expectation differs from \(\nabla f_i(x)\) by a smoothing-bias term proportional to \(Lv\) and Gaussian moments growing polynomially in \(d\). The same analysis states that
\[
\mathrm{Var}(\hat g_{\mathrm{avg}})=\frac{\mathrm{Var}(\hat \nabla f)}{m},
\]
so both sampling and coordinate-wise variances are reduced by a factor \(1/m\) under independence [2405.17761].

The coordinate-wise term is central in zeroth-order settings. The paper gives bounds of the form
\[
\mathbb E_u\big[\|\hat \nabla f(x,u)-\nabla f(x)\|^2\big]
\lesssim 2(d+1)\|\nabla f(x)\|^2 + O(L^2v^2(d+6)^3),
\]
and for a single component,
\[
\mathbb E_u\big[\|\hat \nabla f_i(x,u)-\nabla f(x)\|^2\big]
\lesssim 3(d+1)\|\nabla f(x)\|^2 + 3(d+2)\|\nabla f_i(x)-\nabla f(x)\|^2 + O(L^2v^2(d+6)^3).
\]
These bounds make explicit that random-direction noise scales with dimension and with gradient magnitude [2405.17761].

In composite proximal problems, the nonvanishing-gradient issue is sharper because optimality is
\[
0\in \nabla f(x^*)+\partial\psi(x^*),
\]
so \(\nabla f(x^*)\neq 0\) in general. The paper therefore argues that direction-induced variance need not disappear at the solution; as a consequence, naive ZO-Avg may require large \(m\) to suppress persistent noise, which directly raises per-iteration query cost [2405.17761].

The same variance-reduction logic appears in zeroth-order ADMM minibatching. There, the base one-point estimator is
\[
\hat{\mathbf g}_t=
\frac{f(\mathbf x_t+\beta_t\mathbf z_t;\mathbf w_t)-f(\mathbf x_t;\mathbf w_t)}{\beta_t}\,\mathbf z_t,
\]
with \(\mathbf z_t\) sampled from a distribution satisfying \(\mathbb E[\mathbf z\mathbf z^\top]=I\). Averaging \(q\) directions, \(q\) observations, or \(q_1q_2\) direction-observation pairs reduces the variance term in the regret bound, replacing the dimension-dependent factor \(s(m)=m\) by \(s(m)/(q_1q_2)\) and improving the convex-rate expression to \(O(\sqrt{1+m/(q_1q_2)}/\sqrt{T})\) [1710.07804].

## 3. Fixed query budgets and the multi-query paradox

A major recent result is that the variance reduction of ZO-Avg does not automatically translate into better optimization efficiency once oracle cost is accounted for. Under a fixed total query budget \(Q\), using \(m\) queries per iteration leaves only
\[
T=\frac{Q}{m}
\]
iterations. The paper analyzes this trade-off for Gaussian directions across strongly convex, convex, non-convex, and stochastic convex settings and proves that, for ZO-Avg, allocating more than one query per iteration is always query-inefficient under the stated assumptions [2509.15552].

In the strongly convex case, with stepsize
\[
\eta_t=\frac{m}{L(m+d+1)},
\]
the bound is
\[
\mathbb E[f(x_T)-f(x^*)]
\le
\left(1-\frac{\mu_{\mathrm{sc}}\,m}{L(m+d+1)}\right)^T\big(f(x_0)-f(x^*)\big).
\]
Substituting \(T=Q/m\) yields
\[
\mathbb E[f(x_T)-f(x^*)]
\le
\exp\!\left(-\frac{\mu_{\mathrm{sc}}\,Q}{L(m+d+1)}\right)\big(f(x_0)-f(x^*)\big),
\]
so the exponent is best at \(m=1\). The paper therefore identifies the single-query allocation as optimal under fixed \(Q\) [2509.15552].

The same monotonicity appears in convex and non-convex regimes. For convex \(L\)-smooth objectives, the paper gives
\[
\mathbb E[f(x_T)-f(x^*)]
\le
\frac{L(m+d+1)}{2Q}
\left(\|x_0-x^*\|^2+\frac{2}{L}(f(x_0)-f(x^*))\right),
\]
which is minimized at \(m=1\). For non-convex \(L\)-smooth objectives,
\[
\min_t \mathbb E[\|\nabla f(x_t)\|^2]
\le
\frac{2L(f(x_0)-f^*)(m+d+1)}{Q},
\]
again minimized at \(m=1\) [2509.15552].

The stochastic convex result is formulated for a weighted-average iterate \(\bar x_T\) under a diminishing stepsize \(\eta_t=\eta_0/\sqrt{t+1}\), with \(\eta_t\le 1/(4L)\). The stated asymptotics show that both the numerator and denominator in the bound worsen as \(m\) increases, while the iteration count shrinks to \(Q/m\), leading once more to the single-query optimum [2509.15552].

The scope of this “always” claim is explicitly limited. The paper states it under \(L\)-smoothness, Gaussian or isotropic direction sampling, negligible finite-difference bias in the idealized analysis, and the theorem-specific stepsize choices. Within that framework, the conclusion is categorical: the de facto multi-query ZO-Avg rule is dominated, in query efficiency, by its \(m=1\) specialization [2509.15552].

## 4. Composite proximal optimization and the limits of naive averaging

The fixed-budget paradox does not eliminate the utility of averaging in all zeroth-order settings; rather, it clarifies that *where* and *how* averaging is inserted matters. In composite finite-sum optimization with a nonsmooth proximable regularizer, ZO-Avg reduces both sampling and coordinate-wise variances by \(1/m\), but it pays \(\mathcal O(m)\) stochastic zeroth-order oracle calls per iteration. Because \(\nabla f(x^*)\neq 0\) in general at a proximal optimum, the coordinate-wise variance can remain nonvanishing near the solution, so achieving high accuracy with ZO-Avg may require large \(m\), which the paper describes as costly [2405.17761].

This observation motivates a distinct variance-reduction strategy in the same paper. Zeroth-order Proximal Double Variance Reduction (ZPDVR) combines an SVRG/SPIDER-like control variate
\[
g_k=\hat \nabla f_i(x_k,u_k)-\hat \nabla f_i(w_k,u_k)+\tilde \nabla f(w_k)
\]
with a temporal averaging recursion for an auxiliary estimator \(h_k\),
\[
h_{k+1}=h_k+\frac{1}{d+2}\big[\hat \nabla f(x_k,u)-uu^\top h_k\big].
\]
The scaling \(1/(d+2)\) is tied to the Gaussian identity \(\mathbb E[uu^\top v]=(d+2)v\), and the paper proves a contraction for \(\|h_{k+1}-\nabla f(x^*)\|^2\) showing progressive reduction of coordinate-wise variance over time [2405.17761].

The resulting complexity statement is
\[
\mathcal O\!\big(d(n+\kappa)\log(1/\epsilon)\big)
\]
stochastic zeroth-order oracle queries in the strongly convex smooth setting, with expected \(\mathcal O(1)\) oracle calls per iteration when the snapshot probability is chosen as \(p=1/n\). The point of comparison is explicit: naive ZO-Avg spends \(\mathcal O(m)\) fresh queries every iteration, whereas ZPDVR amortizes averaging across time and suppresses both sampling and direction-induced variance without \(\Omega(d)\) coordinate-wise finite differences [2405.17761].

The empirical comparison in that paper reinforces the theoretical distinction. On \(L_1+L_2\)-regularized logistic regression for a9a, w8a, covtype, and gisette, ZPDVR exhibits linear convergence and the best performance in stochastic zeroth-order complexity plots, while ZPSVRG with random-direction gradients only stalls near the optimum, including stagnation around \(10^{-4}\) on gisette, which the paper attributes to persistent coordinate-wise variance [2405.17761].

## 5. Minibatching, online ADMM, and decentralized averaging

In zeroth-order online ADMM, simple averaging appears not as a fixed-budget paradox but as a minibatching device embedded in a structured primal-dual algorithm. The base estimator is a one-point randomized finite difference,
\[
\hat{\mathbf g}_t=
\frac{f(\mathbf x_t+\beta_t\mathbf z_t;\mathbf w_t)-f(\mathbf x_t;\mathbf w_t)}{\beta_t}\,\mathbf z_t,
\]
and the paper introduces three averaging schemes: gradient sample averaging over \(q\) directions, observation averaging over \(q\) samples \(\mathbf w_{t,i}\), and hybrid averaging over \(q_1\) directions and \(q_2\) observations [1710.07804].

The structural role of ZO-Avg in that framework is straightforward. The ADMM \(x\)-update uses \(\hat{\mathbf g}_t\), the \(y\)-update is a proximal step for the regularizer, and the dual variable is updated as usual. Averaging reduces the variance term in the regret analysis, leading to an improved convex rate
\[
O\!\left(\frac{\sqrt{1+m/(q_1q_2)}}{\sqrt{T}}\right),
\]
where \(m\) denotes the dimension of the optimization variable. When \(q_1q_2=\Theta(m)\), the dimension dependence is mitigated and the rate matches the first-order \(O(1/\sqrt{T})\) form. In the strongly convex case, the single-sample ZOO-ADMM rate is \(O(m\log T/T)\) [1710.07804].

The same paper also reports application-level consequences. In black-box model blending on Yahoo! Music, both gradient sample averaging and observation averaging accelerate convergence, and with minibatch size \(q=50\), ZOO-ADMM reaches the best reported RMSE among the compared approaches after roughly \(10{,}000\) iterations. In sensor selection and sparse Cox regression, the method preserves the proximal and feasibility advantages of ADMM while remaining gradient-free [1710.07804].

Distributed stochastic zeroth-order optimization uses a different notion of simple averaging. The paper does not name a baseline “ZO-Avg,” but it identifies a distributed primal algorithm that performs consensus averaging over neighbors followed by a local zeroth-order descent step:
\[
x_{i,k+1}=x_{i,k}-\gamma\sum_{j=1}^n L_{ij}x_{j,k}-\eta_k g^e_{i,k}.
\]
Each agent draws \(u_{i,k}\in\mathbb S^p\), evaluates the function at \(x_{i,k}\) and \(x_{i,k}+\delta_{i,k}u_{i,k}\), and thus uses exactly two function queries per iteration [2106.02958].

Under connected-graph, smoothness, and state-dependent variance assumptions, this consensus-based scheme achieves
\[
\frac1T\sum_{k=0}^{T-1}\mathbb E\big[\|\nabla f(\bar x_k)\|^2\big]
=
\mathcal O\!\left(\sqrt{\frac{p}{nT}}\right)+\mathcal O\!\left(\frac{n}{T}\right)
\]
with a known-horizon schedule, and under the Polyak–Łojasiewicz condition it achieves
\[
\mathbb E[f(\bar x_T)-f^*]
=
\mathcal O\!\left(\frac{p}{nT}\right)+\mathcal O\!\left(\frac{p}{T^2}\right).
\]
The leading terms are linear-speedup results in the number of agents \(n\) [2106.02958]. Here, simple averaging is fundamentally *consensus averaging of iterates*, not *multi-query averaging of directions*.

## 6. Geometric and dynamical-systems perspectives

On Riemannian manifolds, the paper “Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms” does not explicitly define a baseline called “Simple Averaging (ZO-Avg).” Instead, it proposes Zo-RASA, whose core recursion is an exponential moving average of transported zeroth-order gradient estimators,
\[
g^{k+1}=(1-\tau_k)\,\mathrm{Transport}(g^k)+\tau_k\,\mathrm{Transport}(G_\mu^k),
\]
implemented either with parallel transport or with vector transport plus retractions [2309.14506]. The paper states that a natural uniform-average variant consistent with the framework is
\[
\bar g_{k+1}=\frac{k}{k+1}\,\mathrm{Transport}(\bar g_k)+\frac{1}{k+1}\,\mathrm{Transport}(G_\mu^k),
\]
which aligns all past estimates in the current tangent space.

The distinction between EMA and uniform averaging is consequential. The paper proves its optimal \(O(d/\epsilon^4)\) oracle-complexity results for schedules \(\tau_k=1/\sqrt{N}\) or \(\tau_k=1/\sqrt{dN}\), with one-sample iterations after a warm-start in one regime, but it does **not** state an explicit theorem for the uniform-weight choice \(\alpha_k=1/(k+1)\) [2309.14506]. Thus, in the Riemannian setting, “simple averaging” is best viewed as an admissible construction within the transport-and-averaging template rather than as the theorem-bearing algorithm of the paper.

A different generalization appears in the input-to-state stability analysis of zeroth-order methods. That work does not explicitly name an algorithm “ZO-Avg,” but it uses averaging in two senses. First, it averages \(t\) independent Gaussian oracle calls within each iteration, reducing the variance bound from \(\sigma^2\) to \(\sigma^2/t\). Second, it defines the averaged iterate as the conditional expectation of the random ZO trajectory and shows that the resulting dynamics satisfy the first-order update plus a bounded perturbation [2604.25372]. For gradient descent,
\[
\bar z_{k+1}=\bar z_k-h\nabla f(\bar z_k)+q_k,
\]
with \(\sup_k\|q_k\|\) bounded by a smoothing-bias term proportional to \(h\mu L_1(n+3)^{3/2}\) and a variance term proportional to \(hL_1\sqrt{\sigma^2/t}\), up to the paper’s step-size-dependent constants. Analogous perturbed-recursion results are stated for heavy-ball and Nesterov acceleration [2604.25372].

Under strong convexity or under \(L_2\)-regularized non-strongly-convex dynamics, the paper applies input-to-state stability theorems to conclude that the averaged ZO recursion has the same decay rate as its first-order counterpart and converges to a neighborhood of the first-order fixed point whose radius can be made arbitrarily small by tuning \(\mu\), \(h\), and \(t\) [2604.25372]. In that sense, averaging is no longer merely a finite-sample variance reducer; it becomes the analytical bridge that expresses zeroth-order methods as perturbed first-order systems.

The broader conceptual background is classical first-order stochastic approximation. In the survey of stochastic-gradient averaging techniques, simple averaging refers to Polyak–Ruppert averaging of iterates,
\[
\bar x_T=\frac1T\sum_{t=1}^T x_t,
\]
rather than averaging multiple gradient surrogates inside a single step. Under unbiased stochastic gradients, suitable smoothness and convexity assumptions, and classical stepsize conditions \(\sum_k\eta_k=\infty\), \(\sum_k\eta_k^2<\infty\), the survey states asymptotic normality with optimal covariance and reports non-asymptotic \(O(1/n)\) rates for smooth convex problems [2603.09634]. Tail averaging, window averaging, exponential moving averages, and stochastic weight averaging are then presented as refinements of the same bias-variance logic. The zeroth-order literature inherits that logic but redistributes it across queries, time, agents, or tangent spaces depending on the information model.

A recurrent misconception is therefore that “more averaging” is uniformly better. The literature shows a sharper picture. When ZO-Avg means averaging more random directions at a fixed point under a fixed query budget, the optimal choice can be the single-query case \(m=1\) [2509.15552]. When averaging is amortized across time, embedded in control variates, aligned by transport, or used to form an averaged iterate with bounded perturbations, it can recover strong stability or complexity properties that naive multi-query averaging does not attain [2405.17761, 2309.14506, 2604.25372].

Source: https://www.emergentmind.com/topics/simple-averaging-zo-avg