---
title: Bounded Gradient Variance in Optimization
url: https://www.emergentmind.com/topics/bounded-gradient-variance
type: topic
---

# Bounded Gradient Variance in Optimization

Bounded gradient variance denotes a family of assumptions and estimator-specific guarantees that control the second moment of stochastic gradient-like quantities. In the classical stochastic-approximation formulation, this control appears as a conditional bound on the gradient error, such as $\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2$, while in stochastic variational inequalities it appears as a uniform oracle bound $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$ [1304.5586] [2602.05531]. Recent work has both generalized and localized this notion: some analyses allow variance growth proportional to the squared norm of the iterate, some require finite variance only at a solution, and some replace Euclidean second moments by geometry-dependent surrogates such as $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$ in stochastic mirror descent [2604.22581] [2404.12213]. In parallel, estimator-level analyses now provide explicit variance bounds for zeroth-order finite differences, reparameterization gradients in variational inference, REINFORCE in linear-quadratic control, and clipped first-order methods under heavy-tailed noise [2205.14737] [1906.08241] [1910.01249] [2512.14686].

## 1. Classical bounded-variance formulations

The classical bounded-variance model in stochastic optimization is a second-moment condition on the stochastic error. In perturbed proximal gradient, the error sequence $e_k$ is adapted to the filtration $\mathcal F_k=\sigma(e_1,\dots,e_k)$ and satisfies
\[
\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,
\]
with $\sigma_k^2\to0$ often realized by increasing sample sizes [1304.5586]. Stronger assumptions in the same framework include coordinate-wise sub-Gaussian tails and an almost-sure deterministic bound $\|e_k\|\le U_k$, with the paper explicitly noting the implication chain $C\Rightarrow B\Rightarrow A$ [1304.5586].

In stochastic variational inequalities, the standard assumption is the uniform bounded-variance condition
\[
\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,
\]
for an unbiased oracle $\mathbb E_\xi[G(z,\xi)]=G(z)$ [2602.05531]. This is the form explicitly labeled $(BV)$ in the literature summarized there. It is a global condition, uniform over the domain, and is commonly paired with Lipschitz continuity of the operator $G$ and convexity of the regularizer $r$ [2602.05531].

A related but more algorithm-facing formulation appears in black-box variational Bayesian inference through the $ABC$ condition:
\[
\mathbb E\|g(\theta)\|_2^2 \le 2A(F(\theta)-F^*) + B\|\nabla F(\theta)\|_2^2 + C.
\]
This does not postulate a constant variance bound directly; instead it controls the expected squared norm of the stochastic gradient by a linear combination of function suboptimality, gradient norm, and a residual constant [2303.10472]. The paper shows that black-box variational inference gradients satisfy this structure under smoothness and quadratic-growth assumptions.

These formulations share a common role: they convert stochasticity into a quantitatively tractable perturbation term. What differs is whether the control is uniform in the domain, local to a solution, dependent on geometry, or tied to objective suboptimality.

## 2. Relaxations beyond uniform boundedness

Several recent analyses replace global bounded variance by weaker assumptions that still permit convergence proofs.

| Setting | Variance condition | Source |
|---|---|---|
| Classical SVI | $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$ | [2602.05531] |
| Unbounded-variance SVI | $\mathbb E\|G(z,\xi)-G(z)\|^2 \le B^2\|z-z_0\|^2+\sigma^2$ | [2602.05531] |
| Pseudo-monotone SVI | $\mathbb E[\|F(\xi,x)-T(x)\|^2]\le \sigma(x^*)^2(1+\|x-x^*\|)^2$ | [1703.00260] |
| Stochastic KM | $\exists\,p\in\mathrm{Fix}(T)$ with $\mathbb E_\xi[\|T_\xi(p)-T(p)\|^2]\le \sigma_*^2$ | [2604.22581] |
| Stochastic mirror descent | $\sigma^2_{*,\eta}:=(f^*-f_\eta^*)/\eta$ | [2404.12213] |

The unbounded-variance condition
\[
\mathbb E\|G(z,\xi)-G(z)\|^2 \le B^2\|z-z_0\|^2+\sigma^2
\]
is motivated by the observation that the classical bounded-variance condition fails, for example, for bilinear min-max on $\mathbb R^d$, whereas the relaxed condition holds for stochastic oracles whose variance can grow as fast as the squared norm of the optimization variable [2602.05531]. This changes the problem from one of proving uniform noise control to one of proving that iterates remain sufficiently controlled in expectation.

An earlier pseudo-monotone SVI analysis already used a non-uniform variance model:
\[
\mathbb E\big[\|F(\xi,x)-T(x)\|^2\big]\le \sigma(x^*)^2(1+\|x-x^*\|)^2.
\]
The associated empirical-average errors $\epsilon_1^k,\epsilon_2^k$ then satisfy conditional second-moment bounds proportional to $(1+\|x^k-x^*\|)^2/N_k$, making variance reduction a matter of growing the sample size sequence $N_k$ [1703.00260].

A further relaxation appears in stochastic Krasnosel'skii-Mann iterations. Instead of requiring a uniform variance bound on $T_\xi(x)-T(x)$ for all $x$, the assumption is only that there exists one fixed point $p\in \mathrm{Fix}(T)$ such that
\[
\mathbb E_\xi[\|T_\xi(p)-T(p)\|^2]\le \sigma_*^2.
\]
Under nonexpansiveness, this single-point condition implies the global transfer estimate
\[
\mathbb E[\|T_\xi(x)-T(x)\|^2] \le 2\sigma_*^2 + 8\|x-p\|^2,
\]
which is enough to recover convergence and oracle-complexity guarantees without uniformly bounded variance [2604.22581].

In stochastic mirror descent, the notion of variance is reformulated geometrically. The new quantity
\[
\sigma^2_{*,\eta}:=\frac{f^*-f_\eta^*}{\eta}\ge0,
\qquad
f_\eta(x)=f(x)-\frac1\eta\,\mathbb E_\xi[D_h(x,x^+(\eta,\xi))],
\]
is finite under $L$-relative smoothness and the existence of minimizers of each $f_\xi$ [2404.12213]. The paper explicitly contrasts this with the Euclidean assumption $\sup_x\mathbb E\|\nabla f_\xi(x)-\nabla f(x)\|^2<\infty$, which can fail even in simple linear regression on unbounded domains.

A central implication of these relaxations is that bounded gradient variance is not a single assumption but a design choice in the analysis. Different geometries and operator classes admit different notions of what must remain bounded.

## 3. Explicit variance bounds for concrete gradient estimators

Estimator-specific analysis replaces abstract boundedness assumptions by explicit inequalities.

For stochastic zeroth-order optimization, the gradient estimator
\[
\nabla f_k^\delta(x):=\frac{n}{2\delta k}\sum_{i=1}^k [f(x+\delta v_i)-f(x-\delta v_i)]v_i,
\]
with $[v_1,\dots,v_k]$ drawn uniformly from $\mathrm{St}(n,k)$, satisfies under $(3,L_3)$-smoothness
\[
\mathrm{Var}[\nabla f_k^\delta(x)]
\le
\Bigl(\frac{n}{k}-1\Bigr)\|\nabla f(x)\|^2
+\frac{L_3\delta^2}{3}\Bigl(\frac{n^2}{k}-n\Bigr)\|\nabla f(x)\|
+\frac{L_3^2n^2\delta^4}{36k}.
\]
The paper attributes the three terms, respectively, to projection of the true gradient onto $k$ random orthonormal axes, a cross-term involving the third-derivative remainder, and the squared third-order remainder. When $k=n$, the two leading terms vanish exactly and only the $O(\delta^4/n)$ term remains [2205.14737].

For reparameterization gradients in black-box variational inference with an affine location-scale family $z=C\varepsilon+m$, if $f$ is $M$-smooth and $z^*$ is any stationary point, then
\[
\mathbb E[\|g\|_2^2]
\le
M^2\Bigl((d+1)\|m-z^*\|_2^2 + (d+\kappa)\|C\|_F^2\Bigr),
\]
where $\kappa=\mathbb E[\varepsilon_i^4]$ [1906.08241]. The same work proves that this dependence on $M$, $d$, and $\kappa$ is unimprovable under only the stated assumptions.

A complementary BBVI analysis shows that Monte Carlo gradients satisfy the $ABC$ form with explicit constants. Under the entropy-regularized ELBO assumptions,
\[
\mathbb E\|g_M(\theta)\|_2^2
\le
\frac{4L_{\rm H}^2}{\mu_{\rm KL}M}C(d,\kappa)\bigl(F(\theta)-F^*\bigr)
+\|\nabla F(\theta)\|_2^2
+\frac{4L_{\rm H}^2}{\mu_{\rm KL}M}C(d,\kappa)\bigl(F^*-f_{\rm KL}^*\bigr),
\]
with
\[
C(d,\kappa)=
\begin{cases}
2\kappa\sqrt d+1,&\text{mean-field},\\
d+\kappa,&\text{Cholesky or square-root}.
\end{cases}
\]
The same paper proves a matching lower bound for the matrix square-root parameterization, showing that the $\Theta(d)$ factor cannot be improved in that class [2303.10472].

In policy gradient estimation for finite-horizon LQR, a single-trajectory REINFORCE estimator has variance proxy
\[
E[\mathrm{Tr}(\hat g^\top \hat g)] \le O(\bar n^4\,C_1^2\,C_2^2),
\]
where
\[
C_1=\|\Sigma_a^{-1}\|^{1/2}\mu H'(\|s_1\|+\sigma H),
\qquad
C_2=r\sigma_a^2H+\mu^2(q+2rk^2)(H'\|s_1\|^2+\sigma^2HH'^2),
\]
and $H'=\min\{H,(1-f)^{-1}\}$ with $f=\rho(A+BK)$ [1910.01249]. The paper also provides a scalar lower bound matching the upper bound in all problem parameters except specific horizon dependencies.

Taken together, these results shift the discussion from postulated boundedness to quantified variance structure. The bounds expose how dimension, smoothness, exploration noise, covariance parameterization, and finite-difference geometry enter the stochastic error.

## 4. Variance reduction, clipping, and estimator selection

A first strategy is structural variance reduction. In zeroth-order methods, sampling $k$ orthonormal directions from the Stiefel manifold rather than using one spherical direction reduces variance from
\[
O(n-1 + (n^2-n)\delta^2 + n^2\delta^4)
\]
for the standard $k=1$ estimator to
\[
O\!\Bigl((n/k-1) + (n^2/k-n)\delta^2 + (n^2\delta^4)/k\Bigr),
\]
with the paper giving practical guidelines such as choosing $k\ge n/(1+\varepsilon)$ to drive the $O(1)$ term below $\varepsilon$ and $\delta\lesssim \sqrt{\varepsilon k/n^2}$ to make the $\delta^2$ term $O(\varepsilon)$ [2205.14737].

A second strategy is estimator switching near singular parameter regimes. For Bernoulli latent-variable models, DisARM is unbiased but has second moment
\[
\mathbb E[\hat g_{\mathrm{DisARM},j}^2]
=
\frac{1}{2\theta_j}\,\mathbb E[(f(z_1^{(j)})-f(z_0^{(j)}))^2]
\]
when $\theta_j<1/2$, so the prefactor $1/(2\theta_j)$ blows up as $\theta_j\to0$ [2208.06124]. The bitflip-1 estimator is also unbiased and satisfies
\[
\mathbb E[\hat g_{\mathrm{bitflip\text{-}1},j}^2]
=
K\,\mathbb E[(f(\tilde z^{(j)})-f(z))^2],
\]
hence its variance does not depend on $\theta_j$ and is $O(K)$ [2208.06124]. The unbiased gradient variance clipping estimator
\[
\hat g_{\mathrm{UGC},j}
=
\begin{cases}
\hat g_{\mathrm{bitflip\text{-}1},j},& \min(\theta_j,1-\theta_j)<\tau,\\
\hat g_{\mathrm{DisARM},j},& \min(\theta_j,1-\theta_j)\ge\tau,
\end{cases}
\]
with canonical choice $\tau=1/(2K)$, is proved to have uniformly lower variance than DisARM when $\tau\le 1/(2K)$ [2208.06124].

A third strategy is clipping under heavy-tailed first-order noise. Coordinate-wise clipping at level $\tau$ is defined by
\[
G_{i,\tau}(x;\xi)=\operatorname{sign}(G_i(x;\xi))\min\{|G_i(x;\xi)|,\tau\}.
\]
The clipped bias and variance are quantified as
\[
B(\tau)\le \Bigl|\int_{-\tau}^{\tau}u\,p(u)\,du\Bigr|
+\Bigl|\tau\int_{\tau}^\infty[p(u)-p(-u)]\,du\Bigr|
+O(\tau^{-\alpha}),
\]
and
\[
V(\tau)\le O((\tau+|a|)^{2-\alpha}) + O(\tau^2/(\tau-|a|)^\alpha).
\]
Under the stronger tail-symmetry condition, one may choose $\tau=O(\varepsilon^{-1/\alpha})$ and obtain $\sigma^2(\tau)=O(\varepsilon^{-(2-\alpha)/\alpha})$ [2512.14686].

These constructions show that bounded variance can be induced rather than assumed. Orthogonalization, estimator selection, and clipping all modify the estimator so that the dominant source of variance becomes analyzable and, in some regimes, uniformly controlled.

## 5. Consequences for convergence rates and oracle complexity

Once second moments are controlled, either uniformly or through a weaker surrogate, the resulting optimization rates become explicit.

For perturbed proximal gradient with constant step size $1/L$, the deterministic linear-plus-error inequality
\[
\pi_k-\rho^k\pi_0 \le \frac1\theta \sum_{i=0}^{k-1}\rho^{k-1-i}\|e_i\|^2
\]
connects suboptimality directly to accumulated squared gradient errors [1304.5586]. If the variance schedule decays geometrically, $\mathbb E[\|e_k\|^2]\le \lambda\beta^k$, then
\[
\pi_k-\rho^k\pi_0 = O((\max\{\rho,\beta\})^k)
\]
in expectation, and under sub-Gaussian tails the exceedance probability satisfies
\[
\Pr(\pi_k-\rho^k\pi_0\ge \varepsilon)=O\exp[-c\,\varepsilon/(\max\{\rho,\beta\})^k]
\]
for some $c>0$ [1304.5586].

In pseudo-monotone stochastic variational inequalities, extragradient with iterative variance reduction achieves an $O(1/K)$ rate in the mean-squared natural residual and oracle complexity
\[
O\bigl(\epsilon^{-2}\,\ln(\epsilon^{-1})^{1+b}\bigr),
\]
that is, optimal $\epsilon^{-2}$ up to a logarithm, while allowing an unbounded feasible set, an unbounded operator, and non-uniform variance of the oracle [1703.00260].

For constrained stochastic variational inequalities under the unbounded-variance assumption $(UV)$, three different algorithms—FBF with large mini-batches, inexact Krasnosel'skii-Mann with MLMC, and variance-reduced FBF with Halpern anchoring—each attain
\[
\widetilde O(\varepsilon^{-4})
\]
stochastic first-order oracle calls to make the expected residual norm less than $\varepsilon$ [2602.05531]. The same complexity is shown for both monotone VIs and structured nonmonotone VIs satisfying the $\rho$-weak-Minty condition, with explicit admissible thresholds such as $\rho<1/(12L)$ for mini-batch FBF and $\rho<1/L$ for the MLMC scheme [2602.05531].

In stochastic Krasnosel'skii-Mann iterations, the weaker variance-at-solution assumption still yields
\[
\mathbb E[\|Tx_{N_K}-x_{N_K}\|]=O\!\Bigl(\frac{1}{\sqrt{\sum \lambda_k(1-\lambda_k)}}\Bigr),
\]
almost sure decay of the running minimum residual, and weak convergence of the full sequence when $\lambda_k\in \ell^2\setminus \ell^1$ [2604.22581]. The same paper states that the best-known stochastic oracle complexity is recovered without imposing uniformly bounded variance [2604.22581].

In geometry-aware stochastic mirror descent, the variance quantity $\sigma^2_{*,\eta}$ enters directly into the finite-time bounds. For relatively strongly convex $f$,
\[
\eta\,[\mathbb E f_\eta(x_T)-f_\eta^*]+\mathbb E D_h(x^*,x_{T+1})
\le
(1-\eta\mu)^{T+1}D_h(x^*,x_0)+\frac{\eta}{\mu}\sigma^2_{*,\eta},
\]
while in the convex case
\[
\frac1{T+1}\sum_{k=0}^T \bigl[\mathbb E(f_\eta(x_k)-f_\eta^*)+D_f(x^*,x_k)\bigr]
\le
\frac{D_h(x^*,x_0)}{\eta(T+1)}+\eta\sigma^2_{*,\eta}.
\]
These results recover linear convergence up to an $O(\sigma^2_{*,\eta}/\mu)$ bias in the strongly convex case and the familiar $O(1/T)$-plus-noise trade-off in the convex case [2404.12213].

The practical significance is that convergence proofs no longer hinge solely on a global constant $\sigma^2$. They can be driven by vanishing variance schedules, controlled growth, solution-local second moments, or corrected objective surrogates.

## 6. Conceptual scope, misconceptions, and adjacent notions

A common misconception is that stochastic optimization requires uniformly bounded gradient variance everywhere in the domain. The recent variational-inequality and fixed-point literature shows otherwise: optimal or best-known complexity guarantees can survive under $(UV)$ growth conditions or even under finite variance at a single solution [2602.05531] [2604.22581]. A plausible implication is that the analytic burden often shifts from proving global boundedness to proving that iterates remain sufficiently stable for the weaker noise model to remain effective.

A second misconception is that variance is always a Euclidean second moment of oracle noise. In relative smoothness, the quantity $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$ is not a direct variance of $\nabla f_\xi(x)-\nabla f(x)$, yet it plays the same role in convergence theory and has the small-step limit
\[
\lim_{\eta\to0}\sigma^2_{*,\eta}
=
\frac12\,\mathbb E\|\nabla f_\xi(x^*)\|^2_{\nabla^2 h(x^*)^{-1}}
\]
[2404.12213]. In BBVI, the $ABC$ condition similarly replaces a constant-variance postulate by a growth inequality involving function suboptimality and gradient norm [2303.10472].

A third adjacent notion measures not oracle noise but gradient informativeness across tasks. For a hypothesis class $\mathcal H$ with target sampled from $\chi$, Takhanov studies
\[
\mathrm{Var}_{h\sim\chi}[\nabla_w C_h(w)]
\]
and shows the asymptotic bound
\[
\widetilde O(\varepsilon + e^{-\frac12 \mathcal E_c}),
\]
where $\varepsilon$ measures pairwise independence of the target class and $\mathcal E_c$ is the collision entropy of the input distribution [2505.22158]. This is conceptually distinct from stochastic-gradient noise variance: low variance here means the gradient contains little information about which target function generated the data.

Open questions remain explicit in the current literature. For stochastic variational inequalities without bounded variance, one stated direction is pushing $\rho$ up to the conjectured limit $1/L$ in a single-loop algorithm without large batches; another is lowering the $\varepsilon^{-4}$ dependence under additional structure such as smoothness or interpolation [2602.05531]. For stochastic Krasnosel'skii-Mann iterations, the analysis notes that extensions to weaker noise models or to nonexpansive-plus-error-growth settings would require new ideas [2604.22581].

Bounded gradient variance is therefore best understood as a spectrum of analytic controls on stochastic gradients and gradient-like objects. In some settings it is assumed globally, in others derived locally, transferred geometrically, or enforced algorithmically; and in each case, the precise form of the bound determines the attainable convergence theory.

Source: https://www.emergentmind.com/topics/bounded-gradient-variance