Bounded Gradient Variance in Optimization

• Bounded gradient variance refers to assumptions that control the second moment of stochastic gradients, ensuring that gradient errors remain tractable.
• Recent work generalizes these bounds using non-uniform, geometry-dependent, or solution-local conditions to allow variance growth while retaining convergence guarantees.
• Estimator-specific analyses yield explicit variance bounds for methods like zeroth-order optimization, variational inference, and REINFORCE, driving advances in variance reduction and clipping techniques.

Bounded gradient variance denotes a family of assumptions and estimator-specific guarantees that control the second moment of stochastic gradient-like quantities. In the classical stochastic-approximation formulation, this control appears as a conditional bound on the gradient error, such as $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2$$, while in stochastic variational inequalities it appears as a uniform oracle bound $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$ (Friedlander et al., 2013, Alacaoglu et al., 5 Feb 2026). Recent work has both generalized and localized this notion: some analyses allow variance growth proportional to the squared norm of the iterate, some require finite variance only at a solution, and some replace Euclidean second moments by geometry-dependent surrogates such as $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$ in stochastic mirror descent (Cortild et al., 24 Apr 2026, Hendrikx, 2024). In parallel, estimator-level analyses now provide explicit variance bounds for zeroth-order finite differences, reparameterization gradients in variational inference, REINFORCE in linear-quadratic control, and clipped first-order methods under heavy-tailed noise (Feng et al., 2022, Domke, 2019, Preiss et al., 2019, He, 16 Dec 2025).

1. Classical bounded-variance formulations

The classical bounded-variance model in stochastic optimization is a second-moment condition on the stochastic error. In perturbed proximal gradient, the error sequence $e_k$ is adapted to the filtration $\mathcal F_k=\sigma(e_1,\dots,e_k)$ and satisfies

$$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$

with $\sigma_k^2\to0$ often realized by increasing sample sizes (Friedlander et al., 2013). Stronger assumptions in the same framework include coordinate-wise sub-Gaussian tails and an almost-sure deterministic bound $\|e_k\|\le U_k$, with the paper explicitly noting the implication chain $C\Rightarrow B\Rightarrow A$ (Friedlander et al., 2013).

In stochastic variational inequalities, the standard assumption is the uniform bounded-variance condition

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$

for an unbiased oracle $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$0 (Alacaoglu et al., 5 Feb 2026). This is the form explicitly labeled $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$1 in the literature summarized there. It is a global condition, uniform over the domain, and is commonly paired with Lipschitz continuity of the operator $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$2 and convexity of the regularizer $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$3 (Alacaoglu et al., 5 Feb 2026).

A related but more algorithm-facing formulation appears in black-box variational Bayesian inference through the $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$4 condition: $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$5 This does not postulate a constant variance bound directly; instead it controls the expected squared norm of the stochastic gradient by a linear combination of function suboptimality, gradient norm, and a residual constant (Kim et al., 2023). The paper shows that black-box variational inference gradients satisfy this structure under smoothness and quadratic-growth assumptions.

These formulations share a common role: they convert stochasticity into a quantitatively tractable perturbation term. What differs is whether the control is uniform in the domain, local to a solution, dependent on geometry, or tied to objective suboptimality.

2. Relaxations beyond uniform boundedness

Several recent analyses replace global bounded variance by weaker assumptions that still permit convergence proofs.

Setting	Variance condition	Source
Classical SVI	$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$6	(Alacaoglu et al., 5 Feb 2026)
Unbounded-variance SVI	$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$7	(Alacaoglu et al., 5 Feb 2026)
Pseudo-monotone SVI	$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$8	(Iusem et al., 2017)
Stochastic KM	$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$9 with $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$0	(Cortild et al., 24 Apr 2026)
Stochastic mirror descent	$\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$1	(Hendrikx, 2024)

The unbounded-variance condition

$\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$2

is motivated by the observation that the classical bounded-variance condition fails, for example, for bilinear min-max on $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$3, whereas the relaxed condition holds for stochastic oracles whose variance can grow as fast as the squared norm of the optimization variable (Alacaoglu et al., 5 Feb 2026). This changes the problem from one of proving uniform noise control to one of proving that iterates remain sufficiently controlled in expectation.

An earlier pseudo-monotone SVI analysis already used a non-uniform variance model: $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$4 The associated empirical-average errors $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$5 then satisfy conditional second-moment bounds proportional to $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$6, making variance reduction a matter of growing the sample size sequence $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$7 (Iusem et al., 2017).

A further relaxation appears in stochastic Krasnosel'skii-Mann iterations. Instead of requiring a uniform variance bound on $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$8 for all $\sigma^2_{*,\eta}=(f^*-f_\eta^*)/\eta$9, the assumption is only that there exists one fixed point $e_k$0 such that

$e_k$1

Under nonexpansiveness, this single-point condition implies the global transfer estimate

$e_k$2

which is enough to recover convergence and oracle-complexity guarantees without uniformly bounded variance (Cortild et al., 24 Apr 2026).

In stochastic mirror descent, the notion of variance is reformulated geometrically. The new quantity

$e_k$3

is finite under $e_k$4-relative smoothness and the existence of minimizers of each $e_k$5 (Hendrikx, 2024). The paper explicitly contrasts this with the Euclidean assumption $e_k$6, which can fail even in simple linear regression on unbounded domains.

A central implication of these relaxations is that bounded gradient variance is not a single assumption but a design choice in the analysis. Different geometries and operator classes admit different notions of what must remain bounded.

3. Explicit variance bounds for concrete gradient estimators

Estimator-specific analysis replaces abstract boundedness assumptions by explicit inequalities.

For stochastic zeroth-order optimization, the gradient estimator

$e_k$7

with $e_k$8 drawn uniformly from $e_k$9, satisfies under $\mathcal F_k=\sigma(e_1,\dots,e_k)$0-smoothness

$\mathcal F_k=\sigma(e_1,\dots,e_k)$1

The paper attributes the three terms, respectively, to projection of the true gradient onto $\mathcal F_k=\sigma(e_1,\dots,e_k)$2 random orthonormal axes, a cross-term involving the third-derivative remainder, and the squared third-order remainder. When $\mathcal F_k=\sigma(e_1,\dots,e_k)$3, the two leading terms vanish exactly and only the $\mathcal F_k=\sigma(e_1,\dots,e_k)$4 term remains (Feng et al., 2022).

For reparameterization gradients in black-box variational inference with an affine location-scale family $\mathcal F_k=\sigma(e_1,\dots,e_k)$5, if $\mathcal F_k=\sigma(e_1,\dots,e_k)$6 is $\mathcal F_k=\sigma(e_1,\dots,e_k)$7-smooth and $\mathcal F_k=\sigma(e_1,\dots,e_k)$8 is any stationary point, then

$\mathcal F_k=\sigma(e_1,\dots,e_k)$9

where $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$0 (Domke, 2019). The same work proves that this dependence on $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$1, $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$2, and $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$3 is unimprovable under only the stated assumptions.

A complementary BBVI analysis shows that Monte Carlo gradients satisfy the $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$4 form with explicit constants. Under the entropy-regularized ELBO assumptions,

$$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$5

with

$$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$6

The same paper proves a matching lower bound for the matrix square-root parameterization, showing that the $$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$7 factor cannot be improved in that class (Kim et al., 2023).

In policy gradient estimation for finite-horizon LQR, a single-trajectory REINFORCE estimator has variance proxy

$$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$8

where

$$\mathbb E[\|e_k\|^2\mid \mathcal F_{k-1}] \le \sigma_k^2,$$9

and $\sigma_k^2\to0$0 with $\sigma_k^2\to0$1 (Preiss et al., 2019). The paper also provides a scalar lower bound matching the upper bound in all problem parameters except specific horizon dependencies.

Taken together, these results shift the discussion from postulated boundedness to quantified variance structure. The bounds expose how dimension, smoothness, exploration noise, covariance parameterization, and finite-difference geometry enter the stochastic error.

4. Variance reduction, clipping, and estimator selection

A first strategy is structural variance reduction. In zeroth-order methods, sampling $\sigma_k^2\to0$2 orthonormal directions from the Stiefel manifold rather than using one spherical direction reduces variance from

$\sigma_k^2\to0$3

for the standard $\sigma_k^2\to0$4 estimator to

$\sigma_k^2\to0$5

with the paper giving practical guidelines such as choosing $\sigma_k^2\to0$6 to drive the $\sigma_k^2\to0$7 term below $\sigma_k^2\to0$8 and $\sigma_k^2\to0$9 to make the $\|e_k\|\le U_k$0 term $\|e_k\|\le U_k$1 (Feng et al., 2022).

A second strategy is estimator switching near singular parameter regimes. For Bernoulli latent-variable models, DisARM is unbiased but has second moment

$\|e_k\|\le U_k$2

when $\|e_k\|\le U_k$3, so the prefactor $\|e_k\|\le U_k$4 blows up as $\|e_k\|\le U_k$5 (Kunes et al., 2022). The bitflip-1 estimator is also unbiased and satisfies

$\|e_k\|\le U_k$6

hence its variance does not depend on $\|e_k\|\le U_k$7 and is $\|e_k\|\le U_k$8 (Kunes et al., 2022). The unbiased gradient variance clipping estimator

$\|e_k\|\le U_k$9

with canonical choice $C\Rightarrow B\Rightarrow A$0, is proved to have uniformly lower variance than DisARM when $C\Rightarrow B\Rightarrow A$1 (Kunes et al., 2022).

A third strategy is clipping under heavy-tailed first-order noise. Coordinate-wise clipping at level $C\Rightarrow B\Rightarrow A$2 is defined by

$C\Rightarrow B\Rightarrow A$3

The clipped bias and variance are quantified as

$C\Rightarrow B\Rightarrow A$4

and

$C\Rightarrow B\Rightarrow A$5

Under the stronger tail-symmetry condition, one may choose $C\Rightarrow B\Rightarrow A$6 and obtain $C\Rightarrow B\Rightarrow A$7 (He, 16 Dec 2025).

These constructions show that bounded variance can be induced rather than assumed. Orthogonalization, estimator selection, and clipping all modify the estimator so that the dominant source of variance becomes analyzable and, in some regimes, uniformly controlled.

5. Consequences for convergence rates and oracle complexity

Once second moments are controlled, either uniformly or through a weaker surrogate, the resulting optimization rates become explicit.

For perturbed proximal gradient with constant step size $C\Rightarrow B\Rightarrow A$8, the deterministic linear-plus-error inequality

$C\Rightarrow B\Rightarrow A$9

connects suboptimality directly to accumulated squared gradient errors (Friedlander et al., 2013). If the variance schedule decays geometrically, $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$0, then

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$1

in expectation, and under sub-Gaussian tails the exceedance probability satisfies

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$2

for some $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$3 (Friedlander et al., 2013).

In pseudo-monotone stochastic variational inequalities, extragradient with iterative variance reduction achieves an $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$4 rate in the mean-squared natural residual and oracle complexity

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$5

that is, optimal $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$6 up to a logarithm, while allowing an unbounded feasible set, an unbounded operator, and non-uniform variance of the oracle (Iusem et al., 2017).

For constrained stochastic variational inequalities under the unbounded-variance assumption $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$7, three different algorithms—FBF with large mini-batches, inexact Krasnosel'skii-Mann with MLMC, and variance-reduced FBF with Halpern anchoring—each attain

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$8

stochastic first-order oracle calls to make the expected residual norm less than $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2,$9 (Alacaoglu et al., 5 Feb 2026). The same complexity is shown for both monotone VIs and structured nonmonotone VIs satisfying the $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$00-weak-Minty condition, with explicit admissible thresholds such as $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$01 for mini-batch FBF and $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$02 for the MLMC scheme (Alacaoglu et al., 5 Feb 2026).

In stochastic Krasnosel'skii-Mann iterations, the weaker variance-at-solution assumption still yields

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$03

almost sure decay of the running minimum residual, and weak convergence of the full sequence when $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$04 (Cortild et al., 24 Apr 2026). The same paper states that the best-known stochastic oracle complexity is recovered without imposing uniformly bounded variance (Cortild et al., 24 Apr 2026).

In geometry-aware stochastic mirror descent, the variance quantity $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$05 enters directly into the finite-time bounds. For relatively strongly convex $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$06,

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$07

while in the convex case

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$08

These results recover linear convergence up to an $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$09 bias in the strongly convex case and the familiar $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$10-plus-noise trade-off in the convex case (Hendrikx, 2024).

The practical significance is that convergence proofs no longer hinge solely on a global constant $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$11. They can be driven by vanishing variance schedules, controlled growth, solution-local second moments, or corrected objective surrogates.

6. Conceptual scope, misconceptions, and adjacent notions

A common misconception is that stochastic optimization requires uniformly bounded gradient variance everywhere in the domain. The recent variational-inequality and fixed-point literature shows otherwise: optimal or best-known complexity guarantees can survive under $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$12 growth conditions or even under finite variance at a single solution (Alacaoglu et al., 5 Feb 2026, Cortild et al., 24 Apr 2026). A plausible implication is that the analytic burden often shifts from proving global boundedness to proving that iterates remain sufficiently stable for the weaker noise model to remain effective.

A second misconception is that variance is always a Euclidean second moment of oracle noise. In relative smoothness, the quantity $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$13 is not a direct variance of $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$14, yet it plays the same role in convergence theory and has the small-step limit

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$15

(Hendrikx, 2024). In BBVI, the $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$16 condition similarly replaces a constant-variance postulate by a growth inequality involving function suboptimality and gradient norm (Kim et al., 2023).

A third adjacent notion measures not oracle noise but gradient informativeness across tasks. For a hypothesis class $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$17 with target sampled from $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$18, Takhanov studies

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$19

and shows the asymptotic bound

$\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$20

where $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$21 measures pairwise independence of the target class and $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$22 is the collision entropy of the input distribution (Takhanov, 28 May 2025). This is conceptually distinct from stochastic-gradient noise variance: low variance here means the gradient contains little information about which target function generated the data.

Open questions remain explicit in the current literature. For stochastic variational inequalities without bounded variance, one stated direction is pushing $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$23 up to the conjectured limit $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$24 in a single-loop algorithm without large batches; another is lowering the $\mathbb E\|G(z,\xi)-G(z)\|^2 \le \sigma^2$25 dependence under additional structure such as smoothness or interpolation (Alacaoglu et al., 5 Feb 2026). For stochastic Krasnosel'skii-Mann iterations, the analysis notes that extensions to weaker noise models or to nonexpansive-plus-error-growth settings would require new ideas (Cortild et al., 24 Apr 2026).

Bounded gradient variance is therefore best understood as a spectrum of analytic controls on stochastic gradients and gradient-like objects. In some settings it is assumed globally, in others derived locally, transferred geometrically, or enforced algorithmically; and in each case, the precise form of the bound determines the attainable convergence theory.