Accelerated ZO-SVRG Methods

• The paper’s main contribution is the development of accelerated ZO-SVRG variants that reduce query complexity while enhancing convergence rates.
• These methods employ refined gradient estimators such as coordinate-wise and random-direction averaging to balance accuracy and computational cost.
• Empirical results show significant efficiency gains in memory-limited, large-scale tasks like fine-tuning language models and black-box adversarial training.

Zeroth-order stochastic variance-reduced gradient methods (ZO-SVRG) constitute a prominent class of algorithms designed for stochastic optimization when gradient access is unavailable, but function evaluations are permitted. Accelerated variants of ZO-SVRG deploy refined gradient estimators and improved variance-control strategies, enabling faster convergence—both theoretically and empirically—while reducing function query complexity. Such algorithms are of increasing importance in large-scale learning, black-box model tuning, and regimes where backpropagation is prohibitively expensive or unsupported, particularly in fine-tuning LLMs, black-box adversarial applications, and chemistry/material tasks.

1. Background: Zeroth-Order SVRG and Acceleration Principles

The classic ZO-SVRG paradigm (Liu et al., 2018) replaces first-order gradients in SVRG [Stochastic Variance-Reduced Gradient] with finite-difference or random-direction gradient approximations, leveraging only function values. “Plain” ZO-SVRG employs a two-point random gradient estimator per sample: $$\widehat\nabla f_i(x)=\frac{d}{\mu}[f_i(x+\mu u_i)-f_i(x)]u_i,\quad u_i\sim\mathrm{Uniform}(\mathbb S^{d-1})$$ where $\mu$ is a smoothing parameter and $u_i$ is a random direction. The SVRG update structure is preserved: periodic full gradient (“snapshot”) estimation alternates with inner-loop variance-reduced updates using mini-batches.

All accelerated variants target the statistical and computational bottleneck in ZO-SVRG’s error bound: an $O(1/b)$ bias term (with $b$ the mini-batch size), introduced by the high variance of the naive two-point estimator. Averaging over directions or switching to coordinate-based estimators meaningfully mediates this issue. The most prominent acceleration approaches are:

• Random-direction averaging (ZO-SVRG-Ave)
• Coordinate-wise finite-difference estimators (ZO-SVRG-Coord)
• Seeded, data-parallel SPSA with snapshot-driven variance reduction (e.g., MeZO-SVRG (Gautam et al., 2024))
• Mixed coordinate/random estimators (ZO-SVRG-Coord-Rand (Ji et al., 2019))

2. Core Algorithms and Their Update Rules

Reference ZO-SVRG Skeleton

All ZO-SVRG-type algorithms use a common SVRG structure:

• Epochs: periodic “snapshot” at reference $\tilde x$, estimate anchored full (zo-)gradient.
• Inner loop: update $x_{k+1} = x_k - \eta v_k$, where $v_k$ is a variance-reduced ZO estimator.

Accelerated Variants

Variant	Estimator Type	Update Equation (per sample $f_i$)	Query Complexity per Estim.
ZO-SVRG (plain)	2-point random	$\frac{d}{\mu}[f_i(x+\mu u_i)-f_i(x)]u_i$	$1$
ZO-SVRG-Ave	$q$ random avg.	$$\frac{d}{\mu q}\sum_{j=1}^q [f_i(x+\mu u_j)-f_i(x)]u_j$$	$q$
ZO-SVRG-Coord	coordinate-wise	$$\sum_{\ell=1}^d \frac{f_i(x+\mu e_\ell)-f_i(x-\mu e_\ell)}{2\mu} e_\ell$$	$d$
ZO-SVRG-Coord-Rand	coord. ref. / rand inner	Reference: coord; inner: random 2-point (Ji et al., 2019)	coord/ref: $d$ or $q$
MeZO-SVRG	batchwise SPSA + seed	Shared $z_t\sim N(0,I_d)$ for each batch; 2-point SPSA estimation per batch (Gautam et al., 2024)	$2b$ (minibatch size $b$)

ZO-SVRG-Ave and ZO-SVRG-Coord reduce variance at increased per-gradient query cost. MeZO-SVRG achieves acceleration through variance-reduction, shared noise direction, and minimal memory overhead.

3. Variance Reduction Mechanisms and Estimator Analysis

Variance reduction in these methods hinges on a “control variate”: $$\Delta = v_{\text{current}} - v_{\text{ref}} + g_{\text{ref}}$$ where $g_{\text{ref}}$ is the full-batch ZO estimator at the snapshot point, $v_{\text{current}}$ the minibatch ZO estimator at the current iterate, and $v_{\text{ref}}$ the minibatch estimator at the snapshot. This structure ensures: $$\mathbb{E}[\Delta\,|\,x,x_\text{ref}] = \nabla f(x)$$ with variance substantially smaller than complete independence.

In ZO-SVRG-Ave, averaging $q$ direction vectors per estimate reduces the offending $O(d)$ variance blowup to $O\bigl((d+q)/q\bigr)$, replacing $O(1/b)$ error with $O(1/(b\min\{d,q\}))$. In ZO-SVRG-Coord, deterministic coordinate estimators eliminate this variance source and “extra error” entirely.

MeZO-SVRG applies variance reduction using a shared perturbation vector $z_t$ for the entire batch, attaining the benefits of data-parallel SPSA and the SVRG control variate (Gautam et al., 2024).

4. Theoretical Convergence Guarantees

Accelerated ZO-SVRG variants achieve significantly improved convergence rates compared to ZO-SGD and classic zeroth-order methods. Key theoretical results include:

• ZO-SVRG (plain):

$$\mathbb{E}\|\nabla f(\bar x)\|^2 \le O\left(\frac{d}{T} + \frac{1}{b}\right)$$

Query complexity: $O(nS + bT)$ ($S$ epochs, $T$ steps) (Liu et al., 2018).

• ZO-SVRG-Coord:

$$\mathbb{E}\|\nabla f(\bar x)\|^2 \le O\left(\frac{d}{T}\right)$$

with no $O(1/b)$ term; query complexity $O\bigl(d(nS+bT)\bigr)$ (Liu et al., 2018, Ji et al., 2019).

• ZO-SVRG-Ave:

$$\mathbb{E}\|\nabla f(\bar x)\|^2 \le O\left(\frac{d}{T} + \frac{1}{b\min\{d,q\}}\right)$$

at $q$-times higher query cost per estimator (Liu et al., 2018).

• MeZO-SVRG:

$$\mathbb{E}\|\nabla f(\theta_T)\|^2 \le \frac{f(\theta_0)-f^*}{T\eta}+C_1L\mu^2 d + C_2 \frac{\eta^2 d\sigma^2}{bq}$$

Yields $\epsilon$-stationarity in $O\left(\frac{d}{b\epsilon^2}\right)$ iterations, strictly outperforming ZO-SGD's $O\left(\frac{d}{\epsilon^4}\right)$ (Gautam et al., 2024).

Further, (Ji et al., 2019) establishes for ZO-SVRG-Coord(-Rand) a query complexity of $O(\min\{n^{2/3}d\epsilon^{-1},d\epsilon^{-5/3}\})$, improving on ZO-GD and ZO-SGD in all settings with $n>1$.

5. Practical Implementation: Memory, Hyperparameters, and Use Cases

Accelerated ZO-SVRG methods are particularly suited to applications with tight memory budgets and expensive backpropagation. Key implementation details from (Gautam et al., 2024):

• Memory Footprint: MeZO-SVRG requires only one extra copy of parameters and reference gradient; overhead is $O(d)$ and remains constant in batch size $b$.
  • For large autoregressive models, MeZO-SVRG achieves $\sim 2\times$ GPU memory savings relative to FO-SGD (e.g., 19GB vs. 38GB for GPT2-XL).
  • For large batch sizes, MeZO-SVRG reduces memory 3–4$\times$ versus FO-SGD (e.g., 4.7GB vs. 18.6GB for RoBERTa-large at $b=64$).
• Computation Cost: MeZO-SVRG reaches comparable or higher test accuracy than ZO-SGD (MeZO) in half the GPU-hours or less.
• Hyperparameters: Best empirical performance with perturbation $\mu\approx 10^{-3}$; two learning rates $\eta_1 > \eta_2$, typically $\eta_1 = 10\times\eta_2$; batch sizes $b=32$ or $64$; snapshot frequency $q=2$ or $5$.
• Applications: Large-scale LLM fine-tuning, black-box adversarial training, hyperparameter tuning, settings where only function value access exists, and memory-limited optimization.

6. Comparative Summary and Trade-offs

Accelerated ZO-SVRG variants offer a flexible spectrum of variance reduction and function query cost trade-offs, as summarized below:

Algorithm	Query Complexity (for $\epsilon$-stationarity)	Pros	Cons
ZO-SVRG (plain)	$O(n/\epsilon + d/\epsilon^2)$	Minimal queries per step	Slower convergence, $O(1/b)$ error
ZO-SVRG-Ave	$O(q(n/\epsilon + d/\epsilon^2))$	Lower variance, smaller $O(1/b)$ error	Higher function query cost
ZO-SVRG-Coord	$$O(\min\{n^{2/3}d\,\epsilon^{-1},d\epsilon^{-5/3}\})$$	No $O(1/b)$ error, matches best-known rate	$d$-fold increase in functions
MeZO-SVRG	$O(d/(b\epsilon^2))$	Low memory overhead, fast convergence	Not fully coordinate-wise

Selection among these variants depends on the allowable computational resources (specifically, number of function evaluations per iteration) and the degree of variance reduction required for the application at hand.

7. Empirical Performance and Future Directions

Empirical results from (Gautam et al., 2024) demonstrate that MeZO-SVRG outperforms basic MeZO (plain ZO-SGD) with up to 20 test-accuracy point gains across multiple LLMs and standard GLUE tasks. GPU-hours to target accuracy are halved on large LMs, and memory savings are up to $4\times$ compared to FO-SGD. The methods close the convergence gap with first-order SGD/Adam in the large batch, large parameter regimes, and are especially robust for non-prompted fine-tuning scenarios.

Recent theoretical advances (Ji et al., 2019) further suggest continued improvements via tighter analysis and hybrid estimators, enabling constant stepsizes and improving query complexity beyond all earlier ZO-GD/SGD variants. A plausible implication is expanding deployment in large, black-box, or memory-challenged environments, especially where function-only or low-level API access is available.

• "Variance-reduced Zeroth-Order Methods for Fine-Tuning LLMs" (Gautam et al., 2024)
• "Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization" (Liu et al., 2018)
• "Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization" (Ji et al., 2019)