---
title: 'Zeroth Order Optimization: Techniques and Applications'
url: https://www.emergentmind.com/topics/zeroth-order-optimization-zoo-a05557ca-52dc-48e6-9f4b-fcb06c7992f7
type: topic
---

# Zeroth Order Optimization: Techniques and Applications

Zeroth Order Optimization (ZOO) refers to a class of algorithms for optimizing functions when analytic gradients are unavailable, unreliable, or expensive to compute, and only function evaluations (potentially noisy) are accessible. ZOO methods estimate gradient information via finite differences or surrogate constructions, enabling the application of optimization principles to black-box, non-differentiable, or simulation-based objectives. These methods underpin applications ranging from black-box adversarial machine learning, simulation-based reinforcement learning, memory-efficient deep network training, and distributed multi-agent optimization, to high-dimensional hyperparameter tuning.

## 1. Foundations and Core Methodology

The canonical ZOO paradigm addresses the unconstrained or constrained minimization of $f(x):\mathbb{R}^d\to\mathbb{R}$:
$$
\min_{x\in X} f(x)
$$
where $X$ is typically convex but may be unconstrained, and only queries of $f(x)$ are permitted [2006.06224].

ZOO methods approximate gradient information by evaluating $f$ along various directions. The two most common finite-difference estimators are:

- **One-point random-direction estimator**:
  $$
  \hat{g}_1(x) = \frac{\phi(d)}{\mu}f(x+\mu u)u
  $$
  where $u$ is sampled uniformly from the unit sphere or Gaussian, $\mu>0$ is a smoothing parameter, and $\phi(d)=d$ (sphere) or $1$ (Gaussian).
- **Two-point random-direction estimator**:
  $$
  \hat{g}_2(x) = \frac{\phi(d)}{\mu}\big( f(x+\mu u) - f(x) \big)u
  $$
  This estimator, which is unbiased for the (Gaussian-)smoothed gradient $\nabla f_\mu(x)$, exhibits lower variance and is standard in modern ZOO [2006.06224, 2602.01627].

For high-dimensional problems, coordinate-wise finite-differences or block-wise sampling are also standard to reduce query cost [1708.03999, 2510.19165].

Key theoretical properties:
- **Bias**: $\mathbb{E}\,\hat g_2(x) = \nabla f_\mu(x)$, where $f_\mu(x)=\mathbb{E}_u[f(x+\mu u)]$.
- **Variance**: For $L$-smooth $f$, $\mathbb{E} \|\hat g_2(x)-\nabla f(x)\|^2 = O(d\|\nabla f(x)\|^2 + \mu^2 d^3)$ [2006.06224, 2506.05454].

Standard ZOO schemes lift any first-order method by substituting analytical gradients with ZO estimators, including SGD, Adam, or coordinate descent [2012.11518, 2502.01014].

## 2. Query Complexity, Dimension Dependence, and Regularization

The convergence behavior and query complexity of ZOO methods are strongly affected by the estimator variance and dimension $d$.

- **Classical rates**: ZO-SGD with the two-point estimator provides $O(d/\sqrt{T})$ convergence in nonconvex settings and $O(d/T)$ in the strongly convex case [2006.06224, 2510.19165].
- **Dimension dependence**: The convergence rates and query complexity scale linearly with $d$ unless further structure is exploited.

Recent theoretical advances replace the worst-case $O(d)$ dependence by more refined spectral or effective-dimension metrics [2307.05753, 2510.10945]:
- **Effective-dimension (ED$_\alpha$)**: 
  $$
  ED_\alpha = \sup_{x} \sum_{i=1}^d \sigma_i^\alpha(\nabla^2 f(x))
  $$
  whereby fast Hessian decay (e.g., in shallow nets or certain ML objectives) enables logarithmic or even dimension-independent rates [2307.05753, 2510.10945].

- **Implicit regularization and flat minima**: ZOO with two-point estimators exhibits implicit bias towards flat minima, minimizing the trace of the Hessian among global optima [2506.05454]. This is formalized as:
  $$
  f_\mu(x) \approx f(x) + \tfrac12 \mu^2 \mathrm{Tr}[\nabla^2 f(x)]
  $$
  so that stochastic optimization with ZOO provably favors solutions with reduced sharpness (lower Hessian trace), a critical property for generalization in deep learning.

## 3. High-Dimensional, Structured, and Adaptive ZOO

### Subspace and Block Perturbations

High dimensionality presents a practical bottleneck in ZOO (variance $\sim O(d)$). Structured perturbations reduce this scaling:

- **Subspace perturbations**: Restricting perturbations to sparse, low-rank, or block-coordinated subspaces of stable rank $s \ll d$ reduces the variance to $O(s)$ [2501.19099]. The overall convergence rate becomes $O(d^2 / (s T))$, and in the case of good subspace alignment with the objective's Hessian, dimension-independent rates are attainable.

- **Block coordinate and structured descent**: In large-scale settings (e.g., LLM fine-tuning), block coordinate ZOO (e.g., MeZO-BCD) updates single architectural blocks, enabling both computational efficiency and wall-clock speedups (up to $2.77\times$ over standard MeZO on OPT-13B) [2501.19099].

- **Compressed sensing for sparse gradients**: For objectives with $s$-sparse gradients, new estimators such as GraCe minimize the required function queries to $O(s \log \log(d/s))$ per step, the first double-logarithmic bound, outperforming previous ZORO methods ($O(s \log (d/s))$) [2405.16805].

### Adaptive and Variance-Reduced ZOO

Adaptive ZOO methods, including R-AdaZO and adaptive step size scaling based on local empirical function-value standard deviation, achieve robust convergence across ill-conditioned or non-stationary regimes [2502.01014, 2602.01627]. The empirical variance of sampled function values is shown to tightly approximate gradient norm, enabling principled step normalization [2602.01627].

Refined moment estimation (first and second moments) in adaptive updates can provably reduce ZO variance and speed up convergence. For instance, R-AdaZO provably reduces the second moment estimate's bias by a factor of up to $(1-\beta_1)/(1+\beta_1)$ compared to previous ZO-AdaMM [2502.01014].

## 4. Distributed and Decentralized ZOO Algorithms

Distributed ZOO methods enable optimization over networks or decentralized data splits. Techniques such as gradient tracking, block coordinate communication, and consensus protocols are used to ensure convergence to a global optimum with minimal communication and ZO query cost [2410.05942, 2204.04743].

In distributed, nonconvex settings, single-point gradient-tracking ZOO algorithms can achieve convergence rates $O(K^{-1/3})$, surpassing centralized counterparts (which achieve $O(K^{-1/4})$), even with a single function evaluation per node per iteration [2410.05942]. Decentralized coordinate ZOO with consensus averaging and powerball acceleration, such as ZOOM-PB, provides further variance control and convergence improvements for multi-agent and federated architectures [2204.04743].

## 5. Specializations: Evolution Strategies, RL, and Black-Box Attacks

### Evolution Strategies and Ancestral RL

ZOO forms the basis of Evolution Strategies (ES), in which a distribution over parameters is perturbed, batch-evaluated, and updated by estimating gradients via aggregate returns. The standard ES gradient estimator is [2408.09493]:
$$
\hat{g}^n = \frac{1}{\sigma} \sum_{i=1}^N R_i \epsilon_i
$$
where $\epsilon_i \sim \mathcal{N}(0, I)$ are parameter perturbations.

Ancestral Reinforcement Learning (ARL) combines ZOO with genetic algorithms, maintaining populations with ancestry-based updates and KL-regularization induced by evolutionary selection. The population-fitness objective and its variational form fundamentally endow ZOO RL with exploration-enhancing entropy regularization [2408.09493].

### Adversarial Black-Box Attacks

ZOO methods underpin modern attacks that generate adversarial examples for DNNs where no gradient access is available. Coordinate-wise ZOO, stochastic coordinate descent with importance sampling, dimension reduction (e.g., noise upscaling), and hierarchical attack schemes have been shown to match or surpass the efficacy of white-box attacks under practical query budgets [1708.03999].

### Policy Optimization Equivalence

Recent theory shows that ZOO with Gaussian-smoothing is equivalent to single-step policy optimization—i.e., REINFORCE with a Gaussian policy and baseline subtraction [2506.14460]. The finite-difference estimator precisely matches the REINFORCE gradient, with baseline subtraction reducing variance via the policy gradient paradigm. This connection justifies query reuse and baseline-averaged estimators for further variance reduction.

## 6. Extensions, Best Practices, and Open Problems

### Extensions and Advanced Estimators

Beyond classical finite differences, advanced estimators include:
- **Regression-based one-point estimators**: RESZO constructs a local linear or quadratic surrogate from historical evaluations, achieving two-point-rate convergence with only one function query per iteration [2507.04223].
- **Unbiased estimators via telescoping series**: Optimal, unbiased gradient estimators based on telescoping directional derivatives that sample over step-scales have been constructed, achieving variance bounds that match the optimal $O(d/\epsilon^4)$ query complexity for nonconvex smooth objectives [2510.19953].
- **Complex-step differentiation**: In the presence of analytic objectives and complex-code support, imaginary-step differentiation avoids catastrophic cancellation and attains optimal rates in strongly convex/quadratic problems [2112.07488].

### Best Practices

- Employ subspace/block perturbations or compressed sensing approaches in high dimensions to reduce variance.
- Apply adaptive step-size and variance reduction (e.g., using first/second moment estimates, empirical function-value variance normalization) to mitigate anisotropy and heterogeneity in landscape geometry.
- Use regression or history-based estimators for single-query scenarios when function queries are costly.
- When available, leverage domain structure (e.g., sparsity, block architecture, Hessian spectrum) for sketching and perturbation alignment.
- For distributed systems, apply coordinate descent, gradient tracking, and consensus schemes for efficient scaling.

### Open Questions

- Characterizing tight lower bounds for query complexity in terms of Hessian spectrum or local landscape statistics [2307.05753, 2510.10945].
- Extending ZOO theory and practice to nonsmooth, constrained, or composite-objective settings with complex structure.
- Automating subspace adaptation and perturbation scheduling based on local sensitivity.
- Exploiting the connections to policy optimization for sequential and reinforcement learning tasks [2506.14460, 2408.09493].

## 7. Impact and Representative Applications

ZOO algorithms are foundational for:
- Black-box adversarial attack generation and robustness evaluation in neural networks [1708.03999, 2006.06224].
- Simulation-based policy search and reinforcement learning, especially in non-differentiable or model-free environments [2408.09493].
- Memory-efficient and hardware-constrained training of deep networks, including large language models, where backpropagation is impractical [2501.19099, 2602.10607].
- Distributed/federated optimization over constrained or nonconvex objectives [2410.05942, 2204.04743].

The evolution of ZOO continues to address statistical, computational, and system-scaling challenges by leveraging modern advances in variance reduction, compressed sensing, distributed consensus, and cross-fertilization with reinforcement learning and policy-gradient frameworks.

Source: https://www.emergentmind.com/topics/zeroth-order-optimization-zoo-a05557ca-52dc-48e6-9f4b-fcb06c7992f7