---
title: Zero-Order Optimization
url: https://www.emergentmind.com/topics/zero-order-optimization
type: topic
---

# Zero-Order Optimization

Zero-order optimization (ZOO), also referred to as derivative-free or black-box optimization, encompasses a class of algorithms designed for solving optimization problems where only function evaluations are available and explicit gradient or higher-order information is inaccessible. These methods are foundational in domains where the underlying objective is non-differentiable, inherently noisy, defined implicitly through simulations, or privacy or security constraints preclude direct access to derivatives. ZOO techniques are now central to modern fields including machine learning, black-box adversarial attacks, hyperparameter and policy optimization, robust control, robotics, and privacy-oriented model inversion.

## 1. Fundamental Principles and Gradient Estimation

Zero-order optimization relies on the ability to probe the objective function \( f:\mathbb{R}^d \to \mathbb{R} \) at chosen query points but prohibits the use of analytic or algorithmic derivatives. The core methodological principle is to emulate first-order (and occasionally higher-order) updates using suitably constructed finite-difference or function-smoothing estimators.

### Canonical Gradient Estimators

- **Two-point random directions**: For a query point \( x \), direction \( u \), and smoothing parameter \( \lambda \), the estimator is
  \[
  g_{\lambda}(x, u) = \frac{f(x+\lambda u) - f(x-\lambda u)}{2\lambda} u
  \]
  This estimator is unbiased for the gradient of the Gaussian-smoothed function \( f_\lambda(x) = \mathbb{E}_{u}[f(x+\lambda u)] \) [2506.05454, 2006.06224, 1312.2139].

- **Coordinate-wise finite differences**: Evaluation along standard basis directions; for the \( i \)-th component:
  \[
  \frac{f(x + h e_i) - f(x)}{h}
  \]
  This variant requires \( O(d) \) queries per step [2506.22087, 2006.06224].

- **Single-point estimators**: Using only a single function evaluation per iteration, such as
  \[
  \frac{d}{\delta} f(x + \delta u) u, \quad u \sim \text{Unif}(S^{d-1})
  \]
  These methods have high variance but are beneficial when queries are extremely restricted [2507.04223, 2410.05942].

- **Complex-step derivative**: For analytic \( f \), estimates derivative using imaginary perturbations with improved bias/variance properties, e.g.,
  \[
  \partial f/\partial x_j(x) \approx \Im f(x + i h e_j)/h + O(h^2)
  \]
  Allowing stable estimates even with very small \( h \) [2112.07488].

The design and choice of estimator govern trade-offs between query efficiency, estimator bias, and variance, shaping convergence rates and practical applicability.

## 2. Convergence Theory and Complexity Bounds

ZOO methodology fundamentally alters the attainable convergence rates compared to first-order methods, due to the absence of direct gradients and the scaling of estimator variance with the problem dimension.

### Convex Settings

- **Smooth Convex Optimization**: With two-point estimators, the best attainable (in expectation) optimization error after \( T \) iterations is
  \[
  O\left(\frac{R L \sqrt{d}}{\sqrt{T}}\right)
  \]
  where \( R \) is the feasible domain diameter and \( L \) the Lipschitz constant of the gradient [1312.2139, 2006.06224]. Multiple evaluations per iteration (mini-batching) can improve the \( \sqrt{d} \) factor to \( \sqrt{d/m} \) [1312.2139].

- **Optimality**: Matching lower bounds show the \( \sqrt{d} \) penalty for two-point methods is unimprovable (modulo constants), and single-point estimators incur a yet worse \( d \) penalty [1312.2139].

- **Strong Convexity**: For \( \alpha \)-strongly convex functions, two-point methods achieve error
  \[
  O\left(\frac{d}{\alpha \sqrt{T}}\right)
  \]
  for minimization [2006.07862, 2102.01121].

### Nonconvex and Stochastic Optimization

- **Stationarity**: For \( L \)-smooth nonconvex functions, two-point ZOO achieves
  \[
  \frac{1}{T}\sum_{t=1}^T \mathbb{E}\|\nabla f(x_t)\|^2 = O\left(\frac{\sqrt{d}}{\sqrt{T}}\right)
  \]
  requiring \( O(d/\epsilon^4) \) total queries to reach \( \mathbb{E}\|\nabla f\|^2 \leq \epsilon^2 \) [2006.06224].

- **Single-point ZOO** converges more slowly: typically \( O(d^2/\epsilon^4) \) or O(1/\( K^{1/3} \)) convergence rates to stationarity, depending on whether (centralized or distributed) settings and estimator variance constants [2507.04223, 2410.05942].

- **Noisy/Adversarial Environments**: Recent results show robust performance under adversarial noise, with error scaling appropriately in \( d \), noise level, and network connectivity (for distributed schemes) [2102.01121, 2406.14362].

### Global Black-Box and Discontinuous Objectives

Sampling-based ZOO, using SDE transport and adaptive "zooming", achieves global minimization under minimal regularity (Gibbs-integrability and local growth), with error
\[
\|1-\exp(U_*- \min_{i \le N} U(X_i^\theta))\|_{L^p} \lesssim d/(m\theta) + \exp[-N (\kappa_0/\kappa_1)^{d/m}/2p]
\]
and empirical robustness to nonsmooth/discontinuous objectives [2509.18201].

## 3. Algorithmic Advances and Variants

Recent research has produced significant algorithmic refinements to ZOO, improving both efficiency and capabilities:

- **Block/partial-gradient estimators**: By randomly updating blocks of variables at each step, query complexity per iteration is reduced to \( O(1) \) while achieving optimal O(\( d/\epsilon^4 \)) total complexity for ε-stationary solutions in constrained, nonconvex–concave settings [2510.19165].

- **Regression-based Single-Point ZO (RESZO)**: Uses regression over historical queries to fit local surrogates (linear, quadratic), substantially reducing estimator variance and empirically matching two-point estimator query complexity [2507.04223].

- **Sharpness-Aware Minimization (SAM) integration**: ZOSA leverages ZO gradient estimation and an inner maximization to explicitly bias solutions toward flat minima, thereby improving generalization in few-shot and prompt-tuning applications [2511.09156, 2506.05454].

- **Safe ZOO with quadratic local approximations**: For black-box constrained problems, quadratic models of constraints built from finite-difference gradients yield feasible iterates converging to ε-KKT points with O(\( d^2/\epsilon^2 \)) complexity, outperforming log-barrier and Bayesian methods in constraint satisfaction [2303.16659].

- **Hierarchical ZOO for Deep Neural Networks**: By recursively bisecting the depth axis, hierarchical ZOO achieves O(ML log L) query count (M = width, L = depth), matching backprop accuracy (cosine similarity >0.95) while avoiding the prohibitive O(ML^2) cost of neuron-wise ZOO in deep architectures [2602.10607].

- **Communication-Efficient Federated and Byzantine-Resilient ZOO**: Recent algorithms such as CYBER-0 aggregate only a small number of scalar ZO queries from each client rather than full gradients, using robust aggregation (trimmed mean) to tolerate Byzantine attacks, achieving O(k) communication per round [2406.14362].

- **Distributed and Stochastic Settings**: Extensions include ZOO with gradient tracking in decentralized networks, attaining consensus and O(1/K^{1/3}) convergence in nonconvex stochastic objectives using only single-point oracle calls per iteration [2410.05942].

## 4. Application Domains

Zero-order optimization has found impactful applications across multiple technical domains:

- **Model Inversion and Privacy**: Reconstruction attacks on neural networks, such as "Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space" (DarkerBB), exploit ZOO in a PCA-projected subspace to efficiently reconstruct plausible images with limited queries and without embedding access [2506.09777].

- **Neural Architecture Search**: ZARTS demonstrates the power of ZOO (random search, maximum-likelihood guided smoothing, gradientless descent) to robustly discover performant neural architectures in search spaces where gradient approximations distort the objective landscape [2110.04743].

- **Robotics and Control**: ZOO underlies essential black-box trajectory and policy optimization algorithms, including Predictive Sampling, MPPI, and CMA-ES, solving high-dimensional planning in contact-rich and nonsmooth environments [2506.22087].

- **Adversarial Machine Learning**: ZOO is core to black-box adversarial attack crafting, leveraging two-point and sign-based gradient approximations to efficiently find high-confidence adversarial examples [2006.06224].

- **Federated Learning and Privacy**: Zero-order methods reduce communication and memory overhead and enable privacy-preserving distributed model training [2406.14362].

## 5. Structural Bias and Regularization Effects

A prominent feature of zero-order methods is implicit regularization. Standard two-point estimators, via the smoothing they induce, bias optimization trajectories toward flat minima—those with small Hessian trace—demonstrated both theoretically and empirically in large-scale models [2506.05454, 2511.09156]. Sharpness-aware ZOO, by design, enhances this effect, further increasing generalization in modern deep learning pipelines.

## 6. Limitations and Frontiers

While powerful, ZOO is not free from limitations:

- **Curse of Dimensionality**: Variance of random-direction estimators scales with dimension; single-point methods suffer most (\( O(d^2) \)), and even the best two-point methods pay an inherent \( \sqrt{d} \) penalty unless massive parallelization or block-structure is exploited [1312.2139, 2510.19165].

- **Query Complexity**: High iteration counts are often unavoidable unless prior structure, surrogate modeling, or adaptive strategies can be invoked. Sampling-based global ZOO requires increasing sample counts as dimension grows to maintain error bounds [2509.18201].

- **Bias-Variance Tradeoff**: Fine-tuning smoothing parameters, step sizes, and block sizes is critical and problem-dependent [2510.19165, 2507.04223].

- **Numerical Instability in Finite Differences**: Addressed by complex-step and regression-based approaches, which can yield stable, low-bias estimates even under numerical or stochastic noise [2112.07488, 2507.04223].

## 7. Tables and Summary of Complexity Bounds

| Estimator / Method                  | Per-iteration Query Cost | Overall Complexity (convex, smooth)    | Remarks                                |
|-------------------------------------|-------------------------|----------------------------------------|----------------------------------------|
| Two-point random directions         | \( O(1) \)             | \( O(\sqrt{d}/\epsilon^2) \)           | Minimax optimal for many settings      |
| Single-point estimator              | \( O(1) \)             | \( O(d/\epsilon^2) \)                  | Higher variance, slower in high-d      |
| Coordinate-wise (full) finite diff. | \( O(d) \)             | \( O(1/\epsilon^2) \)                  | Expensive per-iteration in high-d      |
| Block-coordinate/partial ZOO        | \( O(b) \)             | \( O(d/\epsilon^4) \) (nonconvex)      | Efficient for adjustable block size    |
| RESZO (regression-based single pt.) | \( O(1) \)             | \( O(d\sqrt{d}/\epsilon^2) \) approx.  | Surrogate fit using query history      |
| Safe ZOO (quadratic approx.)        | \( O(d) \)             | \( O(d^2/\epsilon^2) \)                | Strict constraint feasibility          |

## References

- [2506.09777] Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space
- [2506.05454] Zeroth-Order Optimization Finds Flat Minima
- [2507.04223] Regression-Based Single-Point Zeroth-Order Optimization
- [2511.09156] Zero-Order Sharpness-Aware Minimization
- [2112.07488] Imaginary Zeroth-Order Optimization
- [1312.2139] Optimal rates for zero-order convex optimization: the power of two function evaluations
- [2510.19165] Query-Efficient Zeroth-Order Algorithms for Nonconvex Optimization
- [2303.16659] Safe Zeroth-Order Optimization Using Quadratic Local Approximations
- [2406.14362] Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimization
- [2410.05942] Single Point-Based Distributed Zeroth-Order Optimization with a Non-Convex Stochastic Objective Function
- [2006.06224] A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning
- [2509.18201] Sampling-Based Zero-Order Optimization Algorithms
- [2102.01121] Distributed Zero-Order Optimization under Adversarial Noise
- [2602.10607] Hierarchical Zero-Order Optimization for Deep Neural Networks
- [2102.01121, 2506.22087, 2110.04743, 2006.07862] and additional sources as referenced above.

Zero-order optimization thus represents a broad, rapidly evolving paradigm in computational mathematics and data science, offering principled frameworks for global, constrained, and distributed learning under profound structural restrictions on information access. Its foundational results, encompassing optimality, robustness, and flexibility, continue to drive advances in algorithmic design for modern machine learning and control systems.

Source: https://www.emergentmind.com/topics/zero-order-optimization