---
title: Zeroth-Order Optimization
url: https://www.emergentmind.com/topics/zeroth-order-optimization-zo
type: topic
---

# Zeroth-Order Optimization

Zeroth-order optimization (ZO) refers to a class of algorithms for solving optimization problems without requiring access to explicit gradient information. Instead, ZO algorithms approximate gradients using only function evaluations, thereby enabling derivative-free solutions for black-box, simulation-based, or non-differentiable problems encountered across machine learning, signal processing, control, and engineering. ZO optimization is essential in scenarios where gradients are unavailable, costly to compute, or subject to privacy or operational constraints.

## 1. Principles and Core Algorithms

ZO optimization methods approximate gradients through finite-difference schemes or randomized smoothing. The basic procedure involves three steps:

1. **Gradient Estimation:** The most common approach is the two-point estimator. For an objective $f:\mathbb{R}^d \to \mathbb{R}$ and a random direction $u$, the estimator is
   \[
   \hat{\nabla} f(x) = \frac{d}{\mu} [f(x + \mu u) - f(x)] u
   \]
   where $\mu > 0$ is a smoothing parameter and $u$ is typically drawn from a uniform distribution on the sphere or a standard Gaussian [2006.06224]. Averaging over minibatches of random directions reduces variance.

2. **Descent Direction Computation:** The estimated gradient is used directly, or after transformation (e.g., $\operatorname{sign}$ operation or adaptive scaling), to form the update direction.

3. **Solution Update:** A typical update is
   \[
   x_{t+1} = x_t - \eta_t \mathbf{m}_t
   \]
   with learning rate $\eta_t$ and $\mathbf{m}_t$ the descent direction [2006.06224].

Alternatives include coordinatewise finite-difference estimation, random or structured subspace perturbations, block coordinate sampling, and surrogate regression-based updates. Recent works have refined gradient estimators for unbiasedness [2510.19953], reduced query dependence on dimension [2307.05753], or used variance reduction and surrogate modeling [2206.07126, 2507.04223].

### Key Algorithms

| Method           | Gradient Estimation      | Query Complexity (per iter) | Remarks                                                          |
|------------------|-------------------------|-----------------------------|------------------------------------------------------------------|
| Two-point ZO     | Random directions       | $O(1)$–$O(d)$               | Standard, high variance in high $d$, unbiased for small $\mu$    |
| Coordinatewise   | Finite difference       | $O(d)$                      | More accurate, requires $d$ queries per gradient                 |
| Hybrid/ZOB-GDA   | Block subset per iter   | $O(b)$ ($b \ll d$)          | Adaptive tradeoff, can reach $O(1)$ per step [2510.19165]        |
| Regression-based | Surrogate from history  | $O(1)$                      | Uses past function values, matches two-point in query efficiency |
| Unbiased ZOO     | Telescoping estimators  | $O(2$–$4)$                  | Constructs unbiased estimators even for finite $\mu$ [2510.19953]|

## 2. Variance Reduction, Bias–Variance Tradeoff, and Query Complexity

A central challenge in ZO optimization is the high variance in gradient estimators, especially in high dimensions. Several works have addressed this via:

- **Averaging/mini-batching:** Reduces estimator variance, but increases query complexity.
- **Coordinate importance sampling and hybrid estimators:** Convex combination of random and coordinatewise estimates with importance sampling on coordinates minimizes variance for fixed query budget [2012.11518].
- **Lazy query reuse:** Techniques such as LAZO [2206.07126] adaptively reuse past function values for variance reduction and query saving, achieving regret bounds matching the symmetric two-point scheme.
- **Learned sampling distributions:** Adaptive learning of the perturbation distribution or direct reinforcement learning over sampling policies can yield significant variance reduction and accelerate convergence [1910.09464, 2104.04405].
- **Subspace and block perturbations:** As high dimensionality is the bottleneck, restricting perturbations and updates to structured low-dimensional subspaces (e.g., block coordinate updates, sparsity patterns, or low-rank projections) reduces the variance term from $O(d)$ to $O(s)$, $s \ll d$, provided the subspace aligns well with curvature directions [2501.19099].
- **Unbiased estimators:** Recent developments formalize telescoping series for the directional derivative to construct unbiased gradient estimators for finite stepsizes $\mu$, eliminating systematic bias present in classic ZO estimators, and optimizing the sampling schedule for minimum variance [2510.19953].

Theoretical bounds indicate that the mean squared error of classic two-point estimators scales as $O(d)$, and achieving convergence rates similar to first-order methods requires $O(d)$ more queries per iteration. By exploiting problem structure, subspace alignment, and learned sampling, several algorithms reduce this dependence, sometimes achieving overall complexity scaling of $\mathcal{O}(d/\epsilon^4)$ for finding $\epsilon$-stationary points [2510.19165, 2510.19953].

## 3. Advances in Adaptive and Meta-learning Approaches

Recent ZO research integrates meta-learning and adaptivity into the optimization pipeline:

- **Learning update rules:** Framing optimizer design as a learning problem, ZO meta-learners use RNNs (e.g., ZO-LSTM) to parameterize and learn both update policies and adaptive variance reduction schemes [1910.09464].
    - UpdateRNN: LSTM-based update rule for historical smoothing.
    - QueryRNN: LSTM-adaptive Gaussian sampling for perturbation, balancing variance-bias at test time.
- **Adaptive moment estimation:** Extensions of adaptive optimizers to ZO (e.g., ZO-AdaMM and R-AdaZO) use moment averaging over estimated gradients for variance reduction, and refine the estimate of the second moment based on smoothed first moments, achieving improved stability and faster convergence [2502.01014].
- **Policy optimization for sampling:** Employing reinforcement learning (e.g., DDPG) or neural networks to learn the sampling policy, allowing the system to focus ZO queries in more informative parameter subspaces [2104.04405].
- **Regression-based single-point ZO (RESZO):** Fits local linear/quadratic surrogates using historical evaluations to provide more accurate gradients using only a single query per iteration, achieving convergence rates comparable to (and sometimes outperforming) two-point methods [2507.04223].

These approaches extend ZO methods beyond hand-designed heuristics, allowing adaptive exploitation of optimization history and problem-specific structure.

## 4. Practical Applications and Impact

ZO optimization has enabled practical advances in several fields:

- **Black-Box Adversarial Attacks:** ZO optimizers consistently outperform hand-designed baselines (e.g., ZO-SGD, ZO-signSGD, ZO-ADAM) in query efficiency and convergence speed when attacking black-box neural network classifiers [1910.09464, 2012.11518].
- **Molecule Optimization:** When used to optimize latent code vectors of molecular autoencoders, ZO methods—especially sign-based updates—demonstrate robustness in challenging non-smooth, discontinuous, and flat landscapes encountered in molecule generation [2210.16099].
- **On-Device and Memory-Constrained Training:** Lightweight ZO protocols (e.g., ElasticZO, PeZO) enable fine-tuning and learning on-memory-constrained devices by avoiding backpropagation and storing only minimal state, making ZO approaches viable for edge/IoT scenarios [2501.04287, 2504.20314].
- **Large Language Model (LLM) Fine-Tuning:** ZO block coordinate descent and memory-efficient subspace perturbation techniques make zeroth-order fine-tuning of models with billions of parameters tractable, reducing both wall-clock time and memory requirements [2501.19099].
- **Constrained Optimization:** ZO algorithms have been extended to composite constrained problems (smooth + nonsmooth, functional constraints), achieving near-optimal query complexity for $\epsilon$-stationary points and supporting resource allocation or adversarial example generation with hard constraints [2112.11420, 2510.19165].
- **Distributed and Federated Settings:** One-point distributed ZO with gradient tracking delivers consensus and convergence in multi-agent, nonconvex, and stochastic environments, even with highly restricted communication and query budgets [2410.05942].

## 5. Theoretical Foundations: Convergence, Optimality, and Implicit Regularization

A robust ZO literature provides:

- **Convergence rates and query complexity:** For smooth nonconvex objectives, optimal complexity $O(d/\epsilon^4)$ is achievable using block coordinate or unbiased estimators [2510.19165, 2510.19953]. Effective dimension analysis (ED$_\alpha$) shows that empirical complexity can be far better than the worst-case $d$-scaling, especially when problem structure limits the number of high-curvature directions [2307.05753].
- **Stationary points and implicit regularization:** ZO methods using random sampling and two-point estimators implicitly bias toward flat minima—solutions with small trace of the Hessian. This is formally connected to minimizing a regularized objective $f(x) + (\lambda^2/2) \operatorname{Tr}(\nabla^2 f(x))$, explaining empirically observed preference for "flat" (more generalizable) minima, even in overparameterized ML models [2506.05454].
- **Block versus full-dimension estimation:** The multi-query paradox is resolved: simple averaging over $q > 1$ queries per step (for fixed total budget) is sub-optimal compared with the single-query approach, unless projection alignment (ZO-Align) is used, in which case full-subspace updates become optimal [2509.15552].
- **Constraint handling and control-theoretic frameworks:** ZO algorithms leveraging feedback linearization and augmented Lagrangian schemes achieve finite-time feasibility without requiring convex or white-box constraints [2112.11420, 2509.24056].

## 6. Implementation and Scaling Considerations

Efficient ZO optimization in real-world systems demands special attention to computational resources and hardware efficiency:

- **Hardware-Friendly ZO Training:** The cost of random number generation for perturbations in ZO is a primary bottleneck in FPGA/ASIC deployment. Approaches such as random number reuse and hardware-friendly adaptive scaling (PeZO) address this by sharing and scaling uniform random vectors, reducing LUT, FF, and power consumption while maintaining convergence [2504.20314].
- **Sparse and Blockwise Methods:** Only a subset of coordinates or subnetworks are updated per step (block coordinate or structured pruning), dramatically lowering per-iteration computational and memory cost while maintaining convergence and generalization [2310.02025, 2501.19099].
- **Parallelism and feature reuse:** Efficient multi-GPU and distributed implementations leverage the decoupled nature of coordinatewise finite-difference estimation for fast, scalable training [2310.02025].

## 7. Open Problems and Frontier Directions

The current landscape of ZO research highlights several promising directions:

1. **Adaptive and learned substructure selection:** Extending block and subspace sampling by learning which directions/subspaces to perturb in an online or data-driven manner for maximal informational gain [2501.19099].
2. **Unbiased, adaptive, and low-variance estimators:** Systematic exploration of telescoping series and adaptive sampling to achieve near-optimal variance/bias tradeoffs in high-dimensional, nonconvex, or constraint-rich settings [2510.19953].
3. **Beyond smooth domains:** Generalizing ZO theory and practice to non-smooth, highly nonconvex, or discrete-structured objectives using e.g. trust-region or double-randomization methods [2006.06224].
4. **Distributed and federated ZO methods:** Addressing scalability, privacy, and communication efficiency in multi-agent and federated optimization [2410.05942].
5. **On-device training and hardware–algorithm co-design:** Integrating ZO solvers directly onto hardware accelerators and developing algorithms/architectures jointly for optimal power, speed, and learning performance [2504.20314].

ZO optimization thus serves as a critical technology for tackling black-box, simulation-based, and hardware-constrained learning tasks, balancing theoretical guarantees with practical efficiency and robustness to problem structure and operational constraints.

Source: https://www.emergentmind.com/topics/zeroth-order-optimization-zo