High-Probability Guarantees for Random Zeroth-Order (Stochastic) Gradient Descent

Abstract: Zeroth-order optimization aims to minimize an objective function using only function evaluations, and is therefore fundamental in black-box optimization, hyperparameter tuning, bandit learning, and adversarial machine learning. While classical zeroth-order methods are well understood in expectation, much less is known about their high-probability behavior, especially for smooth and strongly convex objectives. In this paper, we establish high-probability convergence guarantees for random zeroth-order gradient descent in both deterministic and stochastic settings. For deterministic $L$-smooth and $μ$-strongly convex objectives of $d$-dimension, we show that the classical two-query random zeroth-order method finds an $\varepsilon$-suboptimal solution with probability at least $1-δ$ using [ \mathcal{O}\left( \frac{dL}μ\log\frac{1}{\varepsilon} + \log\frac{1}δ \right) ] function queries. Thus, compared with the standard in-expectation complexity, only an additive logarithmic dependence on the confidence parameter is needed. For stochastic objectives, under a bounded-noise condition and without assuming uniformly bounded stochastic gradients, we prove that random zeroth-order stochastic gradient descent achieves an $\varepsilon$-suboptimal solution with probability at least $1-δ$ using [ \mathcal{O}\left( \frac{ d\log(1/\varepsilon) \left(\log(1/\varepsilon)+\log(1/δ)\right) }{\varepsilon} \right) ] queries. Our results provide high-confidence counterparts to classical expectation-based zeroth-order convergence guarantees and clarify the additional cost required to obtain reliable performance guarantees.

• The paper provides the first sharp high-probability convergence analysis of random zeroth-order gradient descent under strong convexity, using martingale concentration techniques.
• It establishes near-optimal query complexity bounds for both deterministic and stochastic settings with only two function queries per iteration.
• The work relaxes the traditional bounded gradient assumption to bounded noise, broadening the applicability of derivative-free optimization in risk-sensitive scenarios.

High-Probability Zeroth-Order Optimization: Guarantees and Complexity

Introduction

Zeroth-order (ZO), or derivative-free, optimization is fundamental in scenarios where explicit gradients are unavailable or expensive to compute, including black-box optimization, hyperparameter search, bandit learning, and adversarial robustness. Classical ZO algorithms, notably random direction finite-difference schemes, are well-understood in terms of expected convergence for smooth convex functions. However, in high-stakes applications, guarantees in expectation are insufficient; high-confidence (i.e., high-probability) convergence guarantees become critical but are significantly more technically challenging to establish due to the martingale structure and noise properties inherent in ZO estimators. The paper "High-Probability Guarantees for Random Zeroth-Order Gradient Descent" (2604.23613) provides the first sharp high-probability convergence analysis for random ZO gradient descent in both deterministic and stochastic strongly convex regimes.

Problem Setup and Algorithmic Framework

The focus is on minimizing an $L$-smooth, $\mu$-strongly convex function $f:\mathbb{R}^d\to\mathbb{R}$, or a stochastic objective $f(x) = \mathbb{E}_\xi[f(x;\xi)]$, using only function-value queries. Gradient estimates are constructed via two-point finite-difference in random (Gaussian) directions:

$$g(x) = \frac{f(x + \alpha u) - f(x - \alpha u)}{2\alpha} u$$

with $u \sim N(0, I_d)$. Iterative updates are performed by setting $x_{t+1} = x_t - \eta_t g(x_t)$ (or its stochastic counterpart with $f(x;\xi)$).

High-Probability Query Complexity: Deterministic Setting

The core technical accomplishment is to rigorously analyze the high-probability convergence of ZO gradient descent with a minimal number of function queries per iteration, under strong convexity:

• For deterministic $L$-smooth, $\mu$-strongly convex objectives, to produce an $\mu$0-suboptimal point with probability at least $\mu$1, the query complexity is established as

$\mu$2

This result matches the classic in-expectation complexity up to an additive $\mu$3 term, thus showing that high confidence can be achieved without increasing the per-iteration query cost.

• The method relies on a multi-step martingale concentration argument across the iterates, using Beta distribution properties for the ZO estimator and advanced uniform tail bounds to control the sequential dependence.

This sharp result resolves a key open question in the literature: previous high-probability analyses of ZO methods required either large per-step batched queries or incurred superfluous $\mu$4 factors in sample complexity. The uniform-in-time control here is optimal up to constants and is achieved with only two function queries per update (the minimal possible for directional finite-difference).

High-Probability Query Complexity: Stochastic Setting

In the stochastic setting, each function evaluation is corrupted by additional noise, introducing further complexity:

• Under the assumption of bounded noise (i.e., $\mu$5 almost surely), but without requiring the more restrictive assumption of bounded stochastic gradients (as is common in OCO literature), the paper proves that with probability at least $\mu$6,

$\mu$7

function queries are sufficient to guarantee an $\mu$8-suboptimal solution.

• The analysis employs an intricate double-martingale technique to simultaneously control estimator bias and stochastic noise, along with uniform concentration for all iterations.
• Notably, the multiplicative $\mu$9 and $f:\mathbb{R}^d\to\mathbb{R}$0 factors arise from the necessity of uniform control over all iterates and the lack of the bounded gradient assumption, reflecting an almost unavoidable "price of confidence" in the stochastic setting.

This bounds the cost of high-probability, high-accuracy ZO optimization under minimal stochastic regularity assumptions, and nearly matches known lower bounds when $f:\mathbb{R}^d\to\mathbb{R}$1 and the accuracy parameter are not exponentially tight.

Technical Insights

Key technical contributions include:

• Uniform Suffix-Sum Lower Bounds: By leveraging Beta tail bounds and martingale inequalities, the authors obtain for the first time dimension-appropriate high-probability decay rates for the sum of squared directional projections.
• Step-Size Adaptation: Adaptive step-size selection, scaling inversely with the norm of the random Gaussian direction, ensures the correct bias-variance tradeoff, which is necessary for high-probability control (in contrast to the classical $f:\mathbb{R}^d\to\mathbb{R}$2 statically chosen step size).
• No Bounded Gradient Assumption: By assuming only bounded noise, the analysis is applicable in practical regimes (e.g., empirical risk minimization with finite data), but imposes a stronger condition than sub-Gaussianity; however, the techniques can be readily adapted if stronger moment control is available.

Comparison to Prior Work

The paper demonstrates improvements over recent works:

• In the deterministic regime, it closes the gap between expectation and high-probability bounds without introducing superfluous constants or logarithmic factors, outperforming previous approaches which required batch queries per step or incurred extra logarithmic penalties [see comparisons to results in e.g., "Random Coordinate Descent and High-probability Analysis" [ye2025unified]].
• In the stochastic regime, the relaxation from bounded gradient to bounded noise significantly extends applicability relative to prior OCO analyses (which typically require gradient bound assumptions).

Summary of main complexities (for $f:\mathbb{R}^d\to\mathbb{R}$3-dimensional, $f:\mathbb{R}^d\to\mathbb{R}$4-smooth, $f:\mathbb{R}^d\to\mathbb{R}$5-strongly convex $f:\mathbb{R}^d\to\mathbb{R}$6):

Setting	Assumptions	Confidence Type	Query Complexity
Deterministic	None beyond strong convexity	High probability	$f:\mathbb{R}^d\to\mathbb{R}$7
Stochastic	Bounded noise	High probability	$f:\mathbb{R}^d\to\mathbb{R}$8

Practical and Theoretical Implications

• Algorithmic Reliability: These results indicate that standard two-query ZO methods are sufficient for strong probabilistic performance guarantees, enhancing their practicality in safety-critical or adversarial applications.
• Sample Efficiency: In deterministic problems, the extra price for confidence is negligible; in stochastic settings, a logarithmic penalty is incurred, which may be further improvable under refined noise models.
• Generality: The relaxation regarding bounded gradient assumptions greatly broadens the applicability to composite and empirical risk objectives, common in machine learning.
• Future Directions: The discrepancy in log factors between deterministic and stochastic cases suggests further investigation into refined concentration phenomena for ZO estimators under weaker noise moment assumptions, variance reduction, and extensions to nonconvex objectives.

Conclusion

The paper provides the first tight, high-probability query complexity guarantees for standard random ZO gradient descent methods in smooth strongly convex optimization. It shows that high-confidence solutions can be ensured with essentially no penalty over expectation in the deterministic case, and only mild logarithmic overhead in the stochastic setting—without requiring restrictive gradient-boundedness assumptions. These results remove a key practical barrier for deploying ZO solvers in risk-sensitive settings and suggest several promising research directions for tightening stochastic complexity and broadening noise models. The theoretical techniques introduced are also broadly relevant to high-probability analysis of other random-sampling, feedback-limited optimization algorithms.