Randomized Stochastic Gradient-Free Method

• The method leverages randomized smoothing and finite-difference estimators to approximate gradients, enabling optimization without direct gradient access.
• It achieves established convergence rates and improved complexity via techniques like recursive variance reduction and momentum acceleration.
• RSGF is pivotal for optimizing noisy, nonconvex functions in high-dimensional settings, with successful applications in training neural networks.

The randomized stochastic gradient-free (RSGF) method is a class of zeroth-order stochastic optimization algorithms for nonconvex (and possibly nonsmooth) objective functions, which operate in settings where only noisy function evaluations are available. RSGF leverages randomized smoothing and finite-difference gradient estimators to enable stochastic optimization without direct access to gradients. The method achieves established convergence rates to approximate Goldstein stationary points, with practical extensions that yield high-probability guarantees and improved complexity using recursive variance reduction and momentum techniques (Ghadimi et al., 2013, Luo et al., 2019, Lin et al., 2022, Chen et al., 2023).

1. Mathematical Model and Problem Setting

RSGF targets optimization of objectives of the form

$$\min_{x \in \mathbb{R}^d} f(x) := \mathbb{E}_{\xi}[F(x,\xi)]$$

where $F(\cdot,\xi)$ is a possibly nonconvex, stochastic black-box function, and access is limited to (possibly noisy) function evaluations. Depending on the setting:

• $f$ may be smooth or nonsmooth but is typically Lipschitz or mean-squared Lipschitz.
• The accessible oracle satisfies bounded variance: $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$.
• Direct gradient, subgradient, or higher-order information is not available.

The central computational goal is to reach an $\epsilon$-stationary point—characterized by small gradient norm for smooth $f$, or a $(\delta,\epsilon)$-Goldstein stationary point for nonsmooth functions (Ghadimi et al., 2013, Lin et al., 2022).

2. Randomized Smoothing and Gradient-Free Estimation

Without gradient information, RSGF uses randomized smoothing to make $f$ more amenable to (finite-difference) estimation. The standard smoothing constructs a function $f_{\mu}$ or $f_{\delta}$ via convolution: $F(\cdot,\xi)$0 This operation ensures that:

• $F(\cdot,\xi)$1, $F(\cdot,\xi)$2 become differentiable even if $F(\cdot,\xi)$3 is merely Lipschitz.
• $F(\cdot,\xi)$4, i.e., in the Goldstein subdifferential of $F(\cdot,\xi)$5 at $F(\cdot,\xi)$6 (Lin et al., 2022, Chen et al., 2023).
• Smoothing and two-point finite-difference estimation underpin all major RSGF variants.

The RSGF gradient estimator is generally: $F(\cdot,\xi)$7 or the Gaussian variant: $F(\cdot,\xi)$8 These estimators are unbiased for $F(\cdot,\xi)$9 (or $f$0), with variance $f$1 (Ghadimi et al., 2013, Chen et al., 2023).

3. Baseline RSGF Algorithms

All RSGF algorithms iterate as follows:

1. Sample a random direction $f$2 (or $f$3), and a data sample $f$4.
2. Compute the stochastic finite-difference estimator $f$5.
3. Update the iterate: $f$6, for some stepsize $f$7.

A widely-analyzed version (uniform smoothing, two-point estimator) is (Lin et al., 2022, Chen et al., 2023): $f$3 The output is $f$8 for randomly $f$9.

In the setting where $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$0 is smooth, the estimator is precisely tailored so that the bias due to smoothing $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$1 can be matched or dominated by variance via the choice $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$2, delivering optimal bias–variance tradeoff (Ghadimi et al., 2013).

4. Convergence Theory and Complexity

Expectation bounds:

Under Lipschitz assumptions, the smoothed RSGF algorithm guarantees: $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$3 with sample complexity (number of zero-order calls): $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$4 where $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$5 is the dimension, $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$6 is the smoothing parameter, and $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$7 the stationarity target (Lin et al., 2022, Chen et al., 2023).

For the smooth case, if $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$8 is $$\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$$9-smooth and the stepsize/parameters are appropriately chosen,

$\epsilon$0

with required number of calls $\epsilon$1 (Ghadimi et al., 2013).

High-probability guarantees (Two-phase RSGF):

Using $\epsilon$2 independent runs, and validating candidate solutions with a mini-batch estimator, the two-phase RSGF achieves: $\epsilon$3 with total cost

$\epsilon$4

(Lin et al., 2022, Chen et al., 2023).

5. Advanced Extensions: Acceleration and Variance Reduction

Momentum/acceleration

Accelerated RSGF algorithms incorporate momentum, as in: $\epsilon$5 with $\epsilon$6 and $\epsilon$7 a normalization. This approach yields convergence rates $\epsilon$8 for strongly convex objectives (with bias and variance both $\epsilon$9) (Luo et al., 2019).

Recursive variance reduction (SPIDER/SARAH):

Utilizing recursive gradient estimators $f$0, complexity with respect to $f$1 can be improved from $f$2 to $f$3. Specifically, the GFM+ variant [Editor's term] forms

$f$4

with suitable choice of epoch length $f$5, batch sizes $f$6 (Chen et al., 2023).

The total zeroth-order oracle complexity becomes: $f$7 which is a dimension-dependent but tighter rate than vanilla RSGF.

6. Stationarity Concepts and Smoothing-Subdifferential Mapping

For nonsmooth objectives, stationarity is formalized using the Goldstein subdifferential: $f$8 where $f$9 is the Clarke subdifferential. $(\delta,\epsilon)$0-Goldstein stationarity is achieved when

$(\delta,\epsilon)$1

Uniform smoothing ensures $(\delta,\epsilon)$2, and RSGF constructs its stationary guarantees using this mapping. This equivalence is central to both theoretical convergence and complexity analysis (Lin et al., 2022, Chen et al., 2023).

7. Practical Implementation and Applications

Parameter selection:

Step-size choices ($(\delta,\epsilon)$3 for nonsmooth, $(\delta,\epsilon)$4 for smooth) are critical for balancing estimation bias and variance. The smoothing parameter ($(\delta,\epsilon)$5 or $(\delta,\epsilon)$6) is usually tied to $(\delta,\epsilon)$7 via $(\delta,\epsilon)$8 (Ghadimi et al., 2013, Lin et al., 2022, Chen et al., 2023).

Two-phase validation:

Employing multiple independent runs and selecting via post hoc validation using mini-batch gradient estimators yields strong large-deviation bounds.

Applications:

Two-phase RSGF (2-SGFM) has been demonstrated to train small-scale convolutional neural networks on MNIST, showing competitive accuracy with classical gradient-based methods on this task even when only function-value queries are available. Batch size and validation sample size influence empirical stability and match predicted theory (Lin et al., 2022).

Complexity comparison and guidelines:

Algorithm	Complexity (oracle calls)	Key features
RSGF (basic)	$(\delta,\epsilon)$9	Two-point, smoothing
RSGF (two-phase)	$f$0	High-prob confidence
Accelerated RSGF (GFM+)	$f$1	Recursive variance red.
Classic gradient-based	$f$2 (for reference)	First-order only

Summary:

The RSGF framework and its accelerations provide general, robust zeroth-order algorithms for high-dimensional, nonsmooth, nonconvex stochastic optimization, with theoretically grounded oracle complexity and demonstrated practical feasibility (Ghadimi et al., 2013, Luo et al., 2019, Lin et al., 2022, Chen et al., 2023).