---
title: Randomized Stochastic Gradient-Free Method
url: https://www.emergentmind.com/topics/randomized-stochastic-gradient-freer-rsgf-method
type: topic
---

# Randomized Stochastic Gradient-Free Method

The randomized stochastic gradient-free (RSGF) method is a class of zeroth-order stochastic optimization algorithms for nonconvex (and possibly nonsmooth) objective functions, which operate in settings where only noisy function evaluations are available. RSGF leverages randomized smoothing and finite-difference gradient estimators to enable stochastic optimization without direct access to gradients. The method achieves established convergence rates to approximate Goldstein stationary points, with practical extensions that yield high-probability guarantees and improved complexity using recursive variance reduction and momentum techniques [1309.5549][1912.13305][2209.05045][2301.06428].

## 1. Mathematical Model and Problem Setting

RSGF targets optimization of objectives of the form
\[
\min_{x \in \mathbb{R}^d} f(x) := \mathbb{E}_{\xi}[F(x,\xi)]
\]
where $F(\cdot,\xi)$ is a possibly nonconvex, stochastic black-box function, and access is limited to (possibly noisy) function evaluations. Depending on the setting:
- $f$ may be smooth or nonsmooth but is typically Lipschitz or mean-squared Lipschitz.
- The accessible oracle satisfies bounded variance: $\mathbb{E}_{\xi}[(F(x,\xi)-f(x))^2] \leq \sigma_0^2$.
- Direct gradient, subgradient, or higher-order information is not available.

The central computational goal is to reach an $\epsilon$-stationary point—characterized by small gradient norm for smooth $f$, or a $(\delta,\epsilon)$-Goldstein stationary point for nonsmooth functions [1309.5549][2209.05045].

## 2. Randomized Smoothing and Gradient-Free Estimation

Without gradient information, RSGF uses randomized smoothing to make $f$ more amenable to (finite-difference) estimation. The standard smoothing constructs a function $f_{\mu}$ or $f_{\delta}$ via convolution:
\[
f_{\mu}(x) := E_{u \sim N(0,I)}[f(x+\mu u)], \quad
f_{\delta}(x) := E_{u \sim \operatorname{Unif}(B_1(0))}[f(x+\delta u)]
\]
This operation ensures that:
- $f_\mu$, $f_\delta$ become differentiable even if $f$ is merely Lipschitz.
- $\nabla f_{\delta}(x) \in \partial_\delta f(x)$, i.e., in the Goldstein subdifferential of $f$ at $x$ [2209.05045][2301.06428].
- Smoothing and two-point finite-difference estimation underpin all major RSGF variants.

The RSGF gradient estimator is generally:
\[
g_t = \frac{d}{2\delta}[F(x_t + \delta w_t, \xi_t) - F(x_t - \delta w_t, \xi_t)] w_t;\quad w_t \sim \mathrm{Unif}(S^{d-1})
\]
or the Gaussian variant:
\[
G_{\mu}(x,\xi,u) = \frac{F(x+\mu u,\xi) - F(x,\xi)}{\mu} u;\quad u \sim N(0, I)
\]
These estimators are unbiased for $\nabla f_\mu(x)$ (or $\nabla f_\delta(x)$), with variance $O(d\| \nabla f(x) \|^2 + d \sigma_0^2)$ [1309.5549][2301.06428].

## 3. Baseline RSGF Algorithms

All RSGF algorithms iterate as follows:
1. Sample a random direction $w_t$ (or $u_t$), and a data sample $\xi_t$.
2. Compute the stochastic finite-difference estimator $g_t$.
3. Update the iterate: $x_{t+1} = x_t - \eta g_t$, for some stepsize $\eta$.

A widely-analyzed version (uniform smoothing, two-point estimator) is [2209.05045][2301.06428]:
```python
for t = 0, 1, ..., T-1:
    w_t ~ Uniform(S^{d-1});  ξ_t ~ data oracle
    g_t = (d/(2δ)) * [F(x_t + δ w_t, ξ_t) - F(x_t - δ w_t, ξ_t)] * w_t
    x_{t+1} = x_t - η g_t
```
The output is $x_R$ for randomly $R \in \{0, ..., T-1\}$.

In the setting where $f$ is smooth, the estimator is precisely tailored so that the bias due to smoothing $\|\nabla f_\mu(x) - \nabla f(x)\| = O(\mu n^{3/2} L)$ can be matched or dominated by variance via the choice $\mu = \Theta(1 / \sqrt{N})$, delivering optimal bias–variance tradeoff [1309.5549].

## 4. Convergence Theory and Complexity

**Expectation bounds:**  
Under Lipschitz assumptions, the smoothed RSGF algorithm guarantees:
\[
\mathbb{E}[\min_{g \in \partial_\delta f(x_R)} \|g\|] \leq \epsilon
\]
with sample complexity (number of zero-order calls):
\[
O(d^{3/2}\delta^{-1}\epsilon^{-4})
\]
where $d$ is the dimension, $\delta$ is the smoothing parameter, and $\epsilon$ the stationarity target [2209.05045][2301.06428].

For the smooth case, if $f$ is $L$-smooth and the stepsize/parameters are appropriately chosen,
\[
\mathbb{E}\|\nabla f(x_R)\|^2 = O\left(\frac{n L D_f^2}{N} + \frac{n \sigma_0 D_f}{\sqrt{N}}\right)
\]
with required number of calls $N = O(n / \epsilon^2)$ [1309.5549].

**High-probability guarantees (Two-phase RSGF):**  
Using $S = O(\log (1/\alpha))$ independent runs, and validating candidate solutions with a mini-batch estimator, the two-phase RSGF achieves:
\[
\mathbb{P}(\min_{s} \min_{g \in \partial_\delta f(x^{(s)})} \|g\| > \epsilon) \leq \alpha
\]
with total cost
\[
O(d^{3/2}\delta^{-1}\epsilon^{-4}\log(1/\alpha))
\]
[2209.05045][2301.06428].

## 5. Advanced Extensions: Acceleration and Variance Reduction

**Momentum/acceleration**  
Accelerated RSGF algorithms incorporate momentum, as in:
\[
v_k = \gamma_k v_{k-1} + \frac{1}{\alpha_k} s_k;\quad m_k = v_k / Z_k;\quad x_{k+1} = x_k + \alpha_k m_k
\]
with $\gamma_k = (k/(k+1))^p$ and $Z_k$ a normalization. This approach yields convergence rates $O(1 / k^2)$ for strongly convex objectives (with bias and variance both $O(1/k)$) [1912.13305].

**Recursive variance reduction (SPIDER/SARAH):**  
Utilizing recursive gradient estimators $v_t$, complexity with respect to $\epsilon$ can be improved from $O(\epsilon^{-4})$ to $O(\epsilon^{-3})$. Specifically, the GFM+ variant [*Editor's term*] forms
\[
\begin{cases}
v_t = \frac{1}{b'} \sum_{i=1}^{b'} g(x_t; w_i, \xi_i), & t \equiv 0 \mod m\\
v_t = v_{t-1} + \frac{1}{b} \sum_{j=1}^b [g(x_t; w_j, \xi_j) - g(x_{t-1}; w_j, \xi_j)], & \text{else}
\end{cases}
\]
with suitable choice of epoch length $m$, batch sizes $b', b$ [2301.06428].

The total zeroth-order oracle complexity becomes:
\[
O(L^3 d^{3/2} \epsilon^{-3} + \Delta L^2 d^{3/2} \delta^{-1} \epsilon^{-3})
\]
which is a dimension-dependent but tighter rate than vanilla RSGF.

## 6. Stationarity Concepts and Smoothing-Subdifferential Mapping

For nonsmooth objectives, stationarity is formalized using the Goldstein subdifferential:
\[
\partial_\delta f(x) = \mathrm{conv} \{ \partial f(y) : \|y-x\| \leq \delta \}
\]
where $\partial f(y)$ is the Clarke subdifferential. $(\delta,\epsilon)$-Goldstein stationarity is achieved when
\[
\min_{g \in \partial_\delta f(x)} \|g\| \leq \epsilon
\]
Uniform smoothing ensures $\nabla f_\delta(x) \in \partial_\delta f(x)$, and RSGF constructs its stationary guarantees using this mapping. This equivalence is central to both theoretical convergence and complexity analysis [2209.05045][2301.06428].

## 7. Practical Implementation and Applications

**Parameter selection:**  
Step-size choices ($\eta \sim \delta / d^{3/2} G^3 T$ for nonsmooth, $\gamma_k \sim 1 / [L(n+4)]$ for smooth) are critical for balancing estimation bias and variance. The smoothing parameter ($\mu$ or $\delta$) is usually tied to $\epsilon$ via $\delta \sim \epsilon / L$ [1309.5549][2209.05045][2301.06428].

**Two-phase validation:**  
Employing multiple independent runs and selecting via post hoc validation using mini-batch gradient estimators yields strong large-deviation bounds.

**Applications:**  
Two-phase RSGF (2-SGFM) has been demonstrated to train small-scale convolutional neural networks on MNIST, showing competitive accuracy with classical gradient-based methods on this task even when only function-value queries are available. Batch size and validation sample size influence empirical stability and match predicted theory [2209.05045].

**Complexity comparison and guidelines:**

| Algorithm                | Complexity (oracle calls)                           | Key features           |
|--------------------------|-----------------------------------------------------|------------------------|
| RSGF (basic)             | $O(d^{3/2} \delta^{-1} \epsilon^{-4})$              | Two-point, smoothing   |
| RSGF (two-phase)         | $O(d^{3/2} \delta^{-1} \epsilon^{-4} \log(1/\alpha))$ | High-prob confidence   |
| Accelerated RSGF (GFM+)  | $O(d^{3/2} \epsilon^{-3})$                          | Recursive variance red.|
| Classic gradient-based   | $O(1/\epsilon^2)$ (for reference)                   | First-order only       |

**Summary:**  
The RSGF framework and its accelerations provide general, robust zeroth-order algorithms for high-dimensional, nonsmooth, nonconvex stochastic optimization, with theoretically grounded oracle complexity and demonstrated practical feasibility [1309.5549][1912.13305][2209.05045][2301.06428].

Source: https://www.emergentmind.com/topics/randomized-stochastic-gradient-freer-rsgf-method