Randomized Stochastic Gradient Method

• Randomized Stochastic Gradient (RSG) method is a family of algorithms that randomizes iterate selection to achieve optimal complexity guarantees for both convex and nonconvex objectives.
• It incorporates techniques like importance and partial-bias sampling to enhance convergence rates and robustness in large-scale finite-sum and simulation-based optimization problems.
• RSG methods extend to biased and zeroth-order settings with adaptive and two-phase strategies, ensuring near-optimal iteration complexity and high-probability convergence.

The Randomized Stochastic Gradient (RSG) Method encompasses a class of stochastic optimization algorithms that leverage randomization in gradient computation or sampling, attaining optimal, often provably tight, complexity guarantees for both convex and nonconvex objectives under stochastic first-order information. RSG methods are central to modern stochastic optimization for large-scale, nonconvex, simulation-based, or structured problems. They generalize classical stochastic gradient descent (SGD) by incorporating randomization in the iterate selection, gradient sampling distributions, and output procedures, resulting in significant improvements in theoretical convergence, practical robustness, and the ability to address biased or zeroth-order oracle settings.

1. Algorithmic Formulation and Variants

The RSG method addresses smooth (possibly nonconvex), unconstrained stochastic programs of the form

$$\min_{x \in \mathbb{R}^n} \; f(x) = \mathbb{E}_\xi[F(x, \xi)]$$

where $f$ has $L$-Lipschitz gradient and only access to a stochastic first-order oracle (SFO), which returns noisy unbiased estimates $G(x_k,\xi_k)$ of $\nabla f(x_k)$ with bounded variance:

$$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$

The core RSG algorithm executes $N$ stochastic gradient steps, but rather than returning the last iterate, an output index $R \in \{1, \dots, N\}$ is drawn at random (with a specified pmf $P_R$), and $x_R$ is returned. This randomization simplifies analysis and yields stronger complexity guarantees in both expectation and high-probability, compared to returning the last or best iterate (Ghadimi et al., 2013).

Representative RSG update:

Step	Formula	Notes
Gradient estimation	$f$0	SFO or minibatch variants
Parameter update	$f$1	Step-size $f$2
Output selection	$f$3	E.g., $f$4

In finite-sum objectives, RSG encompasses importance and partial-bias sampling, as in minimizing $f$5, where $f$6 are $f$7-smooth. Here, sampling $f$8 via $f$9 (with $L$0) and appropriately reweighting the stochastic gradient substantially alters the resulting iteration complexity and convergence constants (Needell et al., 2013).

2. Convergence Theory and Complexity Guarantees

RSG achieves near-optimal complexity for both convex and nonconvex programs. For nonconvex stochastic objectives with $L$1-smoothness and SFO variance bounded by $L$2, it holds that

$L$3

with $L$4, and thus, to obtain $L$5, one needs $L$6 (Ghadimi et al., 2013). This matches lower bounds for first-order stochastic methods for smooth nonconvex optimization.

For strongly convex, smooth finite-sum objectives, under uniform or importance-weighted sampling with step-size $L$7, RSG exhibits exponential convergence in squared error:

$L$8

where $L$9 is a function of the strong convexity parameter $G(x_k,\xi_k)$0 and the smoothness parameter, improving from a worst-case $G(x_k,\xi_k)$1 to an average-case $G(x_k,\xi_k)$2 iteration complexity by employing importance sampling (Needell et al., 2013).

For stochastic games with $G(x_k,\xi_k)$3 players and a potential function $G(x_k,\xi_k)$4, the RSG method achieves $G(x_k,\xi_k)$5 sample complexity for reaching expected residual norm $G(x_k,\xi_k)$6 in the smooth case (Xiao, 22 Feb 2026).

3. Weighted and Partial-Bias Sampling

A central innovation in modern RSG theory is the use of weighted, importance, or partially biased sampling distributions $G(x_k,\xi_k)$7 in finite-sum objectives:

• Uniform sampling: $G(x_k,\xi_k)$8, yields convergence rate governed by worst-case $G(x_k,\xi_k)$9.
• Importance sampling: $\nabla f(x_k)$0, replaces $\nabla f(x_k)$1 by $\nabla f(x_k)$2, which can be much smaller when the $\nabla f(x_k)$3 vary widely.
• Partial-bias (hybrid): $\nabla f(x_k)$4, $\nabla f(x_k)$5, interpolates between the two, optimizing the trade-off between exponential rate and convergence residual (Needell et al., 2013).

Theoretical bounds demonstrate that importance sampling systematically improves both the rate constant (exponent in the linear convergence regime) and the residual bias present when the gradient variance is nonzero. Proper tuning of $\nabla f(x_k)$6 and step-size is crucial for optimal performance.

4. RSG under Biased and Zeroth-Order Oracles

In settings where only biased or zeroth-order (function value) information is available, RSG theory extends by modeling the gradient estimation procedure's bias and variance explicitly:

• For biased SFO,
  • The estimator $\nabla f(x_k)$7 satisfies

  $\nabla f(x_k)$8

  with variance $\nabla f(x_k)$9. - Properly chosen $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$0 optimizes the rate, ensuring

  $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$1

  leading to iteration complexity $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$2. When the bias-variance trade is eliminated, the rate improves to $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$3 (Bhavsar et al., 2020).

• In zeroth-order scenarios, RSG generalized as RSGF uses Gaussian smoothing $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$4 and gradient estimation via finite differences. The sample complexity scales as $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$5, where $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$6 is the problem dimension (Ghadimi et al., 2013).

5. Two-Phase, Adaptive, and High-Probability Variants

To improve high-probability guarantees, two-phase procedures (e.g., 2-RSG, 2-RSGF) repeatedly run the RSG algorithm to generate a short list of candidates, then post-select the one with minimal estimated gradient norm via additional oracle queries. This dramatically improves the large-deviation behavior, yielding:

$$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$7

with overall call complexity only logarithmic in the inverse confidence parameter $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$8 (Ghadimi et al., 2013).

Adaptive and momentum-based RSG variants ("Remote Stochastic Gradient" and "Adaptive Remote Stochastic Gradient," or ARSG) combine momentum, acceleration, and adaptive preconditioning as in ADAM/AMSGrad, but manage look-ahead via a tunable "remote observation factor," improving convergence rates and noise robustness and outperforming classical optimizers empirically on deep learning tasks (Chen et al., 2019).

6. Applications and Connections

RSG methods are foundational for:

• Large-scale finite-sum optimization and empirical risk minimization, where importance sampling leads to substantial gains (Needell et al., 2013).
• Nonconvex/nonsmooth potential games where RSG achieves optimal complexity for finding equilibrium under minimal assumptions on the payoff structure (Xiao, 22 Feb 2026).
• Simulation-based optimization and black-box settings where only zeroth-order or biased gradient information is available (Ghadimi et al., 2013, Bhavsar et al., 2020).

A key theoretical revelation is the connection between RSG (with importance sampling) and the randomized Kaczmarz algorithm for linear systems: both are instances of randomized projection algorithms with exponential convergence whose rates depend on the choice of sampling strategy (Needell et al., 2013).

7. Practical Considerations and Limitations

The effectiveness of RSG critically depends on the ability to compute or estimate local smoothness constants $$\mathbb{E}[G(x_k, \xi_k)] = \nabla f(x_k), \quad \mathbb{E}[\|G(x_k, \xi_k) - \nabla f(x_k)\|^2] \leq \sigma^2 \;.$$9 for optimal importance sampling. If only black-box access to the data distribution is available, rejection sampling can still achieve near-optimal distributional bias at minimal overhead (Needell et al., 2013). For non-smooth or non-strongly-convex objectives, bias-variance trade-offs in gradient estimation can dominate, requiring adaptive or two-phase schemes for robust performance (Bhavsar et al., 2020, Ghadimi et al., 2013).

In summary, the Randomized Stochastic Gradient method and its extensions provide the foundational methodology for optimal stochastic optimization under oracle noise, game structure, data heterogeneity, and sampling constraints, with rigorous complexity guarantees and a broad spectrum of algorithmic variants covering modern machine learning and statistical learning paradigms.