Random Reshuffling in Stochastic Optimization

• Random reshuffling is a without-replacement data sampling strategy where the dataset is permuted each epoch to reduce variance and enhance convergence over traditional SGD.
• Key variants include shuffle-once, incremental gradient, proximal, variance-reduced, and distributed versions that expand its applicability across convex, nonconvex, and nonsmooth problems.
• Convergence analyses demonstrate linear and accelerated rates under various conditions, with empirical benefits in large-scale optimization and reinforcement learning.

Random reshuffling (RR) is a batchwise, without-replacement data permutation scheme that has become the practical standard for stochastic optimization in large-scale machine learning. In RR, data points are processed in randomly permuted order in cycles ("epochs"), with each data point accessed exactly once per epoch before a new random permutation is drawn for the next epoch. RR contrasts with uniform-with-replacement sampling typical in classical stochastic gradient descent (SGD). Over the last decade, RR's empirical and theoretical performance advantages have been increasingly understood across smooth, nonsmooth, convex, nonconvex, distributed, and variance-reduced optimization, as well as in stochastic sampling and reinforcement learning.

1. Algorithmic Framework and Variants

RR operates on the finite-sum minimization problem:

$$\min_{x \in \mathbb{R}^d} f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x),$$

where each $f_i$ is typically smooth and possibly convex. The basic RR iteration proceeds as follows:

• At epoch $t$, sample a random permutation $\pi^t$ of $\{1, \dots, n\}$.
• Set $x^t_0 = x^t$.
• For $i = 1, \dots, n$:

  $$x^t_i = x^t_{i-1} - \eta \nabla f_{\pi^t(i)}(x^t_{i-1}).$$

• Set $x^{t+1} = x^t_n$.

Key variants include:

• Shuffle-Once (SO): A single permutation drawn at $t=0$ is reused for all epochs.
• Incremental Gradient (IG): Fixed deterministic order (possibly adversarial).
• Proximal RR (ProxRR): A proximal operator is applied at the end (or, heuristically, each step) for composite problems.
• Variance-Reduced RR: Variants embed control variates (e.g., RR-SVRG, SAGA-RR, AVRG) into the reshuffling paradigm (Malinovsky et al., 2021, Ying et al., 2017).
• Distributed RR: Each agent performs local RR, combined via communication and consensus mechanisms (Huang et al., 2023, Huang et al., 2021, Jiang et al., 2021).
• RR with Momentum: Momentum terms are included in the inner iteration, matching popular practical SGD optimizers (Qiu et al., 2024).

For non-smooth or composite objectives, RR is combined with proximal steps, and in distributed/federated optimization, RR is applied locally at each node with periodic communication (Mishchenko et al., 2021, Malinovsky et al., 2022).

2. Convergence Theory and Complexity Results

Smooth Strongly Convex Regime

For $f_i$0-smooth, $f_i$1-strongly convex objectives, RR achieves linear convergence with strictly faster rates and smaller steady-state error than SGD. Specifically:

• Steady-state error: For constant stepsize $f_i$2, RR achieves mean-squared error $f_i$3, while SGD yields $f_i$4 (Ying et al., 2018, Mishchenko et al., 2020).
• Sample Complexity: To achieve accuracy $f_i$5,

$f_i$6

where $f_i$7, matching lower bounds for finite-sum optimization (Mishchenko et al., 2020, Gürbüzbalaban et al., 2015).

• Accelerated Asymptotics: With Polyak–Ruppert-style iterate averaging and stepsize $f_i$8 for $f_i$9, RR can achieve objective suboptimality $t$0, arbitrarily close to $t$1, outperforming SGD's $t$2 (Gürbüzbalaban et al., 2015).

Smooth Nonconvex and PL Regimes

For smooth, possibly nonconvex objectives, RR achieves improved convergence rates compared to SGD:

• Nonconvex ($t$3-smooth):

$t$4

for $t$5 epochs or $t$6 gradients (Mishchenko et al., 2020, Huang et al., 2021, Qiu et al., 2024).

• PL condition (Polyak–Łojasiewicz):

$t$7

with linear convergence to a neighborhood in the presence of persistent variance (Huang et al., 2023, Huang et al., 2021, Fan et al., 2023).

High Probability and Last Iterate Guarantees

RR admits high-probability complexity results for both first- and second-order guarantees. Concentration results for without-replacement sampling have enabled last-iterate stopping rules with finite-sample complexity matching or improving in-expectation bounds (Yu et al., 2023).

Kurdyka–Łojasiewicz (KL) and Asymptotic Convergence

Under the KL property, the full RR sequence converges to a stationary point, with rates depending on the KL exponent. For exponents $t$8, convergence rates $t$9 for iterates and $\pi^t$0 for objective gap are attainable with polynomially-decaying stepsizes (Li et al., 2021, Qiu et al., 2024, Qiu et al., 2023).

Nonsmooth and Proximal Optimization

Proximal RR algorithms extend the above results to composite objectives with weakly convex, nonsmooth regularizers. The normal map–based norm-PRR achieves $\pi^t$1 complexity for nonsmooth, nonconvex problems—improving existing bounds (Qiu et al., 2023). In convex, distributed, or federated settings, ProxRR and FedRR efficiently reduce the cost of expensive proximal operators (Mishchenko et al., 2021, Malinovsky et al., 2022).

3. Variance Reduction and Data Ordering

Variance-reduced algorithms, when deployed with RR (e.g., RR-SVRG, SAGA-RR, AVRG), achieve linear convergence—and, in the big data regime ($\pi^t$2), match optimal $\pi^t$3 epochs (Malinovsky et al., 2021, Ying et al., 2017). Critically, RR and single-shuffle (SO) outperform with-replacement counterparts both in theory and practice.

Greedy or optimized orderings (herding, Gradient Balancing/GraB) further accelerate convergence, in some settings eliminating the $\pi^t$4 dependency to achieve optimal $\pi^t$5 rates for smooth nonconvex objectives, though with increased memory/computation (Lu et al., 2022).

Method	Nonconvex Rate	PL/Strong Convexity Rate
SGD-replacement	$\pi^t$6	$\pi^t$7
RR	$\pi^t$8	$\pi^t$9
Herding/GraB	$\{1, \dots, n\}$0	$\{1, \dots, n\}$1
VR (e.g. RR-SVRG)	$\{1, \dots, n\}$2	$\{1, \dots, n\}$3, big data

4. Distributed, Federated, and Proximal RR

In networked/distributed environments, RR variants (D-RR, FedRR, GT-RR, ED-RR) inherit the improved rates from centralized RR. For distributed smooth strongly convex objectives, RR achieves $\{1, \dots, n\}$4 rates, matching centralized results up to network-dependent constants determined by the spectral gap. For nonconvex objectives, distributed RR matches the centralized $\{1, \dots, n\}$5 rates (Huang et al., 2023, Huang et al., 2021, Jiang et al., 2021).

Federated RR algorithms extend these benefits to practical federated learning, providing resilience to data heterogeneity, reduced communication, and efficient variance reduction and compressed communication, with theoretically guaranteed convergence matching (or surpassing) Local SGD and its extensions (Mishchenko et al., 2021, Malinovsky et al., 2022).

In composite, nonsmooth, or constrained problems, ProxRR and its distributed/federated variants leverage RR to reduce the number of expensive proximal operator evaluations by a factor $\{1, \dots, n\}$6, at no loss of convergence speed compared to per-iteration prox methods (Mishchenko et al., 2021, Qiu et al., 2023).

5. Algorithmic Properties: Bias, Variance, and Practical Implications

RR's key algorithmic advantage is the suppression of stochastic gradient variance within each epoch, leading to:

• Smaller steady-state error: $\{1, \dots, n\}$7 for RR versus $\{1, \dots, n\}$8 for SGD under constant step-size in the strongly convex regime (Ying et al., 2018, Mishchenko et al., 2020).
• Oscillatory intra-epoch error: RR exhibits periodicity reflecting bias/variance tied to position within the epoch, a phenomenon not present in with-replacement SGD (Ying et al., 2018, Mishchenko et al., 2020).
• Reduced bias in stochastic sampling: When used within stochastic gradient Langevin dynamics, RR provably reduces bias in the stationary distribution measured in Wasserstein distance relative to with-replacement schemes (Shaw et al., 27 Jan 2025).
• Empirical acceleration: RR achieves faster epochs-to-solution and improved cache efficiency versus with-replacement methods, crucial for large-scale implementations (Mishchenko et al., 2020, Shaw et al., 27 Jan 2025).

Concentration results for RR have led to tight high-probability complexity bounds and practical stopping criteria, such as the RR-sc rule for first-order optimality (Yu et al., 2023).

6. Applications and Extensions

RR is integral to the practical training of deep neural networks, regularized generalized linear models, matrix-factorization, and reinforcement learning. Recent extensions include:

• Distributed and federated learning: RR improves performance and communication efficiency in distributed and federated systems, including under lossy communication (compression), data heterogeneity, and variance reduction (Huang et al., 2023, Malinovsky et al., 2022, Huang et al., 2021).
• Experience replay: Translating RR’s variance reduction to reinforcement learning settings enhances sample efficiency in off-policy RL with both uniform and prioritized experience replay buffers (Fujita, 4 Mar 2025).
• Stochastic sampling: RR dramatically suppresses sampling bias in stochastic gradient Langevin dynamics, with improved stationary distribution accuracy and practical hardware efficiency (Shaw et al., 27 Jan 2025).
• Nonsmooth and constraint-heavy regimes: Norm-PRR and related methods extend RR’s theory to nonsmooth/nonconvex composite objectives, with efficient constraints handling and sharpened complexity (Qiu et al., 2023).

• "A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimization" (Qiu et al., 2023)
• "Distributed Random Reshuffling Methods with Improved Convergence" (Huang et al., 2023)
• "Stochastic Learning under Random Reshuffling with Constant Step-sizes" (Ying et al., 2018)
• "GraB: Finding Provably Better Data Permutations than Random Reshuffling" (Lu et al., 2022)
• "Random Reshuffling with Momentum for Nonconvex Problems: Iteration Complexity and Last Iterate Convergence" (Qiu et al., 2024)
• "Random Reshuffling Dominates Stochastic Gradient Descent" (Liu, 30 Jun 2026)
• "Random Reshuffling: Simple Analysis with Vast Improvements" (Mishchenko et al., 2020)
• "Experience Replay with Random Reshuffling" (Fujita, 4 Mar 2025)
• "Distributed stochastic proximal algorithm with random reshuffling for non-smooth finite-sum optimization" (Jiang et al., 2021)
• "Random Reshuffling with Variance Reduction: New Analysis and Better Rates" (Malinovsky et al., 2021)
• "Why Random Reshuffling Beats Stochastic Gradient Descent" (Gürbüzbalaban et al., 2015)
• "A simple linear convergence analysis of the reshuffling Kaczmarz method" (Han et al., 2024)
• "Convergence of Random Reshuffling Under The Kurdyka-Łojasiewicz Inequality" (Li et al., 2021)
• "Fast Convergence of Random Reshuffling under Over-Parameterization and the Polyak-Łojasiewicz Condition" (Fan et al., 2023)
• "Proximal and Federated Random Reshuffling" (Mishchenko et al., 2021)
• "Federated Random Reshuffling with Compression and Variance Reduction" (Malinovsky et al., 2022)
• "Variance-Reduced Stochastic Learning under Random Reshuffling" (Ying et al., 2017)
• "High Probability Guarantees for Random Reshuffling" (Yu et al., 2023)
• "Distributed Random Reshuffling over Networks" (Huang et al., 2021)
• "Random Reshuffling for Stochastic Gradient Langevin Dynamics" (Shaw et al., 27 Jan 2025)