Data Shuffling & Stochastic Batching

• Data shuffling and stochastic batching are strategies that use random data ordering and mini-batch grouping to reduce variance and speed up convergence.
• These methods balance I/O, memory, and computational costs by employing techniques like random reshuffling, block sampling, and partial shuffling.
• They are integral to distributed learning and privacy frameworks, offering trade-offs between statistical efficiency, scalability, and robust privacy amplification.

Data shuffling and stochastic batching are core methodologies in modern stochastic optimization and large-scale machine learning, enabling efficient variance reduction, improved convergence, practical distributed learning, and privacy amplification. Stochastic batching refers to grouping randomly sampled data points into mini-batches for execution of stochastic gradient or other update schemes. Data shuffling concerns the order and protocol by which data are accessed—typically via random permutations—at various computational and storage scales. These concepts underpin theoretical and practical advances in optimization complexity, distributed systems, statistical efficiency, and privacy frameworks.

1. Theoretical Foundations of Data Shuffling and Stochastic Batching

Fundamental results establish the empirical risk minimization problem as a finite sum: $F(x) = \frac{1}{n} \sum_{i=1}^n f_i(x),$ where each $f_i$ is a loss term. Stochastic batching refers to constructing updates from randomly selected mini-batches at each iteration, typically via either with-replacement sampling (classical SGD) or without-replacement (shuffling the order per epoch) (Haochen et al., 2018).

Random reshuffling, where a new permutation is drawn every epoch and each data point is visited precisely once per epoch, provably yields strictly faster convergence in the large-sample regime under strong convexity and second-order smoothness. Specifically, after $T$ steps (across $K = T/n$ epochs), random reshuffling achieves the rate $O(1/T^2 + n^3/T^3)$, whereas standard sampling with replacement yields only $O(1/T)$. For sparse data, the $n^3$-term can be replaced by $O(\rho^2 n^3/T^3)$ with sparsity factor $\rho$, and in certain settings can even be omitted (Haochen et al., 2018).

Extensions to nonconvex Polyak–Łojasiewicz objectives and the general convex case show that, under appropriately chosen step-sizes and averaging, reshuffling achieves matching or better rates relative to with-replacement sampling (Haochen et al., 2018).

2. Algorithmic and Practical Schemes for Data Shuffling

Modern workflows rely on a spectrum of shuffling strategies, each offering trade-offs between randomness, I/O efficiency, and statistical performance. These include:

Method	Randomness	I/O Cost	Memory Buffer
Epoch-Shuffle	Full	Heavy random RW	$O(m)$
Shuffle-Once	Full	Initial random RW	$f_i$0
Sliding-Window/MRS	Partial	Sequential	$f_i$1/$f_i$2
Block-Random (BR-SGD)	Block-level	None (if static)	Local per worker
CorgiPile	Near full	$f_i$3 block reads	$f_i$4

Epoch-shuffle and shuffle-once guarantee ideal convergence by ensuring i.i.d. mini-batch composition at the expense of substantial I/O overhead due to random disk seeks or storage space. Practical shuffling methods such as CorgiPile (Xu et al., 2022), which uses hierarchical block plus tuple-level shuffling, and block-random SGD (Mohan et al., 2019) for distributed exascale environments, improve data access efficiency while preserving near-ideal convergence rates by ensuring randomized block access and in-memory shuffling within manageable buffer windows.

Partial shuffle methods, such as the Partial Shuffle algorithm for LLMs (Press, 2019), construct permutations at sub-sequence or batch-row granularity, injecting minimal breaks in the underlying structure to preserve global dependencies while maintaining sufficient stochasticity for effective optimization. Such methods deliver improved generalization and model performance without excessive computational cost.

Variations in batching strategy, e.g., choosing between global and local shuffling in multi-processor settings (Meng et al., 2017), modulate trade-offs between convergence speed and data movement, especially significant for non-convex optimization where the differences are less severe and speedup scales well with the number of machines.

3. Complexity-Optimal and Structured Shuffling: Advances in Algorithmic Design

Advanced optimization protocols exploit refined shuffling and batching rules to optimize both statistical and computational efficiency. The Adjusted Shuffling SARAH algorithm augments SARAH-type variance-reduced methods with shuffling (cyclic, shuffle-once, or random reshuffling), introducing a dynamic gradient weighting

$f_i$5

within the epoch to equalize component contributions and maximize late-epoch exploration. This achieves, for strongly convex objectives,

$f_i$6

gradient complexity, matching or improving upon classical uniform-i.i.d. VR methods without requiring additional convexity or smoothness assumptions—even for inexact/minibatched variants (Nguyen et al., 14 Jun 2025).

Structured shuffling beyond plain reshuffling further reduces variance and order sensitivity. Block reshuffling with paired reversal (as in the APR algorithm) decomposes variance into block-mean and intra-block terms, strictly lowering the variance constant under mild coherence conditions, while the paired reversal cancels the leading second-order permutation error term, pushing order-sensitivity of the update to $f_i$7 in step size, improving optimization stability (Nguyen et al., 31 Mar 2026).

4. Distributed, Storage-Aware, and Large-Scale Regimes

Distributed learning introduces additional constraints from storage models and communication bottlenecks. In the "no-excess-storage" setting, re-shuffling for each mini-batch assignment induces communication overhead; the information-theoretic optimal reshuffle requires

$f_i$8

data transfers per epoch for $f_i$9 workers, unless higher-order coded multicasting or partial caching is leveraged (Attia et al., 2016).

For out-of-core and cloud storage environments, full random access shuffling is expensive. Hierarchical approaches such as CorgiPile (Xu et al., 2022) and Corgi² (Livne et al., 2023) combine block sampling and in-memory tuple mixing, and may further apply a single offline partial shuffle to decorrelate block content, sharply reducing the block-variance $T$0 controlling the convergence rate. As demonstrated empirically, with buffer ratios as low as $T$1, Corgi² matches the convergence of full offline shuffling (Livne et al., 2023).

Block-random strategies enable scalable in-situ learning on exascale architectures by decoupling mini-batch definition from data placement and only randomizing batch order across processors, yielding statistical efficiency competitive with global shuffling at negligible communication and bandwidth cost (Mohan et al., 2019).

5. Privacy, Differential Privacy, and Shuffling Protocols

Shuffling plays a critical role in privacy amplification for differentially private stochastic gradient descent (DP-SGD). Classical DP-SGD relies on Poisson subsampling, producing strong privacy amplification via randomness in batch selection, tightly tracked by RDP or privacy-loss-distribution accountants. In practice, however, many training pipelines replace Poisson sampling with shuffling for batch formation, assuming similar privacy budgets.

Auditing frameworks reveal that such substitution can lead to underestimated privacy leakage—empirical audits show up to $T$2 or $T$3 more leakage in some regimes (Annamalai et al., 2024). Concretely, while Poisson subsampling grants asymptotically optimal privacy-utility trade-offs, shuffled batching results in correlated sampling and weaker guarantees, with no tight general upper bound on $T$4 available for shuffling (Chua et al., 2024, Annamalai et al., 2024).

Empirical studies further demonstrate that batch-then-shuffle or buffer-based partial shuffling can induce pathological privacy breaches, detected by adversarial auditing via likelihood-ratio distinguishing. These results highlight the necessity of either retaining Poisson subsampling (even at scale via MapReduce algorithms) or employing conservative privacy accounting and avoiding optimistic substitution of i.i.d.-based bounds when shuffling is used (Chua et al., 2024, Annamalai et al., 2024).

6. Applications and Empirical Impact

Data shuffling and stochastic batching materially affect downstream tasks in deep learning, language modeling, vision, and physics-based simulation:

• Large-scale systems such as RINAS (Zhong et al., 2023) achieve $T$5 to $T$6 throughput improvements by exploiting within-batch unordered parallel fetching while preserving strict i.i.d. properties.
• Partial shuffle for LLMs (Press, 2019) achieves new state-of-the-art perplexities on PTB and WikiText-2, maintaining long-range dependencies while introducing essential randomness.
• Distributed and block-based strategies enable near-ideal convergence with minimal data movement in exascale scientific computing and cloud infrastructures (Mohan et al., 2019, Xu et al., 2022).
• Hierarchical and hybrid offline-online shuffling schemes (as in Corgi²) achieve statistical performance indistinguishable from full shuffling at a fraction of the access cost, even for highly homogeneous (low-entropy) data sources (Livne et al., 2023).

7. Current Limitations and Research Directions

Despite strong theoretical and engineering progress, several frontiers remain:

• No universally tight upper bounds for central DP-SGD with shuffling are known; empirical audits expose significant gaps (Chua et al., 2024, Annamalai et al., 2024).
• Impracticalities of full shuffling on TB-scale datasets push ongoing work in storage-aware and buffer-efficient shuffling protocols.
• Structured shuffling rules (e.g., block reshuffling, paired reversal) are only beginning to be systematically explored for optimization constants and robustness (Nguyen et al., 31 Mar 2026).
• In high-heterogeneity distributed/federated settings, minimization of variance via synchronized shuffling and block-aware partitioning remains an active area (Yun et al., 2021).

A plausible implication is that continued advances in both theoretical analysis (especially for non-i.i.d. and privacy-critical protocols) and scalable system integration will determine best practices for stochastic optimization and learning in increasingly heterogeneous, distributed, and privacy-sensitive environments.