Large Batch Resampling (LaBER)

• Large Batch Resampling (LaBER) is a family of methods that first draws a large batch to compute fresh priorities and then down-samples for efficient gradient updates.
• LaBER leverages importance sampling with approximate gradient norms or surrogates to optimize convergence in both deep reinforcement learning and scalable statistical inference.
• Empirical evaluations show that variants like LaBER-mean outperform traditional techniques such as PER and BLB by enhancing sample efficiency while reducing computational overhead.

Searching arXiv for the cited LaBER-related papers to ground the article. Large Batch Resampling (LaBER) denotes a family of large-batch or subset-based resampling procedures in which a computationally cheap first-stage sample is used to construct a more informative second-stage sampling or weighting rule. In the materials associated with this term, the label appears in two closely related senses. In deep reinforcement learning, "Large Batch Experience Replay" introduces LaBER as an importance-sampling-based replay strategy designed to approximate the theoretically optimal sampling distribution for stochastic gradient descent while remaining simple and efficient to implement (Lahire et al., 2021). In massive-data inference, the subsampled double bootstrap is presented as a scalable, subset-based resampling scheme, and "Conceptually, SDB is a LaBER method": it approximates the full-batch bootstrap distribution by aggregating information from many cheap resampling operations on small batches (Sengupta et al., 2015). This suggests that LaBER is best understood as a computational pattern rather than a single domain-specific algorithm.

1. Terminology and conceptual scope

In deep RL, LaBER is explicitly the acronym of Large Batch Experience Replay. The method addresses replay-buffer sampling for DQN, distributional RL, and actor-critic methods by drawing a large batch, computing fresh priorities inside that batch, and then down-sampling a mini-batch for the update (Lahire et al., 2021).

In large-scale statistics, the corresponding LaBER perspective is not a separately named estimator but a way of organizing scalable resampling. The subsampled double bootstrap uses small subsets of size $b \ll n$, one weighted bootstrap resample per subset, and aggregation over many subset-resample pairs. The paper states that SDB is a LaBER method because it approximates a computationally infeasible full-batch bootstrap by many small-batch operations (Sengupta et al., 2015).

A common misconception is that LaBER names only a TD-error prioritization heuristic. The RL formulation is more specific: replay sampling is cast as an importance sampling problem for estimating the gradient, and the resulting method is derived from a theoretically optimal sampling distribution. Another plausible misconception is that large-batch resampling in statistics is merely BLB with different hyperparameters. The SDB construction is structurally different: it uses many subsets and one resample per subset, rather than few subsets and many resamples per subset. This suggests that the unifying feature of LaBER is not a particular loss, estimator, or data structure, but the use of a larger preliminary batch to construct a cheaper and more informative second-stage resampling rule.

2. Importance-sampling foundations in replay-buffer learning

The RL formulation begins from the empirical loss over the replay buffer,

$$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$

with gradient

$$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$

For any distribution $p$ over replay indices with $p_i>0$, this gradient can be written through importance sampling as

$$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$

The corresponding unbiased mini-batch estimator therefore depends on the choice of $p$ (Lahire et al., 2021).

Following Wang et al. (2017), the paper defines a convergence speed

$$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$

For SGD updates, the term depending on $p$ reduces to

$$\mathbb{E}_{i\sim p}[G_i^T G_i] = \frac{1}{N^2}\sum_{i=1}^N \frac{g_i^2}{p_i}, \qquad g_i=\|\nabla_\theta \ell(Q_\theta(x_i),y_i)\|_2.$$

Minimizing this quantity yields the optimal sampling distribution

$$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$0

Thus the theoretically optimal replay distribution is proportional to the per-sample gradient norm (Lahire et al., 2021).

This formulation provides a theoretical foundation for Prioritized Experience Replay. For DQN with an $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$1 loss,

$$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$2

If $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$3 is approximately constant across samples, then $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$4, which matches PER’s intuition. However, the paper also states that there exist cases where sampling proportional to TD errors leads to higher variance than uniform sampling. This clarifies that TD error is only an approximation to the optimal criterion, and that the quality of the approximation depends on the variability of $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$5.

3. Algorithmic structure of Large Batch Experience Replay

LaBER approximates the intractable optimal distribution by operating locally on a large batch. Its core loop is: sample a large batch of size $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$6 from the replay buffer, compute up-to-date priorities within this large batch, down-sample a mini-batch of size $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$7 according to those priorities, and perform the gradient update using an importance-weighted or scaled direction (Lahire et al., 2021).

Within the large batch, priorities can be either exact per-sample gradient norms or a surrogate obtained from the forward pass alone. For DQN and critics with identity last layer and $$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$8 loss, the surrogate satisfies

$$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \ell(Q_\theta(x_i),y_i),$$9

and for Huber loss,

$$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$0

For C51, the surrogate gradient norm is

$$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$1

The resulting within-large-batch distribution is

$$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$2

The paper describes three practical update schemes. LaBER-mean approximates the global average priority by the average over the large batch and yields

$$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$3

LaBER-lazy ignores the changing global scaling, and LaBER-max uses a normalization inspired by PER. Empirically, LaBER-mean provides the best stability and performance and is the recommended variant (Lahire et al., 2021).

A practical distinction from PER is that LaBER does not maintain global priority data structures. Priorities are computed on-the-fly inside the current large batch; no per-transition persistent priority needs to be stored in the replay buffer. The paper characterizes the method as “easy-to-code and efficient” because the only new logic is: sample a large batch, compute priorities, then re-sample a mini-batch.

4. Relation to PER, computational profile, and empirical behavior in RL

The contrast with PER is central. PER uses TD error or loss-based priorities stored over the entire replay buffer, updates only sampled items, and depends on hyperparameters such as $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$4, $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$5, and $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$6. LaBER instead uses approximate gradient norms computed fresh for a large batch at each update, applies non-uniform sampling only over the current large batch, and introduces the large-batch multiplier $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$7 as the main new hyperparameter (Lahire et al., 2021).

This distinction matters both theoretically and empirically. The paper states that PER corresponds to using outdated and approximate proxies for $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$8, and that its variance can even exceed uniform replay. By contrast, LaBER is directly derived from the optimal sampling distribution that minimizes SGD variance and improves convergence speed. The larger the batch multiplier $$\nabla_\theta \mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^N \nabla_\theta \ell(Q_\theta(x_i),y_i).$$9, the more representative the large batch is of the replay buffer; the paper summarizes the empirical trend as “The larger the better” up to computational limits, with $p$0 reported to work very well.

The experimental evaluation covers MinAtar, Atari (ALE) with DQN and C51 using Dopamine, and PyBullet continuous control + LunarLander with SAC and TD3 using Stable-Baselines3. The key observations reported are that LaBER with exact norms and LaBER with surrogate norms achieve very similar performance on MinAtar; LaBER with $p$1 outperforms DQN with batch size $p$2 both in sample efficiency and wall-clock time; LaBER-mean is more stable than LaBER-lazy and LaBER-max; and LaBER consistently outperforms DQN, PER, GER, PER-LaBER, and GER-LaBER in MinAtar, with both higher returns and lower variance. On Atari and PyBullet, LaBER improved return and sample efficiency versus base agents, while PER and GER often failed to consistently beat the base agents without extensive tuning (Lahire et al., 2021).

The computational profile is also explicit. LaBER requires additional forward passes on $p$3 transitions, but the backward pass is still done on only $p$4 samples. For DQN with fully-connected or conv nets, the backward pass is roughly twice as expensive as forward. A plausible implication is that LaBER’s efficiency derives from shifting computation toward cheaper forward evaluations while retaining a mini-batch-sized backward pass.

5. Subsampled Double Bootstrap as large-batch resampling

In the massive-data setting, the statistical problem is to approximate the sampling distribution

$p$5

of a root $p$6 and then estimate a functional $p$7, such as a quantile, a moment, MSE, or CI width. The standard bootstrap becomes prohibitively costly because each resample of size $p$8 typically has $p$9 distinct points, so each evaluation of $p_i>0$0 on a bootstrap resample costs roughly as much as computing $p_i>0$1 on the full dataset (Sengupta et al., 2015).

The subsampled double bootstrap is proposed for both independent data and time series data. In the iid case, SDB proceeds by choosing subset size $p_i>0$2, drawing a random subset $p_i>0$3 of size $p_i>0$4 without replacement, computing the subset estimator, generating one weighted bootstrap resample from that subset with multinomial weights

$p_i>0$5

computing the resample estimator on the weighted subset, and forming the root

$p_i>0$6

The empirical distribution of $p_i>0$7 is then used to estimate $p_i>0$8 (Sengupta et al., 2015).

The relation to BLB is precise. BLB uses subset size $p_i>0$9, $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$0 subsets, and $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$1 weighted resamples per subset; SDB uses subset size $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$2, many subsets $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$3, and one resample per subset. For a fixed time budget, SDB therefore covers more of the dataset. The paper states that the subsampled double bootstrap is superior to BLB in terms of running time, more sample coverage and automatic implementation with less tuning parameters for a given time budget. It also states that SDB can be run “until time runs out,” much like the classical bootstrap, because the time-budget allocation problem over $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$4 and $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$5 is largely removed.

Within the LaBER perspective, SDB has four explicit properties: subsample-based approximation, reduced per-iteration complexity, parallelizability, and time-budget friendly behavior. Each SDB iteration uses only $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$6 distinct observations, so its cost is $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$7 rather than $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$8, and each subset-resample iteration is independent.

6. Asymptotic guarantees, dependent-data extension, and open problems

For iid data, the paper develops SDB in an empirical-process framework. Under the assumptions that $$\mathbb{E}_{i\sim p}\left[\frac{1}{N p_i}\nabla_\theta \ell(Q_\theta(x_i),y_i)\right].$$9 is a $p$0-Donsker class, the measurability condition on $p$1 holds, and $p$2, the SDB empirical process

$p$3

converges conditionally to a centered Gaussian process with covariance

$p$4

Using the functional delta method for the bootstrap, the paper concludes that SDB consistently approximates the distribution of $p$5, and hence that continuous functionals $p$6 of the target distribution are consistently estimated (Sengupta et al., 2015).

The method is also extended to dependent data. For a strictly stationary time series, SDB uses contiguous blocks as subsets and moving block bootstrap (MBB) within each subset. Under assumptions including a permissible VC class, moment conditions $p$7 for some $p$8, polynomially decaying $p$9-mixing coefficients

$$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$0

and rate conditions on block length $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$1 and subset size $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$2, the SDB process converges conditionally to a centered Gaussian process with long-run covariance structure

$$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$3

This establishes first-order consistency for a wide range of functionals under dependence (Sengupta et al., 2015).

The empirical evaluation reinforces the time-budget perspective. In iid linear and logistic regression with $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$4, SDB achieves low error fastest, especially under tight time budgets. In time-series simulations for an AR(1) median and time-series regression, SDB error decreases rapidly and stabilizes earlier than BLB and MBB. In the Central England Temperature example, all methods converge to similar CI-width estimates, but SDB stabilizes faster than BLB, while MBB is slowest and shows more oscillation.

Both the RL and statistical LaBER formulations also leave open problems. In RL, limitations include approximate gradient norms, local approximation of $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$5, and the fact that normalization schemes other than LaBER-mean do not guarantee strict unbiasedness. In SDB, open questions include optimal choice of subset size $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$6, joint choice of $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$7 and block length $$S(p)=-\mathbb{E}_{i\sim p}\left[\|\theta_{t+1}-\theta^*\|_2^2-\|\theta_t-\theta^*\|_2^2\right].$$8, higher-order accuracy relative to BLB, and extensions beyond the current dependence assumptions. This suggests that LaBER methods are defined less by exact unbiasedness than by a deliberate computational trade-off: many cheap, informative resampling operations are used to approximate a full-batch target that is theoretically clear but computationally infeasible.