Adaptive Batch Scaling (ABS)

• Adaptive Batch Scaling (ABS) is a dynamic training strategy that adjusts mini-batch sizes based on gradient variance and changing optimization regimes.
• ABS methods use signal-to-noise tests, loss-gap criteria, or curvature estimates to determine when larger batches improve convergence and hardware utilization.
• ABS frameworks extend to distributed systems by integrating system telemetry and dynamic load balancing to improve overall training efficiency.

Searching arXiv for directly relevant papers on adaptive batch size/scaling to ground the article. Search query: adaptive batch size SGD arXiv papers relevant to ABS Adaptive Batch Scaling (ABS) denotes a family of optimization and training strategies that replace a fixed mini-batch size with a time-varying or state-dependent batch size. Across the literature, the motivating tradeoff is stable: smaller batches generally converge in fewer training epochs, whereas larger batches offer more parallelism, lower relative communication overhead, and better hardware utilization. ABS methods therefore aim to preserve the optimization behavior associated with small batches in early or noise-tolerant regimes while exploiting larger batches when variance reduction, synchronization efficiency, or throughput become dominant concerns (Devarakonda et al., 2017, Balles et al., 2016).

1. Historical emergence and problem framing

Early ABS work formulated adaptive batching as a response to the fact that the usefulness of stochasticity changes during training. “Big Batch SGD” proposed to grow the batch whenever the current gradient estimate became too noisy relative to its magnitude, with the explicit goal of maintaining a nearly constant signal-to-noise ratio in the gradient approximation (De et al., 2016). “Coupling Adaptive Batch Sizes with Learning Rates” then made the stronger claim that batch size should not be tuned independently of learning rate, because the variance of the stochastic gradient controls the largest safe step size; its practical rule was presented as a dynamic batch-size adaptation method rather than a preset schedule (Balles et al., 2016).

The term “adaptive batch sizes” was also used in “AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks,” which emphasized the practical systems tradeoff between convergence in epochs and computational efficiency. That work analyzed AlexNet, ResNet, and VGG on CIFAR-10, CIFAR-100, and ImageNet, and reported performance improvements by factors of up to $6.25$ on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than $1\%$ relative to fixed batch sizes (Devarakonda et al., 2017).

Subsequent work broadened ABS in two directions. One direction remained optimizer-centric, deriving batch rules from variance, loss, gradient norm, or curvature. The other direction treated batch size as a systems control variable in distributed or heterogeneous settings, where adaptive scaling determines how efficiently workers, GPUs, or nodes are utilized (Yao et al., 2018, Ma et al., 2021).

2. Optimization principles and canonical formulations

The common mathematical setting is empirical risk minimization. A representative formulation is

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$

with stochastic gradient

$$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$

and update

$w_{k+1}=w_k-\alpha_k g(w_k).$

Under sampling with replacement, the mini-batch covariance scales as

$\mathbf{cov}[g(w)] = \frac{\Sigma(w)}{m},$

so doubling $m$ halves the covariance (Balles et al., 2016). This variance law is the core rationale for ABS.

One major line of work formulates ABS as a signal-to-noise control problem. “Big Batch SGD” used a descent-direction criterion and, in practice, the test

$\|\nabla \ell_B(x_t)\|^2 > \frac{V_B}{|B|},$

where $V_B$ is the sample variance of per-example gradients in the batch. If the inequality failed, the batch was enlarged, in the reported experiments by $\delta_K = 0.1K$, so batch growth was event-driven rather than epoch-driven (De et al., 2016).

A second line derives batch size from optimization-state surrogates. “Improving the convergence of SGD through adaptive batch sizes” proposed

$1\%$0

for PL, strongly convex, and convex regimes, and

$1\%$1

for smooth nonconvex analysis. Its key variance bound was

$1\%$2

so the method enlarged the batch as suboptimality decreased (Sievert et al., 2019).

A closely related but more explicitly two-timescale construction is the TSA scheme, which fixes

$1\%$3

and increases the batch when a transient optimization term $1\%$4 reaches the variance floor $1\%$5. In the strongly convex case, $1\%$6, and the outer-loop decision is to increase $1\%$7 only when variance becomes the bottleneck (Gao et al., 2020). A plausible implication is that much of ABS can be interpreted as replacing step-size decay by controlled variance decay.

3. Methodological families

Variance-aware ABS methods estimate whether the current mini-batch gradient is reliable enough for the next update. CABS is the canonical example: it begins from a gain-per-cost derivation and proposes the practical rule

$1\%$8

implemented online via a diagonal variance estimator

$1\%$9

with $$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$0, exponential moving averages, and clipped batch bounds (Balles et al., 2016). In the same broad family, “Big Batch SGD” uses gradient signal-to-noise tests rather than objective-value surrogates (De et al., 2016).

Loss-driven and error-floor-driven ABS methods use training progress rather than direct variance estimates. The loss-gap and gradient-norm rules of “Improving the convergence of SGD through adaptive batch sizes” belong here, as does TSA, whose batch schedule is piecewise constant and increasing, with additive or multiplicative growth triggered by the comparison $$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$1 (Sievert et al., 2019, Gao et al., 2020).

Curvature-guided ABS adds second-order information. “Large batch size training of neural networks with adversarial training and second-order information” estimates the top Hessian eigenvalue by power iteration and increases both batch size and learning rate when curvature drops sufficiently:

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$2

after which

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$3

Its ABSA variant further combines this mechanism with FGSM-based adversarial training and decays the adversarial ratio as batch size grows (Yao et al., 2018).

Optimizer-coupled ABS extends these ideas beyond plain SGD. AdaBatchGrad retains adaptive batch tests but replaces line search with an AdaGrad-style stepsize

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$4

arguing that this stabilizes test-based ABS under inexact gradient-quality tests (Ostroukhov et al., 2024). AdAdaGrad and AdAdaGradNorm instead integrate progressive batch growth directly with AdaGrad and AdaGrad-Norm, using norm-based or inner-product-based adaptive sampling rules and proving $$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$5-type high-probability stationarity rates for smooth nonconvex objectives (Lau et al., 2024).

A related but narrower idea is batch-size-aware learning-rate scaling rather than direct batch scheduling. AdaScale estimates a gain $$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$6 from gradient variance and uses

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$7

so the optimizer adapts to large-batch training without explicitly choosing a new batch-size schedule (Johnson et al., 2020). This suggests that some papers adjacent to ABS address the same variance–scale coupling through the learning-rate channel instead of the batch-size channel.

4. Systems-level and distributed ABS

In heterogeneous multi-GPU servers, ABS can be used as a load-balancing mechanism. “Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers” treats adaptive batch size scaling as one of three core components of Adaptive SGD, together with dynamic scheduling and normalized model merging. Its per-GPU rule computes

$$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$8

then increases $$\min_{w\in\mathbb{R}^d} F(w) = \frac{1}{M}\sum_{i=1}^M \ell(w;x_i),$$9 for GPUs with $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$0 and decreases $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$1 for GPUs with $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$2, subject to $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$3, while scaling the local learning rate proportionally to the batch-size change (Ma et al., 2021). The stated steady-state objective is that all GPUs perform the same number of model updates between merge points.

A more systems-heavy variant is ABS-SGD, where ABS means Adaptive Batch Size in a delayed synchronous SGD algorithm for heterogeneous GPU clusters. Each worker keeps computing fixed-size reference batches while previous delayed gradients are synchronized; the number of reference batches completed before synchronization ends determines the worker’s actual batch size $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$4. The update uses a one-iteration delayed gradient with compensation,

$$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$5

and the paper reports that, when training ResNet18 with 4 workers, ABS-SGD increases convergence speed by $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$6 on average compared with the best baseline algorithm (Zhou et al., 2023).

DYNAMIX formulates ABS as an online control problem in BSP-style distributed training. It uses PPO to select per-worker batch-size adjustments from the discrete action set

$$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$7

based on network-level, system-level, and training-efficiency state variables, with batch sizes clipped to $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$8. Its reported headline result is up to $$g(w)=\frac{1}{m}\sum_{i\in\mathcal B}\nabla \ell(w;x_i),$$9 improvement in final model accuracy and $w_{k+1}=w_k-\alpha_k g(w_k).$0 reduction in total training time (Dai et al., 9 Oct 2025). This is a distinct systems interpretation of ABS: batch size becomes a runtime control variable driven jointly by optimization and cluster telemetry.

5. Empirical patterns and domains of application

Across supervised image-classification benchmarks, ABS methods consistently exhibit a small-batch-early, larger-batch-later pattern. In CABS, the bottom panels of the reported plots show that the method often holds the minimum batch size for a substantial early phase and then increases batch size roughly linearly later; on SVHN, CIFAR-10, and CIFAR-100 it yields the fastest reduction in training loss and the best final test accuracy, while on MNIST it is not best and may overestimate gradient variance on a very homogeneous dataset (Balles et al., 2016).

AdaBatch evaluates AlexNet, ResNet, and VGG on CIFAR-10, CIFAR-100, and ImageNet, positioning adaptive batch growth as a way to obtain the convergence behavior of small batches with performance similar to large batches (Devarakonda et al., 2017). The second-order ABS/ABSA framework extends this logic to large-batch ImageNet training: for ResNet18 on ImageNet, the paper reports wall-clock training times of $w_{k+1}=w_k-\alpha_k g(w_k).$1 sec for the baseline, $w_{k+1}=w_k-\alpha_k g(w_k).$2 sec for GG, $w_{k+1}=w_k-\alpha_k g(w_k).$3 sec for ABSA, and $w_{k+1}=w_k-\alpha_k g(w_k).$4 sec for ABSA Tuned, while also reporting a resize overhead of only $w_{k+1}=w_k-\alpha_k g(w_k).$5 of total training time in one summary statement (Yao et al., 2018).

ABS has also been extended beyond standard supervised deep learning. In on-policy reinforcement learning, “Scalable On-Policy Reinforcement Learning via Adaptive Batch Scaling” adapts rollout length $w_{k+1}=w_k-\alpha_k g(w_k).$6 rather than the number of environments, so the effective batch is $w_{k+1}=w_k-\alpha_k g(w_k).$7. With default $w_{k+1}=w_k-\alpha_k g(w_k).$8 and rollout range $w_{k+1}=w_k-\alpha_k g(w_k).$9, the effective batch ranges from $\mathbf{cov}[g(w)] = \frac{\Sigma(w)}{m},$0 to $\mathbf{cov}[g(w)] = \frac{\Sigma(w)}{m},$1 samples, and the adaptation signal is Behavioral Divergence, defined as the fraction of reference states whose greedy action changes between policy snapshots (Park, 20 May 2026). The reported Atari-10 comparison states that ABS achieves the best score in 8 out of 10 environments, outperforming both fixed smaller-batch and fixed larger-batch PQN variants (Park, 20 May 2026).

The broad empirical pattern is therefore not a single universal schedule but a family resemblance: early stochasticity is retained, later variance is reduced, and the control signal can come from gradient variance, loss, curvature, update-count imbalance, policy non-stationarity, or system telemetry.

6. Terminological ambiguity, limitations, and recurring caveats

The acronym “ABS” is not semantically stable across the literature. In distributed parameter-server training, “ABS” may denote Adaptive Bounded Staleness rather than Adaptive Batch Scaling. That method adaptively chooses the number of workers the parameter server waits for and refreshes overly stale workers; it is explicitly about staleness control, not batch-size adaptation, even though it changes the number of worker contributions aggregated per round (Tan et al., 2023). This ambiguity is a persistent source of confusion.

A second recurring caveat is that the strongest ABS rules are often easier to analyze than to implement. CABS requires online variance estimation and reports roughly $\mathbf{cov}[g(w)] = \frac{\Sigma(w)}{m},$2 extra computation because estimating gradient second moments duplicates about half of the backward-pass work (Balles et al., 2016). The loss-based ABS theory of “Improving the convergence of SGD through adaptive batch sizes” requires evaluating the training loss on the entire dataset every model update, and its practical RadaDamp variant replaces that exact signal by a rolling average of mini-batch losses (Sievert et al., 2019). AdAdaGrad’s coordinate-wise adaptive batch test is central to its theorem for AdaGrad, but the paper states that the coordinate-wise test is not computationally practical for high-dimensional networks (Lau et al., 2024).

A third caveat is that ABS is not uniformly beneficial. The CABS experiments explicitly report that small fixed batch size 32 and the competitor method outperform it on MNIST (Balles et al., 2016). The RL ABS paper reports strong aggregate gains but also notes environments where a fixed alternative can edge out the adaptive policy in isolated cases (Park, 20 May 2026). Systems-focused methods likewise have scope limits: ABS-SGD is evaluated on CIFAR-10 with ResNet18 and 4 workers on a homogeneous 4-GPU machine with simulated heterogeneity (Zhou et al., 2023), and Adaptive SGD assumes dynamic batch dispatching on heterogeneous multi-GPU servers rather than a static distributed data partition (Ma et al., 2021).

In encyclopedic terms, ABS is therefore best understood not as a single algorithm but as a recurrent design principle in stochastic optimization: batch size is treated as a dynamic variable whose value should track the current noise regime, curvature regime, systems regime, or data-generation regime. The specific control law varies substantially across papers, but the unifying claim is that fixed-batch training discards exploitable structure that changes over the course of optimization (Balles et al., 2016, Devarakonda et al., 2017).