Dual Batch Size Learning Scheme

• The paper introduces a dual batch size learning scheme that treats batch size as a dynamic control variable to balance efficiency and gradient variance.
• It outlines methods where either concurrent large and small batches or coupled learning rate–batch size schedules optimize throughput while improving generalization.
• Adaptive criteria and optimizer-specific formulations enable dynamic adjustment of batch sizes, reducing training iterations and wall-clock time.

Searching arXiv for the core paper and closely related work on dual/adaptive batch size scheduling. A dual batch size learning scheme is a training strategy in which batch size is treated as a dynamical control variable rather than a fixed hyperparameter. In recent literature, the term appears in at least two closely related senses: as a distributed method that uses a large batch size $B_L$ and a small batch size $B_S$ simultaneously to combine hardware efficiency with better generalization (Lu et al., 30 Sep 2025), and as a coupled learning-rate–batch-size schedule in which reductions in learning rate are replaced by coordinated, compensating changes in both learning rate and batch size (Meterez et al., 16 Oct 2025). This suggests a broader umbrella notion: batch size is co-designed with optimization dynamics, variance control, and systems constraints rather than chosen once at initialization.

1. Conceptual scope and recurring formulations

The modern literature treats dual batch size design as a response to a persistent tension. Large batches improve throughput and reduce the number of parameter updates or communication rounds, but they also reduce gradient variance and can degrade generalization; small batches inject stochasticity that can help optimization escape sharp regions, but they are less hardware-efficient (Lu et al., 30 Sep 2025). A related line of work reframes the same issue as a duality between learning-rate decay and batch-size growth: instead of decaying the learning rate to attenuate stochastic noise, one can preserve or reshape the same optimization trajectory by increasing the batch size (Meterez et al., 16 Oct 2025).

Three formulations recur across the literature.

Formulation	Defining mechanism	Representative paper
Concurrent dual batches	Large-batch and small-batch workers run simultaneously	(Lu et al., 30 Sep 2025)
Stagewise or adaptive ramp	Training starts with smaller batches and enlarges them by stage or trigger	(Yao et al., 2018)
Coupled LR–batch schedule	Learning-rate changes are compensated by batch-size changes	(Meterez et al., 16 Oct 2025)

A dual scheme is therefore not a single algorithmic template. It can mean simultaneous two-batch execution, multi-stage small-to-large schedules, or a coupled rule in which learning rate and batch size are treated as mathematically linked control variables. The common objective is to manage the bias–variance–throughput trade-off more explicitly than fixed-batch training.

2. Learning-rate–batch-size duality

The most explicit formalization of learning-rate–batch duality is Seesaw. If a baseline schedule applies a step decay

$\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$

Seesaw keeps the same decay times but replaces the event by

$$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$

For the common case $\alpha=2$, this becomes

$\eta \to \eta/\sqrt{2},\qquad B \to 2B.$

The paper proves, for noisy linear regression, a finite-sample equivalence between learning-rate decay and batch-size ramp-up for SGD, and extends the same logic to normalized SGD, used as a tractable proxy for Adam in a variance-dominated regime (Meterez et al., 16 Oct 2025).

The invariants depend on the optimizer model. For phased SGD schedules

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$

the relevant equivalence condition is $\alpha\beta=\text{constant}$. For normalized SGD, the corresponding condition is

$\alpha\sqrt{\beta}=\text{constant},$

which implies that Adam-like dynamics are governed by $\eta\sqrt{B}$ rather than $B_S$0. Seesaw chooses the boundary case $B_S$1, described there as the most aggressive non-divergent point. Under that rule, a cosine or step-decay learning-rate schedule can be converted into a dual LR–batch schedule that preserves loss dynamics while reducing serial steps (Meterez et al., 16 Oct 2025).

A complementary theoretical analysis of four SGD schedulers reaches a compatible conclusion. Constant batch size with decaying learning rate does not always minimize the expectation of the full gradient norm of the empirical loss, whereas increasing batch size with decaying learning rate, increasing batch size with increasing learning rate, and increasing batch size with warm-up decaying learning rate do. The same study shows that the latter two schedulers accelerate mini-batch SGD, provided the growth factors satisfy $B_S$2 when $B_S$3 scales like $B_S$4 and $B_S$5 scales like $B_S$6 (Umeda et al., 2024).

Empirically, Seesaw trains 150M, 300M, and 600M-parameter decoder-only models at Chinchilla scale using a constant critical batch size, matches cosine decay at equal FLOPs, and reduces wall-clock time by $B_S$7, close to the continuous-time limit

$B_S$8

The same study reports that naive batch-only ramps underperform cosine and Seesaw, and that more aggressive pairs on the same equivalence line can become unstable (Meterez et al., 16 Oct 2025).

3. Concurrent dual-batch execution in distributed systems

The explicit “dual batch size learning scheme” of Efficient Distributed Training via Dual Batch Sizes and Cyclic Progressive Learning is defined on a parameter-server framework with asynchronous parallel updates. Its central construction is simultaneous use of two batch sizes: a large batch $B_S$9, chosen near the maximal batch size that fits the hardware, and a smaller batch $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$0, introduced to inject more stochasticity and improve generalization (Lu et al., 30 Sep 2025).

Workers are partitioned into two static roles. Large-batch workers always use $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$1; small-batch workers always use $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$2. Let $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$3 and $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$4 be the numbers of large- and small-batch workers, and let $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$5 and $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$6 be their assigned data volumes per epoch, constrained by

$\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$7

The training-time model assumes per-batch time $\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$8, yielding the approximation

$\eta \leftarrow \eta/\alpha,\qquad B \leftarrow B,$9

for a worker processing $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$0 samples with batch size $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$1. A target epoch time $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$2, with $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$3, defines the allowable extra time over the all-large-batch baseline and determines the small-batch workload and the resulting value of $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$4 (Lu et al., 30 Sep 2025).

Because small-batch and large-batch workers see different amounts of data per epoch, the method introduces a model-update factor. Large-batch workers use factor $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$5; small-batch workers use either

$$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$6

or

$$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$7

The linear factor $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$8 is adopted as the default because it gives consistent improvements across the reported configurations (Lu et al., 30 Sep 2025).

This formulation differs from stagewise ramps and LR–batch coupling in an important way: the two batch regimes are concurrent, not sequential. The scheme therefore combines low-variance and high-variance gradient contributions within the same training run. This suggests a distinct systems-oriented interpretation of “dual batch size”: batch size becomes a distributed role assignment, not only a time schedule.

4. Adaptive criteria for batch-size transitions

A large portion of the literature does not fix the switch points in advance, but instead derives them from curvature, variance, error, or validation feedback.

Curvature-driven methods are exemplified by Adaptive Batch Size and Adaptive Batch Size Adversarial. These methods estimate the largest Hessian eigenvalue $$\eta \leftarrow \eta/\sqrt{\alpha},\qquad B \leftarrow \alpha B.$$9 by distributed power iteration and increase both batch size and learning rate by a factor $\alpha=2$0 when

$\alpha=2$1

or when a duration counter reaches $\alpha=2$2. ABSA additionally uses FGSM adversarial training, decreases the adversarial ratio by $\alpha=2$3 at each batch-size increase, and turns adversarial training off at epoch $\alpha=2$4. The paper reports negligible resizing overhead of $\alpha=2$5 of time for ResNet18 ImageNet training, Hessian computations of about $\alpha=2$6 of wall-clock time in ABSA, and improvements of up to $\alpha=2$7 in accuracy and $\alpha=2$8 in the number of SGD iterations (Yao et al., 2018).

Signal-to-noise rules appear in Big Batch SGD. There the practical batch-size condition is

$\alpha=2$9

where $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$0 is the sample variance of per-example gradients over the current batch. If the condition fails, the batch is enlarged and the gradient recomputed. The method thereby adapts batch size to maintain a nearly constant signal-to-noise ratio in the gradient approximation and supports constant stepsizes, Armijo backtracking, and BB-based stepsize selection without classical stepsize decay (De et al., 2016).

Two-scale adaptive scheduling makes the same logic explicit at the phase level. In TSA, SGD runs on an inner timescale with fixed $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$1 and on an outer timescale where $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$2 is enlarged additively or multiplicatively only when a rate term $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$3 falls below a variance term $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$4. In the strongly convex case the recommended constant stepsize is $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$5, while the variance term becomes $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$6. The paper proves exact asymptotic convergence and reports reduced sample complexity relative to fixed-batch SGD (Gao et al., 2020).

History-gradient aided adaptation applies to variance-reduced methods. For SVRG and SPIDER, the outer-loop batch size is chosen from the previous epoch’s average squared stochastic-gradient norm

$\eta \to \eta/\sqrt{2},\qquad B \to 2B.$7

via

$\eta \to \eta/\sqrt{2},\qquad B \to 2B.$8

This removes backtracking and condition checking while preserving the best-known complexity orders for the corresponding variance-reduced algorithms (Ji et al., 2019).

Meta-learned adaptation replaces analytic tests with a validation-driven policy. Arbiter treats batch size as a learnable hyperparameter, generates candidate sizes $\eta \to \eta/\sqrt{2},\qquad B \to 2B.$9, forms a continuous proxy

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$0

and updates the agent from meta-gradients of a validation objective without unrolled optimization. In experiments it acts as a stand-alone scheduler, complements fixed schedules, and regulates variance in hypergradient learning-rate adaptation (MacLellan et al., 2022).

5. Optimizer-specific formulations and invariance

Dual batch size design depends strongly on the optimizer, because the effective noise scale and preconditioning differ between SGD, AdaGrad, Adam, and their variants.

For Adam-like methods, batch-size invariant Adam modifies the second-moment estimator. Standard distributed Adam first averages micro-batch gradients and then squares,

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$1

whereas batch-size invariant Adam first squares micro-gradients and then averages,

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$2

This removes dependence on the number of micro-batches in expectation and avoids the variance-dominance assumption required by square-root learning-rate scaling. The paper reports nearly overlapping training curves across a wide range of global batch sizes for ResNet and ViT settings when linear scaling of $$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$3, $$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$4, and $$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$5 with global batch size is used (Wang et al., 2024).

AdaGrad-specific dual schemes appear in AdAdaGrad and AdAdaGrad-Norm. Here the batch size is increased on the fly through adaptive sampling tests. Under the norm test,

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$6

and under the augmented inner-product test the corresponding statistic uses the variance of $$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$7. The paper proves that AdAdaGrad-Norm converges with high probability at rate $$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$8 to a first-order stationary point, and reports that AdAdaGrad consistently achieves better training loss and validation accuracy than AdAdaGrad-Norm and AdaSGD on MNIST and CIFAR-10 (Lau et al., 2024).

Stagewise Enlargement of Batch Size places SGD, momentum SGD, and AdaGrad in a unified stagewise theory. Its main claim is that a proper batch size is related to the gap between initialization and optimum, and that enlarging batch size geometrically by stage can reduce the number of parameter updates without increasing generalization error. The paper explicitly states that SEBS is suitable for SGD, momentum SGD, and AdaGrad, and theoretically proves equivalence in training error between decreasing learning rate by stage and increasing batch size by stage, while improving iteration complexity (Zhao et al., 2020).

A more direct coupling of adaptive batch size with learning rate appears in CABS. Starting from a gain-per-cost model, the practical rule is

$$(\eta_k,B_k)=\left(\eta\,\alpha^{-k},\,B\,\beta^k\right),$$9

where $\alpha\beta=\text{constant}$0 is the learning rate, $\alpha\beta=\text{constant}$1 is estimated from the stochastic-gradient covariance, and $\alpha\beta=\text{constant}$2 is the current loss. The coupling is explicit: larger $\alpha\beta=\text{constant}$3 implies larger $\alpha\beta=\text{constant}$4, and as $\alpha\beta=\text{constant}$5 decreases the rule increases the batch size, replacing the usual learning-rate decay by direct variance control (Balles et al., 2016).

6. Empirical profile, limitations, and open questions

Empirically, dual batch size schemes have been evaluated under substantially different regimes. Seesaw reports decoder-only Transformers on C4 with the OLMo codebase, trained at Chinchilla scale $\alpha\beta=\text{constant}$6, where Seesaw and cosine curves almost overlap in validation loss versus FLOPs, while Seesaw reaches a given loss about $\alpha\beta=\text{constant}$7 earlier in wall-clock time (Meterez et al., 16 Oct 2025). The explicit distributed dual-batch scheme with cyclic progressive learning reports, for ResNet-18, an improvement of $\alpha\beta=\text{constant}$8 in accuracy and a reduction of training time by $\alpha\beta=\text{constant}$9 on CIFAR-100, and an improvement of $\alpha\sqrt{\beta}=\text{constant},$0 in accuracy with a reduction of training time by $\alpha\sqrt{\beta}=\text{constant},$1 on ImageNet (Lu et al., 30 Sep 2025). AdaBatch reports performance improvements of up to $\alpha\sqrt{\beta}=\text{constant},$2 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than $\alpha\sqrt{\beta}=\text{constant},$3 relative to fixed-batch training (Devarakonda et al., 2017).

The central limitation is that dual scheduling is regime-dependent. Seesaw’s equivalence relies on a variance-dominated regime near critical batch size and underperforms cosine once batch size moves well past the critical batch size (Meterez et al., 16 Oct 2025). The parameter-server dual-batch formulation is tied to asynchronous parallel updates and a load-balancing model fitted to the tested TensorFlow systems; adapting it to all-reduce training requires a different aggregation design (Lu et al., 30 Sep 2025). Curvature-driven methods require Hessian–vector products and additional systems support, even when the measured overhead is modest in the reported ImageNet setting (Yao et al., 2018). Per-sample-gradient methods such as AdAdaGrad and adaptive optimal-batch-size SGD incur nontrivial overhead and currently remain much more tractable in small or medium-scale settings than in transformer-scale pretraining (Lau et al., 2024).

There is also a conceptual controversy over which quantity should remain invariant as batch size changes. For SGD on noisy quadratics, classical heuristics and formal analyses favor invariants related to $\alpha\sqrt{\beta}=\text{constant},$4 or constant signal-to-noise ratio; for Adam-like methods, Seesaw’s normalized-SGD analysis gives $\alpha\sqrt{\beta}=\text{constant},$5, whereas batch-size invariant Adam rejects square-root scaling as a general principle and instead changes the second-moment estimator itself (Meterez et al., 16 Oct 2025). This suggests that “dual batch size learning scheme” is not a single universal scaling law. It is a research program centered on the joint control of stochastic noise, optimizer geometry, and distributed efficiency, with different invariants becoming appropriate under different optimizer models and system assumptions.