Dynamic Batch Size Scheduler

• Dynamic Batch Size Scheduling is an adaptive method that adjusts neural network batch sizes on-the-fly to optimize convergence, utilization, and latency.
• It employs approaches such as statistical feedback, variance-norm tests, meta-learning, and RL-based techniques to balance system constraints and optimization efficiency.
• This methodology is applied in large-scale pretraining, distributed training, and LLM inference, yielding measurable gains in throughput, latency reduction, and energy savings.

A dynamic batch size scheduler is a system that adapts the effective batch size of neural network workloads in real time, optimizing for objectives such as convergence rate, resource utilization, generalization, responsiveness, latency, or energy efficiency. Unlike static scheduling, where batch size is fixed as a hyper-parameter, dynamic scheduling leverages current system state, optimization dynamics, or workload characteristics to adjust batch size on-the-fly. This class of methods has become central to state-of-the-art practice in large-scale LLM pretraining, distributed training on heterogeneous clusters, GPU-based LLM inference serving, and latency-sensitive ML services, as evidenced by numerous recent developments across academia and industry.

1. Principles and Objectives of Dynamic Batch Scheduling

The context for dynamic batch size scheduling has shifted from early SGD-centric heuristics, where batch size was essentially a variance control knob, to modern systems objectives spanning throughput maximization, SLO-compliant latency, energy minimization, and generalization preservation (Zhou et al., 8 Jan 2026, Pang et al., 7 Mar 2025, Lyu et al., 16 Oct 2025, Lau et al., 2024, Belias et al., 5 Nov 2025). Key principles include:

• Adaptive variance control: Dynamic schedulers often seek to maintain the variance of stochastic gradients below a desired threshold relative to the current gradient or loss magnitude, as in variance-norm tests or gradient-noise coupling (Balles et al., 2016, Lau et al., 2024).
• Resource and workload elasticity: Online adjustment of batch size to match available memory, compute throughput, and job size, with mechanisms for reactivity to traffic surges or hardware variations (Pang et al., 7 Mar 2025, Tyagi et al., 2023, Harshbarger et al., 10 Oct 2025).
• Optimization–system co-design: Increasing evidence shows that batch size should be an actively controlled decision variable, not merely a static hyper-parameter, to enable maximal utilization of accelerators and maintain compliance with downstream performance constraints such as SLOs or data efficiency (Pang et al., 7 Mar 2025, Zhou et al., 8 Jan 2026, Umeda et al., 2024).
• Generalization and stability: Some approaches explicitly monitor signals of overfitting or sharp-loss surfaces (e.g., gradient norm or loss variation) and adapt batch growth accordingly to avoid degradation of test accuracy (Lau et al., 2024, Belias et al., 5 Nov 2025, MacLellan et al., 2022).

2. Architectures and Methodologies

The methodologies for dynamic batch size scheduling can be categorized as follows:

• Statistical feedback controllers: These systems use fast online estimators for memory usage, latency, or gradient variance, updating batch size at regular intervals via rule-based or control-theory-inspired policies (Pang et al., 7 Mar 2025, Poduri, 9 Oct 2025, Tyagi et al., 2023, Umeda et al., 2024).
• Variance-norm or gradient-noise coupling: Approaches such as CABS (Coupled Adaptive Batch Size) and DDP-Norm use local or distributed estimates of per-batch gradient variance compared to mean or loss, scaling batch size to maintain a target noise-to-signal ratio (Balles et al., 2016, Lau et al., 2024).
• Meta-learning and hyper-learning: Meta-objective–driven agents (as in Arbiter) optimize batch size by differentiating validation performance through a differentiable proxy without unrolling over entire inner training trajectories (MacLellan et al., 2022).
• MDP/SMDP-based formulations: These methods cast dynamic batching as a (semi-)Markov decision process, optimizing the policy over state (queue, memory, power) and action (batch selection) spaces for long-run average or SLO-targeted cost (Xu et al., 4 Jan 2025).
• Hybrid and RL-based schedulers: Hierarchical systems may use reinforcement learning (PPO or similar) at the cluster/serving layer to jointly select batch size, compute allocation, and other resource control variables, with local algorithmic (greedy or backpressure) policies at individual servers (Harshbarger et al., 10 Oct 2025).

Scheduler Type	Decision Signal	Control Method
Statistical Feedback	Memory, latency, throughput	Rule-based, control loop
Variance-Norm	Gradient norm, variance	Analytical update
Meta-Learning	Validation loss/accuracy	Meta-gradient descent
MDP/SMDP	System queue, power, state	Policy iteration, RVI
RL-based	Cluster/global system state	RL policy + greedy agent

3. Key Algorithms and Mathematical Models

Multiple concrete algorithms underlie dynamic batch size schedulers, each grounded in specific mathematical models:

• CABS (Coupling Adaptive Batch Sizes with Learning Rates):

The batch size $m$ is updated as $m = \alpha \cdot [\operatorname{tr}(\Sigma)/F]$, where $F$ is the current empirical risk and $\operatorname{tr}(\Sigma)$ is the estimated trace of the gradient covariance—effectively matching learning rate to gradient noise (Balles et al., 2016).

• Variance-norm test (as in FSDP-Norm):

The update rule computes $T_k = [\hat{\sigma}_k^2 / b_k]/[\eta^2 \|g_k\|^2]$ and sets $b_{k+1} = \lceil T_k \rceil$ if $T_k > b_k$; this maintains per-coordinate variance within $\eta^2$ of the squared mean gradient (Lau et al., 2024).

• MDP/SMDP Batch Scheduling:

The optimal policy is computed via Bellman equations over the cost of waiting and the cost of service (energy, response time), with discretization and tail-aggregation (abstract costs) to make the theoretical solution tractable (Xu et al., 4 Jan 2025).

• Seesaw rule (loss-equivalence scheduler):

At each scheduled decrease of the learning rate by a factor $\alpha$, the learning rate is instead divided by $\sqrt{\alpha}$ and the batch size is multiplied by $\alpha$, provably matching optimization dynamics (for SGD and normalized variants) and reducing serial training steps (Meterez et al., 16 Oct 2025).

• Fair resource allocation (LLM inference):

Adaptive batch capacity is determined from slack variables derived from SLO-per-token deadlines; batch formation dynamically prioritizes urgent decode, then prefill, using a linear time-cost model and group-ordered admission (Lyu et al., 16 Oct 2025).

4. Applications and System Integration

Dynamic batch size schedulers have found adoption across diverse operational contexts:

• LLM inference serving: By monitoring memory utilization and enforcing SLA-based latency constraints, token throughput is maximized without violating per-decode latency deadlines (Pang et al., 7 Mar 2025, Lyu et al., 16 Oct 2025). Dynamic batching in vLLM and similar systems has yielded 8–28% throughput gains and 22% capacity increases compared to static batching.
• Large-scale pretraining pipelines: Replacement of static "critical batch size" guidelines with WSD-tuned $B_{min}$ and $B_{opt}$, and dynamic tracking of $B_{opt}$ as training accumulates tokens, has reduced total token consumption and improved test accuracy in billion-parameter LLMs (Zhou et al., 8 Jan 2026).
• Distributed training on heterogeneous clusters: Per-worker dynamic batch size control (using P-controller logic) equalizes iteration times and mitigates stragglers or staleness, yielding up to 4× speedup on heterogenous hardware or spot-instance clouds (Tyagi et al., 2023).
• Tabular and data-diff workloads: Dynamic batch- and worker-size scheduling, using memory-safe guards and latency-focused hill-climb, reduces p95 latency by 23–28% and prevents OOMs under strict memory budgets (Poduri, 9 Oct 2025).
• Hybrid serving on slimmable models: PPO policies along with local greedy scheduling allow inference width and batch size to adapt at runtime, trading off energy, latency, and accuracy under variable traffic and hardware utilization (Harshbarger et al., 10 Oct 2025).

5. Empirical Performance and Comparative Analyses

Empirical evaluations consistently report substantial gains in both system and statistical efficiency:

• Throughput and capacity: Memory-aware and SLA-constrained dynamic batching boosts LLM inference throughput by 8–28% and QPS by 22% (Pang et al., 7 Mar 2025). In distributed DDP and FSDP-based LLM training, adaptive batch size reduces required steps by 25% and closes generalization gaps against fixed or warmup heuristics (Lau et al., 2024).
• Training acceleration: Coupled batch–learning rate schedulers (Seesaw, CABS) accelerate minimization of gradient norms and often achieve the same or better test accuracy with substantially fewer serial steps (as much as a 36% reduction in optimizer updates for transformer pretraining) (Meterez et al., 16 Oct 2025, Balles et al., 2016, Umeda et al., 2024).
• QoS and fairness: In LLM serving, resource-fair dynamic schedulers such as FairBatching yield up to 2.3× improvement in TTFT tail latency and up to 54% higher cluster capacity while enforcing per-token SLO constraints (Lyu et al., 16 Oct 2025).
• Robustness and architectural dependency: Adaptive batch-size methods show highly architecture-dependent efficacy, with lightweight and medium-depth models reaping both accuracy and wall-clock speedup. Systematic profiling or baseline characterization of gradient stability is crucial when deploying adaptation in highly stable (e.g., ViT) or unstable (deep ResNet) regimes (Belias et al., 5 Nov 2025).

6. Limitations, Algorithmic Design, and Best Practices

Despite broad applicability, dynamic batch size scheduling faces nontrivial challenges:

• Architecture and workload sensitivity: Not all network architectures benefit equally. Profiling-based characterization and architecture-aware thresholding are essential for robust application (Belias et al., 5 Nov 2025).
• Safety boundaries: Hard caps on memory, latency, and utility need to be embedded to prevent OOMs or SLO violations. Controllers must feature safety guards, cooldowns, and backpressure mitigation (Poduri, 9 Oct 2025, Pang et al., 7 Mar 2025, Lyu et al., 16 Oct 2025).
• Overhead and update frequency: For techniques relying on full-gradient norm checks or distributed variance estimation, computational overhead can be significant. Approximations, test-interval tuning, and sliding-window statistics are recommended for high scalability (Lau et al., 2024).
• Statistical effects and learning rate coupling: Adjusting batch size often necessitates adaptive coupling of the learning rate to preserve variance scaling, convergence properties, and statistical efficiency, especially in noisy or large-batch regimes (Balles et al., 2016, Meterez et al., 16 Oct 2025, Umeda et al., 7 Aug 2025, Umeda et al., 2024).
• Integration complexity: In practice, most dynamic schedulers can be realized as lightweight controller modules with minimal lines of code, hooking into per-iteration logging and scheduling logic (Pang et al., 7 Mar 2025).

7. Research Directions and Open Questions

Ongoing directions and emerging themes in dynamic batch size scheduling include:

• Theory–system alignment: The gap between statistical/mathematical analyses (critical batch size, variance control, finite-sample equivalences) and practical systems design (traffic surges, hardware-induced variability) is an active domain (Zhou et al., 8 Jan 2026, Umeda et al., 2024).
• Unified controllers for multi-task/multi-tenant settings: Extension from single-server to cluster-scale, heterogeneous, and multi-tenant deployments, requiring either hierarchical or distributed/approximate dynamic programming policies (Harshbarger et al., 10 Oct 2025, Xu et al., 4 Jan 2025).
• Fairness and global SLOs: Investigations into fair batching for multi-modal workloads indicate the need for joint optimization of heterogenous SLOs and admission control beyond monolithic objectives (Lyu et al., 16 Oct 2025).
• Optimization for non-SGD methods: Existing analyses are centered around (momentum-based) SGD and Adam; adaptation to more exotic optimizers and reinforcement learning settings is an ongoing research topic (Meterez et al., 16 Oct 2025, Umeda et al., 7 Aug 2025).
• Meta-learned and learned heuristics: Agents directly trained on meta-objectives and validation performance present an avenue for learnable, task- and context-aware dynamic batch scheduling (MacLellan et al., 2022).

Dynamic batch size schedulers represent a mature, theoretically principled, and empirically validated paradigm for improving the efficiency, responsiveness, and fairness of deep learning workloads. Continued progress in this area is central to both the robust scaling of large-scale AI systems and the efficient deployment of inference and training pipelines in production clusters.