Dynamic Batch Size Scheduling

• Dynamic batch size scheduling is an adaptive approach that adjusts batch sizes based on workload dynamics and system heterogeneity to optimize performance.
• It employs optimization-based, control-heuristic, and meta-learning algorithms to balance trade-offs between throughput, latency, convergence, and resource utilization.
• Empirical results demonstrate improved GPU utilization, accelerated convergence, and enhanced system responsiveness compared to traditional static batching methods.

Dynamic batch size scheduling refers to algorithms and system policies that adaptively determine the size of processing or training batches at runtime, rather than using static, pre-defined batch sizes. This approach is increasingly critical in both model training (SGD variants, distributed optimization) and high-throughput model serving (inference microservices, GPU-based online systems), where static batch sizes often result in suboptimal trade-offs between throughput, responsiveness, convergence, hardware utilization, and resource efficiency. Dynamic scheduling mechanisms span rule-based control, statistical and optimization-based heuristics, and meta-learning: they adjust batch sizes in response to observed workload dynamics, optimizer states, system heterogeneity, user-specific service-level constraints, and available parallelism. Recent work provides theoretical, algorithmic, and empirical foundations for dynamic batch scheduling across these varied application settings.

1. Core Objectives and Problem Formulations

Dynamic batch scheduling addresses the fundamental tension between hardware efficiency, algorithmic stability/convergence, and end-user responsiveness:

• Serving (Inference/Online Systems): Larger batches usually yield better GPU/TPU utilization and lower energy-per-request, but cause increased queuing and service latency. The scheduling problem is often posed as a trade-off: minimize a composite cost (e.g., weighted average of mean response time and power usage, subject to SLA constraints) by deciding, at each decision point, when and how many requests or jobs to process together (Xu et al., 4 Jan 2025, Bhimaraju et al., 2023, Chang et al., 24 Jun 2025).
• Training (SGD and Variants): Large batches reduce gradient variance (accelerating convergence, stabilizing updates), but can degrade generalization (by converging to sharp minima) and are often limited by hardware memory. Small batches are noisier but have favorable generalization and can exploit parallel hardware less efficiently. The optimization problem is then to adapt batch size dynamically to match the instantaneous optimization regime, data efficiency, or hardware constraints (Balles et al., 2016, Umeda et al., 7 Aug 2025, Meterez et al., 16 Oct 2025, Lau et al., 2024, Belias et al., 5 Nov 2025, Zhou et al., 8 Jan 2026).
• Distributed/Heterogeneous Systems: Resource heterogeneity (in CPU/GPU speed, bandwidth, spot preemptions) demands variable per-worker batch sizes for straggler mitigation and aggregate throughput maximization (Tyagi et al., 2023, Lin et al., 2019, Bian et al., 2021).
• Multiuser/Edge/Online Arrivals: Asynchronous or deadline-constrained arrivals introduce further batching challenges, requiring joint optimization of batch membership, batch start times, and resource assignment to balance total throughput against individual waiting costs (Cang et al., 2023, Bhimaraju et al., 2023).

2. Methodologies and Scheduling Algorithms

Dynamic batch scheduling algorithms demonstrate substantial structural diversity but share several methodological patterns:

• Optimization-Based Scheduling:
  • SMDP Markov Models: Service systems with parallel hardware are modeled as semi-Markov decision processes. States represent queue lengths; actions are chosen batch sizes or defer/wait; transitions depend on arrival/service time distributions. Optimal policies are computed by minimizing average cost per time (combining latency and energy/power) using Bellman equations, discretization, and offline policy table computation. Advances include tail-state aggregation for tractable scaling (Xu et al., 4 Jan 2025).
  • Shortest-Path and Competitive Online Algorithms: For generic batchable queueing systems, optimal offline schedules map to shortest-paths in acyclic graphs, while online policies (e.g., Wait-Till-α) provide proven competitive ratios with respect to optimality under non-anticipatory constraints (Bhimaraju et al., 2023).
• Control- and Heuristic-Based Algorithms:
  • PID/Proportional Controllers: For distributed training on heterogeneous or transient clusters, per-worker batch sizes are adaptively adjusted to equalize round times, using proportional (and integral/derivative) controllers based on measured throughput and lag (Tyagi et al., 2023).
  • Evolutionary/Genetic Algorithms: Global cluster batch assignments can be orchestrated via lightweight genetic search, optimizing directly for throughput or job completion time under hard memory constraints (Bian et al., 2021).
• Statistical and Gradient Signal-Based Schedulers:
  • Norm-Test and Variance Rules: Schedule batch increases only when the per-batch gradient estimate passes a variance-to-mean norm test (Byrd-inspired), thus tying batch growth to actual noise reduction in the gradient estimation. These rules can be efficiently implemented in both data- and model-parallel distributed settings, and integrated with AdamW for nonconvex objectives (Lau et al., 2024, Balles et al., 2016).
  • Meta-Learning and Hypergradient-Based Policies: Batch size scheduling can itself be automated as a meta-optimization problem, where a neural agent learns to propose batch sizes that minimize validation loss after simulated inner optimization rollouts, using hypergradients without costly unrolled computation (MacLellan et al., 2022).
• Algorithmic Coupling with Learning Rate and Momentum:
  • Coupled Scaling and Ramping: Directly linking batch size with learning rate—e.g., Seesaw replaces each halving of learning rate by a 1/√2 shrink and batch doubling (η→η/√2, B→2B)—preserves statistical dynamics at lower serial cost and achieves sample-equivalence for both SGD and adaptively normalized variants under variance-dominated regimes (Meterez et al., 16 Oct 2025, Kondo et al., 5 Aug 2025).
  • Critical Batch Size and Gradient Norm Feedback: Adaptive joint (batch size, LR) schedulers monitor running gradient norms, stagewise increasing batch and LR whenever the full-gradient magnitude drops below thresholds predicted by analytic SFO complexity or revised E(S) (sample-step) relationships (Umeda et al., 7 Aug 2025, Zhou et al., 8 Jan 2026).
• Rule-Based Adaptive Schedulers:
  • Signal-Driven Policies: Recent systems (e.g., DEBA) use multi-signal scheduling: maintain rolling windows of gradient variance, gradient-norm, and loss statistics; trigger increases, rollbacks, or holds based on architecture-specific thresholds and sufficient cooldown windows for batch normalization and momentum stabilization (Belias et al., 5 Nov 2025).

3. Theoretical Foundations and Guarantees

• Optimality and Convergence Rates:
  • For dynamic SGD under elastic resource counts, properly smoothed momentum compensation and "linear scaling" of the learning rate to batch size yield minimax optimal convergence to stationary points, matching the classic O(1/√T) rates when η∝B and per-step variance adapts smoothly (Lin et al., 2019).
  • Recent Lyapunov–based analyses for SGDM establish a hierarchy: fixed batch and decaying LR yield suboptimal rates with a non-vanishing variance floor; increasing batch size with decaying LR eliminates variance floor and achieves polynomial decay; synchronized increases of both batch size and LR yield provable exponential convergence in the number of update phases, even in the presence of momentum (Kondo et al., 5 Aug 2025).
  • SMDP-based serving policies can be made ε-accurate with respect to the true infinite-state system by aggregating tail-states and introducing calibrated overflow penalties (Xu et al., 4 Jan 2025).
  • Online policies (Wait-Till-α) for batching with unknown arrivals have competitive ratios between 2 and 3, far below their theoretical worst-case in practice (Bhimaraju et al., 2023).
• Sample/Efficiency Equivalence:
  • Formal analysis of batch ramp-up strategies (Seesaw) shows that, provided scaling conditions like α √β=const., schedules that trade learning rate decreases for batch-size increases maintain statistical equivalence in SGD and its variance-dominated analogues for Adam (Meterez et al., 16 Oct 2025).
• Oracle Complexity Minimization:
  • There exists a critical batch size b*, O(1/ε²) in the desired gradient precision ε, that minimizes the stochastic first-order oracle (SFO) complexity in SGD. Schedulers that adapt BS/LR in tandem to track this b* across decreasing gradient regimes provably accelerate convergence (Umeda et al., 7 Aug 2025).
• Generalization and Model Stability:
  • Adaptive batch size schedules exploiting small-to-large growth, monitored by gradient norm or variance, help prevent convergence to sharp minima and can improve test accuracy compared to static- or heuristically ramped baselines, especially in neural architectures with moderate intrinsic stability (Belias et al., 5 Nov 2025, Lau et al., 2024, Zhou et al., 8 Jan 2026).

4. System Architectures and Engineering Considerations

• Distributed and Parallel Systems:
  • Implementation of variable per-worker batch sizes requires fine-grained control at the training script and data pipeline level. Gradient averaging must reweight updates by batch size to maintain statistical correctness (Tyagi et al., 2023, Lin et al., 2019).
  • Batch size elasticity in production queueing systems (for model serving) is efficiently realized by storing precomputed policy lookup tables, batch-action schedules, and supporting batched admission control logic informed by continuous system profiling (Xu et al., 4 Jan 2025, Chang et al., 24 Jun 2025).
  • In shared or resource-heterogeneous clusters, dynamic orchestration engines (such as ONES) must tightly integrate with resource managers, checkpoint/resume machinery, and per-job throughput profilers to implement evolutionary scheduling at small (∼minute) intervals (Bian et al., 2021). Full integration for HPC (e.g., malleable jobs in SLURM) leverages new primitives for real-time resource reallocation and dynamic communicator resizing in MPI (Chadha et al., 2020).
• In Training Frameworks:
  • Effective adaptive batch sizing in PyTorch FSDP/DDP is achieved by augmenting with low-overhead per-batch variance estimation (no individual sample gradients necessary), adjusting chunk sizes with synchronized all-reduce/all-gather operations, and supporting periodic control steps to minimize implementation overhead (Lau et al., 2024, Meterez et al., 16 Oct 2025).
  • Real-world efficiency and stability depend on careful selection of smoothing (EMA) parameters, update frequency, granularity of batch size changes, and robust estimation in the presence of nonstationary loss/gradient statistics (Balles et al., 2016, Belias et al., 5 Nov 2025).
• Serving Architectures:
  • For LLM-inference (e.g., SABER), dynamic scheduling rests on accurate concurrency-to-throughput profiling, per-request admission tests grounded in real-time speed/latency SLAs, and runtime prioritization of requests to maximize fraction of SLA-compliant completions while minimizing GPU idling (Chang et al., 24 Jun 2025).

5. Empirical Results and Performance Impact

• Throughput, Utilization, and Latency:
  • Dynamic scheduling strategies demonstrably outperform static batching and hand-tuned ramp schedules in diverse environments. For instance, ONES reduces mean job completion time by 25% and increases GPU utilization to 82% in multi-job cluster settings (Bian et al., 2021). PID-controlled mini-batching reduces training wall time by 4× compared to uniform batching in highly heterogeneous clusters (Tyagi et al., 2023).
  • SMDP policy-based serving achieves flexible working points on the latency-energy Pareto frontier and adapts efficiently to demand surges (Xu et al., 4 Jan 2025).
• Goodput and SLA Compliance:
  • SABER achieves up to 26% improvement in goodput (fraction of requests within SLA) and 45% reduction in latency variability compared to the best static batching options for CodeLLM serving, without engine modifications or ad-hoc tuning (Chang et al., 24 Jun 2025).
• Training Efficiency and Accuracy:
  • Adaptive batch size schedules (DEBA, DDP/FSDP-Norm) achieve 30–60% acceleration in wall time and consistent 1–7 pp accuracy gains over fixed-batch baselines for architectures in the moderate-stability regime; however, benefit is highly architecture-dependent, requiring explicit stability profiling and cooldown management (Belias et al., 5 Nov 2025, Lau et al., 2024).
  • For large-scale pretraining (LLMs), dynamic schedulers tied to critical batch size or gradient norm tracking outperform both constant batch and warmup-stage heuristics, reducing final loss and improving downstream MMLU by 0.3–0.5% (Zhou et al., 8 Jan 2026).

6. Best Practices, Tuning Guidelines, and Limitations

• Profiling and Calibration: Always profile model stability (gradient variance, norm fluctuation) before deploying adaptive heuristics to select appropriate thresholds or eligibility regimes (Belias et al., 5 Nov 2025).
• Coupled Learning Rate/Batch Growth: Couple batch-size increases directly to learning-rate schedule or observed reductions in gradient norm for maximal efficiency; use smoothed or momentum-based ramps to avoid instability (Balles et al., 2016, Meterez et al., 16 Oct 2025, Umeda et al., 7 Aug 2025).
• Cooldown and Sliding Windows: Enforce cooldown periods (≥5 epochs) between batch-size changes, and use sliding window statistics rather than full-history for timely yet stable adaptation (Belias et al., 5 Nov 2025).
• Resource and System Integration: Employ lightweight, online-optimizing controllers in multi-tenant environments; ensure batch size elasticity is compatible with checkpoint/resume, dynamic reallocation, and per-job memory limits (Bian et al., 2021, Tyagi et al., 2023, Chadha et al., 2020).
• Theoretical Limits and Stability: Do not exceed critical batch size thresholds except where variance-dominated dynamics are guaranteed; aggressive batch-size ramping can cause divergence if learning-rate scaling conditions are violated (Meterez et al., 16 Oct 2025, Kondo et al., 5 Aug 2025).
• Limitations: Estimation of full-gradient norms for theory-optimal adaptation can be expensive in very large models; practical hybrid schemes rely on periodic estimation or surrogate statistics. For ultra-unstable or extremely stable networks, adaptive scheduling may yield little benefit or require aggressive rollback/hold heuristics (Belias et al., 5 Nov 2025).

7. Recent Advances and Research Directions

• Adaptive Scheduling for SLA-Aware Serving: The introduction of mechanisms such as SABER demonstrates the feasibility and necessity of integrating fine-grained SLA-aware admission control and throughput modeling in LLM-based code generation and completion services (Chang et al., 24 Jun 2025).
• Meta-Learned Scheduling and Hyperparameter Transfer: Neural agents (e.g., Arbiter) that meta-learn batch size scheduling from validation-loss gradients offer promising avenues for automated, architecture-agnostic policy search, albeit with additional computational overhead (MacLellan et al., 2022).
• Unified Theoretical Analysis: The Lyapunov-based convergence analysis for SGDM provides clear, phase-separated guidance for designing dynamic batch size and learning rate schedules with provable acceleration and practical recipes for deployment (Kondo et al., 5 Aug 2025).
• Multi-Dimensional, Resource-Coordinated, Edge/Distributed Systems: Joint optimization of batch sizes, batch start times, resource allocation, and network scheduling in edge-inference and multiuser AI services leverages convex relaxations and combinatorial optimization for near-optimal throughput under asynchronous or deadline-driven arrivals (Cang et al., 2023).
• Data-Efficiency Optimization for Pre-training: The use of revised E(S) curves and batch size scheduling rules tightly coupled to the WSD paradigm explain and optimize data-token efficiency for contemporary LLM pretraining, yielding practical algorithms that close the gap between theoretical and observed scaling (Zhou et al., 8 Jan 2026).

---

Dynamic batch size scheduling thus forms a crucial foundation for both scalable training and responsive inference in modern ML systems. Its continued evolution is driven by advances in optimization theory, system and resource heterogeneity, workload diversity, and the ongoing demand for both higher efficiency and adaptable, user-centric service guarantees.