Dynamic Batching Algorithms

• Dynamic batching algorithms are adaptive procedures that assemble operations at runtime using queue observations and control methods to optimize efficiency and minimize latency.
• They leverage real-time data such as request arrival times, resource loads, and instance heterogeneity to dynamically tune batch sizes across domains like ML inference and distributed computing.
• Empirical studies show these techniques can yield up to 28% throughput improvements in LLM inference and significantly reduce kernel launch overhead in dynamic computation graphs.

Dynamic batching algorithms are procedures that adaptively assemble and process groups of operations, data items, or computational requests at runtime to maximize hardware efficiency, reduce overhead, or optimize some domain-specific cost function subject to resource constraints. Unlike static batching—where batch sizes, contents, or launch times are fixed or determined by simple rules—dynamic batching leverages runtime observations (e.g., queue states, resource loads, instance heterogeneity, request arrival times) to make decisions that account for stochastic variation and workload diversity. Dynamic batching is a central theme in machine learning workloads (especially for irregular neural architectures and inference serving), online systems with economies of scale, high-performance distributed computing, and dynamic graph algorithms.

1. Principles and Formal Models

Dynamic batching fundamentally addresses the trade-off between the computational/communication economies realized by processing larger groups and the cost associated with increased latency, complexity, or resource contention.

• Queue and Arrival Models Many settings, particularly in online services, can be formalized as queueing systems with inbound requests. Dynamic batching in these cases involves policies that decide when and how to dispatch jobs, aiming to minimize weighted sums of waiting time and processing cost. For instance, if $a_1 \leq a_2 \leq \dots \leq a_n$ are arrival times, and $f(k)$ is the cost for a batch of size $k$, the objective is to partition arrivals into batches $B_1, ... , B_m$ and schedule batch processing times $s_1 < \dots < s_m$ to minimize

$$J = \frac{1}{n} \left[ \sum_{i=1}^n (d_i - a_i) + \sum_{j=1}^m f(|B_j|) \right]$$

subject to causality ($d_i \geq a_i$) and resource constraints. Online algorithms such as “Wait-Till-$\alpha$” (WTA) release batches when cumulative waiting cost reaches a threshold proportion of batch-processing cost, with competitive ratios that can be tightly analyzed (Bhimaraju et al., 2023).

• Resource- and SLA-Constrained Batching In LLM inference serving, dynamic batching must jointly optimize for GPU-memory usage (e.g., key/value cache size), throughput, and latency SLA violations (Pang et al., 7 Mar 2025). Here, the batch size $b_t$ at scheduling interval $t$ solves:

$f(k)$0

under constraints on memory overflow probability, per-token decoding latency, and hard bounds $f(k)$1.

• Dynamic Computation Graphs For neural networks operating on variable shapes (e.g., trees, graphs), dynamic batching algorithms seek to minimize the number of kernel launches (maximize operation fusion) while respecting both data dependencies and shape compatibility at runtime (Looks et al., 2017, Neubig et al., 2017, Chen et al., 2023, Zha et al., 2019). Formally, this entails partitioning the nodes $f(k)$2 of a computation graph $f(k)$3 into batches subject to dependency and operation-compatibility constraints, an NP-hard scheduling problem (Chen et al., 2023).

2. Core Algorithms and Techniques

Application	Dynamic batching approach	Key trade-offs
GNN Training (Speckhard et al., 2 Feb 2025)	Pad minibatches to dataset-driven #node/edge budgets	Balances kernel recompile frequency vs. memory overhead; dynamic cuts down waste for high-variance graph sizes
Neural architectures (Looks et al., 2017, Neubig et al., 2017, Chen et al., 2023, Zha et al., 2019)	Group ops at equal topo-depth, auto-batch by signature, or via learned FSM	Kernel launches reduced by $f(k)$4; may incur upfront analysis time or gather/scatter costs
LLM inference (Pang et al., 7 Mar 2025)	Real-time memory- and SLA-driven batch size control	Achieves up to $f(k)$5 throughput improvement while maintaining latency bounds
Distributed ML (Tyagi et al., 2023)	Per-worker batch resizing via proportional/PID-style control	Equalizes iteration times in heterogeneous clusters, reducing straggler effects
Online arrivals (Bhimaraju et al., 2023)	Wait-until-accumulated-waiting matches batch-processing gain	Constant-factor competitive ratios for adversarial inputs
Dynamic SpGEMM (Grinten et al., 2022)	Only process delta blocks; aggregate communication around updates	$f(k)$6–$f(k)$7 speedup over static methods when batch is sparse

Common algorithmic elements include:

• Offline budget estimation (e.g., sample dataset to estimate average instance “size”, used as per-batch budget (Speckhard et al., 2 Feb 2025)).
• Online queue monitoring and threshold release (e.g., WTA policies (Bhimaraju et al., 2023)).
• Control-theoretic tuning (e.g., PID- or binary search-style controllers to stabilize batch size adaptively (Pang et al., 7 Mar 2025, Tyagi et al., 2023)).
• Dynamic matching/grouping of runtime operations (e.g., signature- or FSM-driven batching in dynamic computation graphs (Neubig et al., 2017, Chen et al., 2023)).

3. Implementations in Specific Computational Domains

• Deep Graph Models and GNNs Dynamic batching is essential for high-throughput GNN training where graphs have variable size. Library support (Jraph, TensorFlow GNN, PyTorch Geometric) aligns batch assembly with hardware memory layouts and kernel compilation idiosyncrasies (Speckhard et al., 2 Feb 2025). Padding budgets can be computed by sampling, and batch assembly can terminate when adding the next graph would exceed node or edge budgets.
• Dynamic Computation Graphs in Neural Nets Algorithms such as depth-wise grouping (batching operations at the same topological layer and op-type), agenda heuristics (prioritizing ready nodes sharing signatures), and just-in-time compilation with future objects have been used to reduce per-sample overhead (Looks et al., 2017, Zha et al., 2019, Neubig et al., 2017, Chen et al., 2023). FSM-based policies learn optimal batch action sequences using reinforcement learning, achieving up to $f(k)$8 reductions in kernel launches and 1.2–2.5× end-to-end speedups on tree and lattice models (Chen et al., 2023).
• Batch-Dynamic Algorithms in Parallel Computation Numerous dynamic graph problems—connectivity, $f(k)$9-core, spanners, clique counting—feature batch-dynamic algorithms with work and depth bounds dependent on batch size (Acar et al., 2019, Man et al., 2024, Liu et al., 2021, Ghaffari et al., 8 Jul 2025). For example, maintaining connectivity via level-set forests or cluster forests yields amortized per-update work of $k$0 or $k$1 for batch size $k$2 (Acar et al., 2019, Man et al., 2024).
• LLM Inference and Serving In high-throughput serving of LLMs, dynamic batching jointly optimizes expected GPU token throughput and SLA satisfaction by continuously adjusting batch sizes in response to observed queue lengths, memory statistics, and latency metrics. The main architecture couples a probabilistic memory-usage estimator with a latency feedback controller (Pang et al., 7 Mar 2025), leading to observed gains of 8–28% in throughput and 22% in capacity under tight SLA constraints.
• Distributed ML and Heterogeneous Systems Per-worker dynamic batch sizing in BSP/ASP training compensates for diverse node capabilities, minimizing training duration under global batch-size constraints (Tyagi et al., 2023). The core control is a proportional penalty on batch assignments, with task rebalancing only when relative deviation exceeds a threshold.

4. Theoretical Guarantees and Analysis

• Competitive Analysis and Hardness In online queueing, “wait-till-threshold” policies are provably $k$3–$k$4 competitive with the offline optimum under general concave batch-costs, with tight lower bounds for online algorithms (Bhimaraju et al., 2023).
• Complexity Batching policies based on operation scheduling/batching in dynamic graphs have complexity scaling favorably with batch size (e.g., $k$5 for $k$6 dynamic updates (Acar et al., 2020)). Dynamic computation graph batching is generally NP-hard, but approximation via depth- or signature-based heuristics, or by RL-learned FSMs, yields near-optimal speedups empirically (Chen et al., 2023).
• Trade-offs Between Padding, Compilation, and Overhead For GNNs, static-64 batching minimizes gradient update time over long runs, while dynamic batching may help in regimes where frequent recompilation is costly. Padding trade-offs are essential when long-tailed input sizes dominate batch memory (Speckhard et al., 2 Feb 2025).

5. Empirical Evaluation and Performance

Experimental results across domains show clear throughput and latency benefits for dynamic batching approaches:

• GNNs: Maximum observed per-step speedup is $k$7, with static-64 batching outperforming in stable long runs but dynamic batching being competitive when input-size variance is high or recompilation is costly (Speckhard et al., 2 Feb 2025).
• LLM Inference: Dynamic batching yields 2.7–28.2% increases in token throughput and 10–25.9% improvements in concurrent request capacity at bounded latency (Pang et al., 7 Mar 2025).
• Dynamic Computation Graphs: Techniques such as agenda-based batching or FSM-learned policies enable $k$8–$k$9 reduction in kernel launches and up to $B_1, ... , B_m$0 raw throughput gains on tree-structured models (Neubig et al., 2017, Chen et al., 2023, Zha et al., 2019).
• Distributed ML: In heterogeneous clusters, dynamic batch controllers reduce wall time by up to $B_1, ... , B_m$1; even in the presence of resource churn, quickly adapting batch sizes mitigates straggler and staleness effects (Tyagi et al., 2023).

6. Design Considerations, Limitations, and Extensions

• Domain-Specific Choices The optimal batching scheme depends on workload: static approaches (padded to hardware-friendly sizes) are best when input variability is moderate and overhead amortizes; fully dynamic approaches are critical for highly irregular data, tight SLAs, or heterogeneity.
• Analysis Overheads Depth-based and isomorphism analysis in JIT batching schemes introduce upfront costs that may dominate gains for small batches, but can be mitigated by caching and subgraph-level rewrites (Zha et al., 2019).
• Resource Adaptation Real-world deployments must tune design parameters such as batch-size bounds, padding targets, feedback controller gain, and dead-band thresholds to avoid instability or oscillation (Tyagi et al., 2023, Pang et al., 7 Mar 2025).
• Memory Management For dynamic computation graphs, PQ-tree-based scheduling can align memory layouts to reduce data movement, but is currently limited to static subgraph kernels (Chen et al., 2023).
• Hardness and Approximation Determining the optimal batching policy or schedule is generally NP-hard, so practical systems use heuristics, approximations, or learned policies (Chen et al., 2023). Further theoretical work could tighten bounds or design better online/dynamics-aware algorithms.

Dynamic batching remains an active area of research at the interface of high-performance computing, machine learning systems, operations research, and algorithmic scheduling, with evolving methodologies and domain-specific adaptations shaping both theoretical frameworks and practical deployments.