---
title: Dynamic Batching Algorithms
url: https://www.emergentmind.com/topics/dynamic-batching-algorithms
type: topic
---

# Dynamic Batching Algorithms

Dynamic batching algorithms are procedures that adaptively assemble and process groups of operations, data items, or computational requests at runtime to maximize hardware efficiency, reduce overhead, or optimize some domain-specific cost function subject to resource constraints. Unlike static batching—where batch sizes, contents, or launch times are fixed or determined by simple rules—dynamic batching leverages runtime observations (e.g., queue states, resource loads, instance heterogeneity, request arrival times) to make decisions that account for stochastic variation and workload diversity. Dynamic batching is a central theme in machine learning workloads (especially for irregular neural architectures and inference serving), online systems with economies of scale, high-performance distributed computing, and dynamic graph algorithms.

## 1. Principles and Formal Models

Dynamic batching fundamentally addresses the trade-off between the computational/communication economies realized by processing larger groups and the cost associated with increased latency, complexity, or resource contention.

- **Queue and Arrival Models**  
  Many settings, particularly in online services, can be formalized as queueing systems with inbound requests. Dynamic batching in these cases involves policies that decide when and how to dispatch jobs, aiming to minimize weighted sums of waiting time and processing cost. For instance, if $a_1 \leq a_2 \leq \dots \leq a_n$ are arrival times, and $f(k)$ is the cost for a batch of size $k$, the objective is to partition arrivals into batches $B_1, ... , B_m$ and schedule batch processing times $s_1 < \dots < s_m$ to minimize
  $$
  J = \frac{1}{n} \left[ \sum_{i=1}^n (d_i - a_i) + \sum_{j=1}^m f(|B_j|) \right]
  $$
  subject to causality ($d_i \geq a_i$) and resource constraints. Online algorithms such as “Wait-Till-$\alpha$” (WTA) release batches when cumulative waiting cost reaches a threshold proportion of batch-processing cost, with competitive ratios that can be tightly analyzed [2309.16911].

- **Resource- and SLA-Constrained Batching**  
  In LLM inference serving, dynamic batching must jointly optimize for GPU-memory usage (e.g., key/value cache size), throughput, and latency SLA violations [2503.05248]. Here, the batch size $b_t$ at scheduling interval $t$ solves:
  $$
  \max_{b_t}\ \Phi(t) = \frac{b_t}{\tau_{\text{step}}(b_t)}
  $$
  under constraints on memory overflow probability, per-token decoding latency, and hard bounds $B_{\min} \leq b_t \leq B_{\max}$.

- **Dynamic Computation Graphs**  
  For neural networks operating on variable shapes (e.g., trees, graphs), dynamic batching algorithms seek to minimize the number of kernel launches (maximize operation fusion) while respecting both data dependencies and shape compatibility at runtime [1702.02181, 1705.07860, 2302.03851, 1904.07421]. Formally, this entails partitioning the nodes $V$ of a computation graph $G=(V,E)$ into batches subject to dependency and operation-compatibility constraints, an NP-hard scheduling problem [2302.03851].

## 2. Core Algorithms and Techniques

| Application       | Dynamic batching approach                           | Key trade-offs                                     |
|-------------------|-----------------------------------------------------|----------------------------------------------------|
| GNN Training [2502.00944]  | Pad minibatches to dataset-driven #node/edge budgets         | Balances kernel recompile frequency vs. memory overhead; dynamic cuts down waste for high-variance graph sizes  |
| Neural architectures [1702.02181, 1705.07860, 2302.03851, 1904.07421] | Group ops at equal topo-depth, auto-batch by signature, or via learned FSM | Kernel launches reduced by $>10\times$; may incur upfront analysis time or gather/scatter costs   |
| LLM inference [2503.05248] | Real-time memory- and SLA-driven batch size control | Achieves up to $28\%$ throughput improvement while maintaining latency bounds  |
| Distributed ML [2305.12213] | Per-worker batch resizing via proportional/PID-style control | Equalizes iteration times in heterogeneous clusters, reducing straggler effects  |
| Online arrivals [2309.16911] | Wait-until-accumulated-waiting matches batch-processing gain | Constant-factor competitive ratios for adversarial inputs  |
| Dynamic SpGEMM [2202.08808] | Only process delta blocks; aggregate communication around updates | $2\times$–$6\times$ speedup over static methods when batch is sparse  |

Common algorithmic elements include:
- Offline budget estimation (e.g., sample dataset to estimate average instance “size”, used as per-batch budget [2502.00944]).
- Online queue monitoring and threshold release (e.g., WTA policies [2309.16911]).
- Control-theoretic tuning (e.g., PID- or binary search-style controllers to stabilize batch size adaptively [2503.05248, 2305.12213]).
- Dynamic matching/grouping of runtime operations (e.g., signature- or FSM-driven batching in dynamic computation graphs [1705.07860, 2302.03851]).

## 3. Implementations in Specific Computational Domains

- **Deep Graph Models and GNNs**  
  Dynamic batching is essential for high-throughput GNN training where graphs have variable size. Library support (Jraph, TensorFlow GNN, PyTorch Geometric) aligns batch assembly with hardware memory layouts and kernel compilation idiosyncrasies [2502.00944]. Padding budgets can be computed by sampling, and batch assembly can terminate when adding the next graph would exceed node or edge budgets.

- **Dynamic Computation Graphs in Neural Nets**  
  Algorithms such as depth-wise grouping (batching operations at the same topological layer and op-type), agenda heuristics (prioritizing ready nodes sharing signatures), and just-in-time compilation with future objects have been used to reduce per-sample overhead [1702.02181, 1904.07421, 1705.07860, 2302.03851]. FSM-based policies learn optimal batch action sequences using reinforcement learning, achieving up to $3\times$ reductions in kernel launches and 1.2–2.5× end-to-end speedups on tree and lattice models [2302.03851].

- **Batch-Dynamic Algorithms in Parallel Computation**  
  Numerous dynamic graph problems—connectivity, $k$-core, spanners, clique counting—feature batch-dynamic algorithms with work and depth bounds dependent on batch size [1903.08794, 2411.11781, 2106.03824, 2507.06338]. For example, maintaining connectivity via level-set forests or cluster forests yields amortized per-update work of $O(\log n \log(1+n/\Delta))$ or $O(\log^2 n)$ for batch size $\Delta$ [1903.08794, 2411.11781].

- **LLM Inference and Serving**  
  In high-throughput serving of LLMs, dynamic batching jointly optimizes expected GPU token throughput and SLA satisfaction by continuously adjusting batch sizes in response to observed queue lengths, memory statistics, and latency metrics. The main architecture couples a probabilistic memory-usage estimator with a latency feedback controller [2503.05248], leading to observed gains of 8–28% in throughput and 22% in capacity under tight SLA constraints.

- **Distributed ML and Heterogeneous Systems**  
  Per-worker dynamic batch sizing in BSP/ASP training compensates for diverse node capabilities, minimizing training duration under global batch-size constraints [2305.12213]. The core control is a proportional penalty on batch assignments, with task rebalancing only when relative deviation exceeds a threshold.

## 4. Theoretical Guarantees and Analysis

- **Competitive Analysis and Hardness**  
  In online queueing, “wait-till-threshold” policies are provably $2$–$3$ competitive with the offline optimum under general concave batch-costs, with tight lower bounds for online algorithms [2309.16911].
- **Complexity**  
  Batching policies based on operation scheduling/batching in dynamic graphs have complexity scaling favorably with batch size (e.g., $O(k \log(1+n/k))$ for $k$ dynamic updates [2002.05129]).  
  Dynamic computation graph batching is generally NP-hard, but approximation via depth- or signature-based heuristics, or by RL-learned FSMs, yields near-optimal speedups empirically [2302.03851].

- **Trade-offs Between Padding, Compilation, and Overhead**  
  For GNNs, static-64 batching minimizes gradient update time over long runs, while dynamic batching may help in regimes where frequent recompilation is costly. Padding trade-offs are essential when long-tailed input sizes dominate batch memory [2502.00944].

## 5. Empirical Evaluation and Performance

Experimental results across domains show clear throughput and latency benefits for dynamic batching approaches:
- **GNNs**: Maximum observed per-step speedup is $\sim33\%$, with static-64 batching outperforming in stable long runs but dynamic batching being competitive when input-size variance is high or recompilation is costly [2502.00944].
- **LLM Inference**: Dynamic batching yields 2.7–28.2% increases in token throughput and 10–25.9% improvements in concurrent request capacity at bounded latency [2503.05248].
- **Dynamic Computation Graphs**: Techniques such as agenda-based batching or FSM-learned policies enable $6$–$20\times$ reduction in kernel launches and up to $6\times$ raw throughput gains on tree-structured models [1705.07860, 2302.03851, 1904.07421].
- **Distributed ML**: In heterogeneous clusters, dynamic batch controllers reduce wall time by up to $4\times$; even in the presence of resource churn, quickly adapting batch sizes mitigates straggler and staleness effects [2305.12213].

## 6. Design Considerations, Limitations, and Extensions

- **Domain-Specific Choices**  
  The optimal batching scheme depends on workload: static approaches (padded to hardware-friendly sizes) are best when input variability is moderate and overhead amortizes; fully dynamic approaches are critical for highly irregular data, tight SLAs, or heterogeneity.
- **Analysis Overheads**  
  Depth-based and isomorphism analysis in JIT batching schemes introduce upfront costs that may dominate gains for small batches, but can be mitigated by caching and subgraph-level rewrites [1904.07421].
- **Resource Adaptation**  
  Real-world deployments must tune design parameters such as batch-size bounds, padding targets, feedback controller gain, and dead-band thresholds to avoid instability or oscillation [2305.12213, 2503.05248].
- **Memory Management**  
  For dynamic computation graphs, PQ-tree-based scheduling can align memory layouts to reduce data movement, but is currently limited to static subgraph kernels [2302.03851].
- **Hardness and Approximation**  
  Determining the optimal batching policy or schedule is generally NP-hard, so practical systems use heuristics, approximations, or learned policies [2302.03851]. Further theoretical work could tighten bounds or design better online/dynamics-aware algorithms.

Dynamic batching remains an active area of research at the interface of high-performance computing, machine learning systems, operations research, and algorithmic scheduling, with evolving methodologies and domain-specific adaptations shaping both theoretical frameworks and practical deployments.

Source: https://www.emergentmind.com/topics/dynamic-batching-algorithms