--- title: Dynamic Rebatching Techniques url: https://www.emergentmind.com/topics/dynamic-rebatching type: topic --- # Dynamic Rebatching Techniques Dynamic rebatching is a set of methodologies for adaptively grouping computation tasks—data samples, requests, or computational graph nodes—into executable batches at runtime, rather than statically fixing batch structure or size. The goal is to maximize throughput and hardware efficiency while satisfying domain-specific constraints such as latency, SLAs, or memory limits. This approach is essential for applications with highly dynamic workloads, control-intensive computation graphs, or heterogeneous computational resources, and spans use cases ranging from LLM inference serving and distributed DNN training to adaptive experimentation and dynamic graph deep learning. ## 1. Formal Definitions and Theoretical Models Dynamic rebatching generalizes static batching by making the batch size, composition, and even the batch boundary itself a real-time decision variable. In systems terms, dynamic rebatching can be formulated as a control policy $\pi$ that, at each decision epoch, observes system state $s$—for example, queue length, request ages, memory usage, or graph-node frontiers—and selects a batch action $a$ (e.g., batch size, group of tasks, node signature to execute) to optimize long-run objectives. A canonical formalization is via a semi-Markov decision process (SMDP), as in GPU inference serving, minimizing a weighted sum of average response time and average power consumption: $$ J(\pi) = \limsup_{T \to \infty} \frac{1}{T} \mathbb{E}^\pi \Big[ w_1\int_0^T s(t)dt + w_2\sum_{i:b_i>0} \zeta(b_i) \Big], $$ where $s(t)$ is the instantaneous system state (e.g., queued requests), $b_i$ is the batch size at decision $i$, and $\zeta(b)$ is the energy cost of batch size $b$ [2501.02181]. For computation graphs, the problem is a partitioning of a global job-set $N$ into batches $B_1,\ldots,B_m$ such that all $u,v \in B_k$ are eligible for joint execution (e.g., same operator type and shape), and all data dependencies are honored [2305.10611, 1705.07860, 2302.03851]. In the context of distributed ML, dynamic rebatching policies seek per-worker batch sizes $\{b_i\}$ so that all workers approximately equalize iteration latency $T_i(b_i) \approx T_j(b_j), \forall i,j$ [2305.12213, 2007.11831]. ## 2. Algorithmic Strategies Dynamic rebatching frameworks instantiate a range of algorithmic strategies, often dictated by system constraints and workload characteristics: ### Operator/Node-Level Rebatching for Dynamic Graphs - **Agenda-based Batching:** Maintains an agenda of ready nodes (op type/signature), greedily flushing large compatible batches to minimize kernel launches [1705.07860]. - **FSM/Policy Learning:** Uses finite-state machine encodings of the batching process and tabular Q-learning to discover near-optimal batching policies, with reward structured to both penalize kernels and encourage frontier advancement [2302.03851]. - **Hybrid Static + Dynamic Compilation:** Applies compile-time control-flow and taint analysis to annotate nodes with, e.g., depth or inline phase, reducing dynamic scheduling cost at runtime and ensuring correctness across control-flow divergence [2305.10611]. ### SLA- or Resource-Constrained Batching for Inference Serving - **SMDP-Based Decision Making:** At each service completion or arrival, selects batch action $a$ to minimize a Bellman-style average-cost-to-go equation, typically using tail-cost penalization for tractable finite-state reductions [2501.02181]. - **Memory/LATENCY Feedback Loops:** Uses periodic measurements of queue/memory state and recent latency to adapt batch size per interval, blending a memory-aware scheduler (to avoid OOM) with a latency bracket adjustment under SLA constraints [2503.05248]. - **Admission Control with Predictive Modeling:** Applies per-request SLA feasibility checks, using “Universal Scalability Law” fits of per-request speed under contention to admit additional requests only if their SLA can be met under current resource allocation [2506.19677]. ### Model/Experiment-Aware Dynamic Batching - **Variance/Efficiency-Driven Training Batch Sizing:** Tracks per-microbatch gradient direction change; accumulates until the gradient direction begins to fluctuate, then triggers a parameter update, thereby adapting the batch dynamically with provable empirical gains [2005.02008]. - **Quadrature-Precision-Driven Batch Sizing:** In Bayesian optimization or active learning, frames batch selection as a kernel quadrature problem, adjusting batch size at every round to meet a prescribed worst-case integration error, with explicit LP-based selection [2306.05843, 2003.08579]. - **Batched Bandit Experimentation:** Uses residual-horizon model predictive optimization, dynamically computing batch sizes for each arm in a finite-horizon adaptive experiment, based on a Gaussian dynamic programming approximation [2303.11582]. ## 3. Dynamic Rebatching in Dynamic Graph Frameworks Implementing dynamic rebatching for models with instance-dependent control flow or dynamic dataflow graphs necessitates sophisticated policy and scheduling approaches: - **Signature-Based Grouping:** Nodes/ops are grouped by a deterministic signature (operator type, shape, parameter ID), enabling batch formation only for compatible nodes. Hashing and canonicalization ensure O(1) amortized cost per node [1705.07860, 1904.07421]. - **Depth/Agenda/Sorted-Multiset State Encodings:** Scheduling may use depth-based groupings, agenda-based priorities, or state compressions such as sorted frontier multisets (for FSM-based learners) to efficiently determine which nodes to batch [2302.03851, 1705.07860]. - **PQ-Tree Memory Planning:** Efficient in-place memory allocation layouts are computed offline, ensuring all batched operands are contiguous and aligned, thus avoiding scattered copies or index-gather kernels at runtime [2302.03851]. - **Reinforcement Learning for Policy Discovery:** Batch scheduling policy can be learned via tabular Q-learning over the FSM representation, with a reward structure that provably recovers the first step of an optimal schedule in some cases [2302.03851]. Empirical results on chain, tree, and lattice models (e.g., BiLSTM-Tagger, TreeLSTM, LatticeGRU) show kernel-launch reductions of up to 3.27× over heuristic strategies and throughput speedups up to 3.13× on GPU [2302.03851]. ## 4. Rebatching for SLA and Hardware Utilization In inference serving for LLMs and DNNs, rebatching is essential for controlling latency, memory consumption, and quality-of-service: - **SMDP-derived Control-Limit Policies:** Optimal SMDP policies often have a "threshold" form: wait until $s