---
title: Bucket-Based Batching Strategies
url: https://www.emergentmind.com/topics/bucket-based-batching
type: topic
---

# Bucket-Based Batching Strategies

Bucket-based batching refers to a class of strategies for grouping elements (such as requests, inputs, or jobs) into batches based on similarity in key properties—most often sequence length, execution time, or size. By operating on length-homogeneous or workload-homogeneous subsets ("buckets"), these methods exploit structural characteristics of the data to improve computational efficiency, resource utilization, and predictability in both training and inference settings for machine learning systems. Unlike fully sorted or strictly random batching, bucket-based batching enables fine-grained trade-offs between padding overhead, memory footprint, scheduling flexibility, and shuffling/randomization requirements.

## 1. Core Principles and Bucket Formation

Bucket-based batching operates by partitioning incoming data or requests into several contiguous intervals according to a salient feature, most commonly sequence length or predicted execution time. Each interval defines a "bucket," within which elements are assumed to be sufficiently homogeneous to allow batching with minimal artificial resource overhead.

For example, in the context of LLM inference [2507.17120, 2412.04504], the process is as follows:

- Maintain a set of buckets $B = \{[L_b, U_b)\}$ that partition the range $[0, L_{\mathrm{max}})$.
- Upon arrival, a request $r$ with sequence length $S$ is assigned to the unique bucket $b \in B$ such that $L_b \le S < U_b$.
- Similar principles are applied in speech enhancement training [2301.10587], where utterances are assigned to buckets based on duration; bucket boundaries $\{\tau_0, \dots, \tau_M\}$ are chosen via uniform or quantile-based spacing.

Within each bucket, elements are relatively uniform in resource requirements, reducing resource wastage from padding and underutilization. The optimal bucket boundaries (for expected padding waste minimization) satisfy

$$
U_b^* = \frac{\int_{L_b}^{U_b} S f(S) dS}{\int_{L_b}^{U_b} f(S) dS}
$$

where $f(S)$ is the density of arrival lengths [2507.17120]. In practice, boundaries may be estimated by bisecting the interval or distributing elements by quantiles [2301.10587, 2412.04504].

## 2. Padding Overhead and Memory Efficiency

The key computational benefit of bucket-based batching is the reduction of zero-padding overhead, quantified for neural sequence tasks as

$$
\mathrm{Waste\_Ratio} = \frac{S_{\max} - S_{\mathrm{avg}}}{S_{\max}}
$$

where $S_{\max}$ is the maximum sequence length in a batch, and $S_{\mathrm{avg}}$ is the average [2507.17120]. Across all buckets, expected padding waste is

$$
\mathbb{E}[\mathrm{Waste}] = \sum_{b=1}^K \int_{L_b}^{U_b} \left(1 - \frac{S}{U_b}\right) f(S) dS
$$

A direct empirical comparison for end-to-end speech enhancement [2301.10587] reveals that random batching with a large data budget incurs a zero-padding rate (ZPR) of approximately 24.1%, sorted (by length) only 0.4%, and bucket batching (with $M=10$) 5.2%. For LLM inference, bucket-based batching reduces memory footprint due to the following batch-wise KV-cache requirement:

$$
\mathrm{Memory}_{\mathrm{KV\ cache}} = 2 \times L H D B \times S_{\max} \times N
$$

where $L$ is the number of layers, $H$ attention heads, $D$ the per-head dimension, $B$ bytes per element, and $N$ batch size [2507.17120].

## 3. Dynamic Batch Size and Online Adaptation

To prevent out-of-memory (OOM) failures and improve resource utilization under dynamic workloads, dynamic batch size adjustment is tightly integrated with bucket-based batching. The available GPU memory is monitored, and a safety budget is enforced (e.g., $M_{\mathrm{safe}} = 0.9 \times M_{\mathrm{remain}}$). The maximum batch size $N_{\max}$ within a bucket is selected to be the largest $N$ such that the sum of the memory required by all $N$ elements does not exceed $M_{\mathrm{safe}}$ [2507.17120]:

$$
N_{\max} = \max \left\{N \in \mathbb{N} : 2L H D B \sum_{i=1}^N S_i \leq M_{\mathrm{safe}} \right\}
$$

Additionally, batch sizes can be made dynamic with respect to aggregate input duration, notably in speech applications [2301.10587]:

$$
B_b = \left\lfloor \frac{C}{\bar T_b} \right\rfloor
$$

where $C$ is the total data budget per batch and $\bar T_b$ the average duration in bucket $b$.

In dynamic environments (e.g., online LLM serving), buckets are continually split or merged in response to workload pressure. Merge all requests into a single bucket when load is low; when bucket queue sizes exceed certain thresholds, bisect existing buckets to maintain balanced request distribution [2507.17120].

## 4. Scheduling for Latency, Throughput, and SLO Response

Priority-aware scheduling is an essential complement to bucket-based batching, facilitating both throughput maximization and Service Level Objective (SLO) compliance. Within each bucket:

- Throughput-oriented tasks may use Shortest-Job-First (SJF) or Longest-Job-First (LJF).
- Latency-sensitive online tasks are scheduled by arrival order within bucket, with buckets reprioritized if deadlines approach [2507.17120].
- Separating prefill and decoding phases (as in LLM serving) allows for further control over scheduling, preventing head-of-line blocking.
- SLO tracking is global, and buckets can be dynamically reprioritized to raise requests nearing deadline, maintaining high SLO attainment rates.

In queueing-theoretic terms, a multi-bin batching algorithm achieves a theoretical throughput $c_k = B / \mathbb{E}[t_{\mathrm{service},k}]$, where $k$ is the number of bins (buckets) [2412.04504]. For practical systems, $k=3$–$5$ often suffices to capture the majority of gain with minimal latency increase.

## 5. Empirical Results and Performance Benchmarks

Empirical evaluations across modalities demonstrate substantial resource and performance improvements:

- In LLM inference [2507.17120], BucketServe increased throughput up to $3.58\times$ versus UELLM and $1.31\times$ versus DistServe (offline, LLaMA-2-13B), with average GPU utilization $81.66\%$. At 80% SLO attainment, load capacity (server RPS) improved $1.93\times$ (mixed workload) and $1.975\times$ (Alpaca) over UELLM.
- Bucket-based batching adds negligible pipeline overhead, accounting for less than $1\%$ of end-to-end latency.
- For training with variable-length speech data, bucket batching with $M=10$ and dynamic batch sizing reduced epoch runtime by approximately $18\%$ versus random batching at large batch budgets, with ZPR held near the sorted baseline [2301.10587].
- Queueing-theoretic analysis for LLM inference shows that increasing the number of bins improves throughput steadily: at $k=4$, 20–30% gains are typical; at $k=8$, gains reach up to 40–60%; with $k=32$, throughput is up to 70% higher than for $k=1$. Even with misclassification of bin assignments at error rates $p_e = 20$–$30\%$, bucket-based batching outperforms $k=1$ (i.e., no binning) [2412.04504].

| System/Setting                        | Padding Overhead (ZPR) | Throughput Boost | GPU Memory | End-to-End Latency Overhead |
|---------------------------------------|------------------------|------------------|------------|-----------------------------|
| Speech (Sorted) $C=128$s [2301.10587] | 0.4%                   | --               | 20.1 GB    | --                          |
| Speech (Bucket, $M=10$)               | 5.2%                   | 1.18x            | 20.1 GB    | --                          |
| Speech (Random)                       | 24.1%                  | 1.0x             | 20.2 GB    | --                          |
| LLM (BucketServe) [2507.17120]        | Minimized              | $3.58\times$     | --         | <1% pipeline overhead       |

## 6. Theoretical Foundations and Generalizations

Bucket-based batching principles extend to parallel load balancing and dynamic batching in online settings. The batched allocation model [2203.13902] considers allocating $m$ jobs into $n$ servers in batches of size $b$. When jobs can have variable weights, allocation processes (e.g., Two-Choice, $(1+\beta)$, quantile, graphical) are analyzed. High-probability bounds for the load gap $G(b, n)$ are established:

$$
G(b,n) = O \left(\frac{b}{n} + \log n \right)
$$

for $b \in [n, n^3]$, with tightness proofs. Theoretical and experimental results suggest that less "aggressive" allocation strategies can outperform more powerful schemes in high-parallelism regimes—a non-monotonic effect.

In queueing-theoretic analysis of multi-bin batching for LLMs [2412.04504], to achieve throughput $\geq c_{\max}-\epsilon$, the number of bins required scales as $O(1/\epsilon)$. Analytically, optimal bin boundaries (under uniform input lengths) are linear interpolations between $l_{\mathrm{min}}$ and $l_{\mathrm{max}}$.

## 7. Application Domains and Practical Guidelines

Bucket-based batching is broadly applicable in LLM inference serving, neural sequence modeling, online service scheduling, and any scenario with variable-size or unpredictable workloads. Practical recommendations include:

- Select bucket boundaries either via profiling (e.g., cumulative distribution function quantiles) or uniform partitioning.
- When training, use 10 buckets for length bins and batch according to constant total duration, balancing randomization (for SGD) and resource locality [2301.10587].
- For online inference, use dynamic splitting and merging, and priority-aware scheduling to optimize both throughput and SLO attainment [2507.17120].
- In scenarios involving economies of scale and waiting costs, online batching can be driven by threshold policies such as Wait-Till-α, ensuring constant-factor competitiveness [2309.16911].

By integrating bucket-based batching, practitioners achieve near-optimal resource efficiency and throughput, adaptive scaling under nonstationary demand, and robust latency compliance, all within a rigorous theoretical framework.

Source: https://www.emergentmind.com/topics/bucket-based-batching