Bucket-Based Batching Strategies

• Bucket-based batching is a technique that groups data based on sequence length or workload features to minimize padding and enhance resource utilization.
• It enables dynamic batch sizing and online adaptation, reducing memory overhead while ensuring efficient throughput and latency compliance.
• Empirical benchmarks show significant performance gains in LLM inference and speech processing versus random or fully sorted batching.

Bucket-based batching refers to a class of strategies for grouping elements (such as requests, inputs, or jobs) into batches based on similarity in key properties—most often sequence length, execution time, or size. By operating on length-homogeneous or workload-homogeneous subsets ("buckets"), these methods exploit structural characteristics of the data to improve computational efficiency, resource utilization, and predictability in both training and inference settings for machine learning systems. Unlike fully sorted or strictly random batching, bucket-based batching enables fine-grained trade-offs between padding overhead, memory footprint, scheduling flexibility, and shuffling/randomization requirements.

1. Core Principles and Bucket Formation

Bucket-based batching operates by partitioning incoming data or requests into several contiguous intervals according to a salient feature, most commonly sequence length or predicted execution time. Each interval defines a "bucket," within which elements are assumed to be sufficiently homogeneous to allow batching with minimal artificial resource overhead.

For example, in the context of LLM inference (Zheng et al., 23 Jul 2025, Guldogan et al., 2024), the process is as follows:

• Maintain a set of buckets $B = \{[L_b, U_b)\}$ that partition the range $[0, L_{\mathrm{max}})$.
• Upon arrival, a request $r$ with sequence length $S$ is assigned to the unique bucket $b \in B$ such that $L_b \le S < U_b$.
• Similar principles are applied in speech enhancement training (Gonzalez et al., 2023), where utterances are assigned to buckets based on duration; bucket boundaries $\{\tau_0, \dots, \tau_M\}$ are chosen via uniform or quantile-based spacing.

Within each bucket, elements are relatively uniform in resource requirements, reducing resource wastage from padding and underutilization. The optimal bucket boundaries (for expected padding waste minimization) satisfy

$$U_b^* = \frac{\int_{L_b}^{U_b} S f(S) dS}{\int_{L_b}^{U_b} f(S) dS}$$

where $f(S)$ is the density of arrival lengths (Zheng et al., 23 Jul 2025). In practice, boundaries may be estimated by bisecting the interval or distributing elements by quantiles (Gonzalez et al., 2023, Guldogan et al., 2024).

2. Padding Overhead and Memory Efficiency

The key computational benefit of bucket-based batching is the reduction of zero-padding overhead, quantified for neural sequence tasks as

$\mathrm{Waste\_Ratio} = \frac{S_{\max} - S_{\mathrm{avg}}}{S_{\max}}$

where $[0, L_{\mathrm{max}})$0 is the maximum sequence length in a batch, and $[0, L_{\mathrm{max}})$1 is the average (Zheng et al., 23 Jul 2025). Across all buckets, expected padding waste is

$[0, L_{\mathrm{max}})$2

A direct empirical comparison for end-to-end speech enhancement (Gonzalez et al., 2023) reveals that random batching with a large data budget incurs a zero-padding rate (ZPR) of approximately 24.1%, sorted (by length) only 0.4%, and bucket batching (with $[0, L_{\mathrm{max}})$3) 5.2%. For LLM inference, bucket-based batching reduces memory footprint due to the following batch-wise KV-cache requirement:

$[0, L_{\mathrm{max}})$4

where $[0, L_{\mathrm{max}})$5 is the number of layers, $[0, L_{\mathrm{max}})$6 attention heads, $[0, L_{\mathrm{max}})$7 the per-head dimension, $[0, L_{\mathrm{max}})$8 bytes per element, and $[0, L_{\mathrm{max}})$9 batch size (Zheng et al., 23 Jul 2025).

3. Dynamic Batch Size and Online Adaptation

To prevent out-of-memory (OOM) failures and improve resource utilization under dynamic workloads, dynamic batch size adjustment is tightly integrated with bucket-based batching. The available GPU memory is monitored, and a safety budget is enforced (e.g., $r$0). The maximum batch size $r$1 within a bucket is selected to be the largest $r$2 such that the sum of the memory required by all $r$3 elements does not exceed $r$4 (Zheng et al., 23 Jul 2025):

$r$5

Additionally, batch sizes can be made dynamic with respect to aggregate input duration, notably in speech applications (Gonzalez et al., 2023):

$r$6

where $r$7 is the total data budget per batch and $r$8 the average duration in bucket $r$9.

In dynamic environments (e.g., online LLM serving), buckets are continually split or merged in response to workload pressure. Merge all requests into a single bucket when load is low; when bucket queue sizes exceed certain thresholds, bisect existing buckets to maintain balanced request distribution (Zheng et al., 23 Jul 2025).

4. Scheduling for Latency, Throughput, and SLO Response

Priority-aware scheduling is an essential complement to bucket-based batching, facilitating both throughput maximization and Service Level Objective (SLO) compliance. Within each bucket:

• Throughput-oriented tasks may use Shortest-Job-First (SJF) or Longest-Job-First (LJF).
• Latency-sensitive online tasks are scheduled by arrival order within bucket, with buckets reprioritized if deadlines approach (Zheng et al., 23 Jul 2025).
• Separating prefill and decoding phases (as in LLM serving) allows for further control over scheduling, preventing head-of-line blocking.
• SLO tracking is global, and buckets can be dynamically reprioritized to raise requests nearing deadline, maintaining high SLO attainment rates.

In queueing-theoretic terms, a multi-bin batching algorithm achieves a theoretical throughput $S$0, where $S$1 is the number of bins (buckets) (Guldogan et al., 2024). For practical systems, $S$2–$S$3 often suffices to capture the majority of gain with minimal latency increase.

5. Empirical Results and Performance Benchmarks

Empirical evaluations across modalities demonstrate substantial resource and performance improvements:

• In LLM inference (Zheng et al., 23 Jul 2025), BucketServe increased throughput up to $S$4 versus UELLM and $S$5 versus DistServe (offline, LLaMA-2-13B), with average GPU utilization $S$6. At 80% SLO attainment, load capacity (server RPS) improved $S$7 (mixed workload) and $S$8 (Alpaca) over UELLM.
• Bucket-based batching adds negligible pipeline overhead, accounting for less than $S$9 of end-to-end latency.
• For training with variable-length speech data, bucket batching with $b \in B$0 and dynamic batch sizing reduced epoch runtime by approximately $b \in B$1 versus random batching at large batch budgets, with ZPR held near the sorted baseline (Gonzalez et al., 2023).
• Queueing-theoretic analysis for LLM inference shows that increasing the number of bins improves throughput steadily: at $b \in B$2, 20–30% gains are typical; at $b \in B$3, gains reach up to 40–60%; with $b \in B$4, throughput is up to 70% higher than for $b \in B$5. Even with misclassification of bin assignments at error rates $b \in B$6–$b \in B$7, bucket-based batching outperforms $b \in B$8 (i.e., no binning) (Guldogan et al., 2024).

System/Setting	Padding Overhead (ZPR)	Throughput Boost	GPU Memory	End-to-End Latency Overhead
Speech (Sorted) $b \in B$9s (Gonzalez et al., 2023)	0.4%	--	20.1 GB	--
Speech (Bucket, $L_b \le S < U_b$0)	5.2%	1.18x	20.1 GB	--
Speech (Random)	24.1%	1.0x	20.2 GB	--
LLM (BucketServe) (Zheng et al., 23 Jul 2025)	Minimized	$L_b \le S < U_b$1	--	<1% pipeline overhead

6. Theoretical Foundations and Generalizations

Bucket-based batching principles extend to parallel load balancing and dynamic batching in online settings. The batched allocation model (Los et al., 2022) considers allocating $L_b \le S < U_b$2 jobs into $L_b \le S < U_b$3 servers in batches of size $L_b \le S < U_b$4. When jobs can have variable weights, allocation processes (e.g., Two-Choice, $L_b \le S < U_b$5, quantile, graphical) are analyzed. High-probability bounds for the load gap $L_b \le S < U_b$6 are established:

$L_b \le S < U_b$7

for $L_b \le S < U_b$8, with tightness proofs. Theoretical and experimental results suggest that less "aggressive" allocation strategies can outperform more powerful schemes in high-parallelism regimes—a non-monotonic effect.

In queueing-theoretic analysis of multi-bin batching for LLMs (Guldogan et al., 2024), to achieve throughput $L_b \le S < U_b$9, the number of bins required scales as $\{\tau_0, \dots, \tau_M\}$0. Analytically, optimal bin boundaries (under uniform input lengths) are linear interpolations between $\{\tau_0, \dots, \tau_M\}$1 and $\{\tau_0, \dots, \tau_M\}$2.

7. Application Domains and Practical Guidelines

Bucket-based batching is broadly applicable in LLM inference serving, neural sequence modeling, online service scheduling, and any scenario with variable-size or unpredictable workloads. Practical recommendations include:

• Select bucket boundaries either via profiling (e.g., cumulative distribution function quantiles) or uniform partitioning.
• When training, use 10 buckets for length bins and batch according to constant total duration, balancing randomization (for SGD) and resource locality (Gonzalez et al., 2023).
• For online inference, use dynamic splitting and merging, and priority-aware scheduling to optimize both throughput and SLO attainment (Zheng et al., 23 Jul 2025).
• In scenarios involving economies of scale and waiting costs, online batching can be driven by threshold policies such as Wait-Till-α, ensuring constant-factor competitiveness (Bhimaraju et al., 2023).

By integrating bucket-based batching, practitioners achieve near-optimal resource efficiency and throughput, adaptive scaling under nonstationary demand, and robust latency compliance, all within a rigorous theoretical framework.