Bucket-Based Adaptive Batching

• Bucket-Based Adaptive Batching is a technique that partitions variable-length inputs into size-specific buckets to minimize padding and optimize compute resource allocation.
• It dynamically adjusts batch sizes by setting bucket boundaries based on input distribution, ensuring efficient memory use and improved system throughput.
• Empirical results demonstrate significant benefits, including reduced padding overhead and up to 3.58× improved throughput in LLM inference scenarios.

Bucket-Based Adaptive Batching refers to a family of data and request grouping strategies that partition inputs according to size or computational footprint, assigning each item to a "bucket" so batches formed from each bucket require minimal padding and enable dynamic allocation of compute or memory resources. This approach is designed for settings—such as speech enhancement model training and LLM inference serving—where inputs are highly variable in length or complexity, and where static, uniform batching policies lead to inefficiencies and possible memory-overload errors. By adaptively selecting batch sizes and bucket boundaries in real time, bucket-based adaptive batching achieves significant reductions in padding overhead, stabilizes resource use, and often improves overall model performance or system throughput (Gonzalez et al., 2023, Zheng et al., 23 Jul 2025).

1. Fundamental Concepts and Definitions

In bucket-based adaptive batching, the core idea is to subdivide the dataset (or request stream) into disjoint groups based on input size. Each bucket $k$ is associated with a range $[T_{k-1}, T_k]$ of input sizes; an input $i$ of size $T_i$ is assigned to bucket $k$ if $T_{k-1} < T_i \le T_k$ (Gonzalez et al., 2023). For training neural networks, the batch size $B_k$ for each bucket is selected so the total padded size in the batch does not exceed a target compute budget $M$ (such as seconds, frames, or tokens). In LLM inference serving, bucket boundaries $[L_b, U_b)$ segment requests by sequence length, minimizing the expected padding waste per batch and facilitating dynamic adjustment according to instantaneous queue statistics and memory availability (Zheng et al., 23 Jul 2025).

Throughout, constraints such as $B_{\mathrm{min}} \leq B_k \leq B_{\mathrm{max}}$ are enforced to avoid pathological batch sizes.

2. Algorithmic Workflow and Implementation Details

The bucket-based adaptive batching pipeline typically comprises:

• Bucket Threshold Selection: Thresholds for bucket sizes are determined via uniform partitioning, quantile binning, or empirical optimization to minimize padding overhead. In BucketServe for LLMs, Equation (4) provides a closed-form condition for setting bucket boundaries to minimize expected padding waste:

$$U_b^* = \frac{\int_{L_b}^{U_b} S f(S) dS}{\int_{L_b}^{U_b} f(S) dS}$$

where $f(S)$ is the empirical input length distribution (Zheng et al., 23 Jul 2025).

• Assignment and Grouping: Each incoming item is assigned to a bucket, the contents of which are randomly shuffled or scheduled according to workload policies.
• Dynamic Batch Sizing: For each bucket, batch size $B_k$ is computed as $B_k = \left\lfloor {M}/{L_k} \right\rfloor$ where $L_k$ is the representative length in bucket $k$. Additional constraints clamp $B_k$ within predefined bounds (Gonzalez et al., 2023).
• Adaptive Splitting and Merging: BucketServe adaptively splits buckets when the intra-bucket input length distribution becomes skewed (using a threshold $\theta$) or merges buckets when the overall load falls below safety limits, simplifying scheduling when demand is low (Zheng et al., 23 Jul 2025).
• Padding and Collation: Within each batch, items are padded to the maximum length in that batch; a mask is applied in the loss or serving pipeline to ignore padded regions (Gonzalez et al., 2023).
• Priority-Aware Scheduling: Buckets can be processed according to Shortest-Job-First (SJF), Longest-Job-First (LJF), or first-come-first-served (FCFS), depending on whether maximizing throughput or minimizing latency is desired (Zheng et al., 23 Jul 2025).
• GPU Memory Safety: Real-time memory queries determine maximal dispatchable batch size $N_{\max}$ according to

$$2 L H D B \times S_{\max} \times N_{\max} \le M_{\mathrm{safe}}$$

where $L$, $H$, $D$, $B$ are model parameters and $M_{\mathrm{safe}}$ is reserved GPU memory (Zheng et al., 23 Jul 2025).

Table: Bucket-Based Adaptive Batching Workflow

Step	Speech Enhancement (Gonzalez et al., 2023)	LLM Inference (Zheng et al., 23 Jul 2025)
Bucket definition	Uniform/quantile duration bins	Length intervals, optimized by Eq. 4
Batch size assignment	$B_k = \lfloor M / L_k \rfloor$	Memory bound, Eq. 6
Split/merge adaptivity	Fixed buckets	Dynamic splitting/merging
Scheduling per bucket	Random within bucket	SJF/LJF or FCFS, priority-aware

3. Empirical Performance and Statistical Analysis

Empirical benchmarks highlight several advantages of bucket-based adaptive batching:

• Speech Enhancement (Conv-TasNet, (Gonzalez et al., 2023)): For bucket batching with dynamic batch size:
  • Zero-padding rate (ZPR) reduced to $<10\%$ (e.g., 5.2%) compared to $>24\%$ for random batching.
  • Training time reduced by $18-26\%$ relative to random batching.
  • GPU memory stabilized across batch sizes.
  • Enhancement performance measured by $\Delta$PESQ and $\Delta$ESTOI improved as batch size decreased, with best metrics reported for $M=4$ s: $\Delta$PESQ (match) = 0.477, (mismatch) = 1.05.
• LLM Inference Serving (BucketServe, (Zheng et al., 23 Jul 2025)): On LLaMA-2-13B, BucketServe:
  • Improved throughput up to $3.58\times$ over UELLM and $1.31\times$ over DistServe.
  • Achieved GPU utilization of $81.66\%$ versus $23\%$ for static batching.
  • Supported $1.93\times$ more request load with $80\%$ SLO attainment compared to DistServe.
  • Bucket management overhead $<1\%$ of end-to-end latency even with $>20$ buckets.

A plausible implication is that bucket-based schemes realize substantial resource savings in both training and inference, particularly under high input-length variance and dynamic demand.

4. Comparative Analysis and Failure Modes of Alternative Strategies

Bucket-based adaptive batching directly addresses several inefficiencies inherent in static or continuous batching:

• Static Batching: Assigns uniform batch sizes regardless of input variability, causing excessive padding (and attendant compute/memory waste) and risking out-of-memory errors during workload spikes (Zheng et al., 23 Jul 2025).
• Continuous (Elastic) Batching: Allows for variable batch size but does not separate request lengths, so padding waste remains high under heterogeneous arrivals (Zheng et al., 23 Jul 2025).
• Fixed Batching in RL (ABP theory, (Merlis, 15 Jan 2026)): Fixed batch sizes in multi-step lookahead can be exponentially suboptimal, as optimal planning requires adapting batch size to the state and remaining horizon.

Bucket-based adaptation, by contrast, dynamically aligns batch composition to actual input statistics, maintaining minimal padding and optimizing resource utilization.

5. Theoretical Foundations: Adaptive Batching in Reinforcement Learning

In tabular RL with multi-step lookahead, adaptive batching policies (ABPs) offer a dynamic framework where batch size is state-dependent. Formally, for each episode step $h$ and state $s$, the batching map $B_h^\pi(s)$ selects a batch size, and the within-batch policy $\phi^h$ determines the action sequence given current lookahead. The associated Bellman equations for optimal ABPs (Merlis, 15 Jan 2026) are:

$$V_h^*(s) = \max_{B = 1 \ldots \ell_h} \mathbb{E}_{I \sim \mathcal{I}_{h,B}(s)} \left[ Q_h^*(s, B; I, V_{h+B}^*) \right]$$

where $Q_h^*$ is the maximal expected cumulative reward-plus-followup value over batch-size $B$.

Learning ABPs in unknown environments utilizes a variance-based optimistic algorithm (AL-UCB), achieving regret

$$O\left( \sqrt{H^3 S K \ell \ln(S H \ell K/\delta)} + \sqrt{H^3 S^2 \ell} \ln(S H \ell K/\delta) \right)$$

with polynomial dependence on state count ($S$), horizon ($H$), episodes ($K$), and lookahead ($\ell$); independent of action size ($|A|$).

6. Practical Guidelines and Deployment Recommendations

Research demonstrates several guidelines for implementing bucket-based adaptive batching (Gonzalez et al., 2023, Zheng et al., 23 Jul 2025):

• Select $K$ between 8–16 buckets for balanced randomization and padding minimization.
• Set target batch resource budget $M$ for dynamic batch sizing; $M=4–8$ s for speech yields best generalization.
• Constrain per-bucket batch size within reasonable bounds ($B_{\mathrm{min}}=1$, $B_{\mathrm{max}}=8$–$16$).
• Use dynamic bucket resizing (splitting/merging) in serving systems to mitigate resource fragmentation and adapt to demand fluctuations in real time.
• Apply SJF for latency-sensitive workloads or LJF for maximizing throughput in LLM inference.
• Use masking to ignore padded elements in loss calculations and output decoders.

Empirical findings suggest that these recommendations maximize resource efficiency and maintain high model accuracy or system responsiveness.

7. Broader Impact and Applicability

Bucket-based adaptive batching is broadly applicable in domains with high input-length variance or fluctuating demand, such as speech/audio processing, LLM inference serving, and reinforcement learning with lookahead. The method enhances memory and compute utilization, minimizes padding and associated inefficiencies, and provides scalability and adaptability under dynamic workloads. Its principles are directly extensible to scenarios where input grouping strategies materially affect convergence rate, model generalization, or system SLO compliance. The approach encapsulates a generic paradigm for real-time, data-driven resource management in machine learning and sequential decision-making systems (Gonzalez et al., 2023, Zheng et al., 23 Jul 2025, Merlis, 15 Jan 2026).