Tail Batching Techniques

• Tail batching is a collection of tail-aware batching and scheduling techniques that target long-tail latency, underutilization, and rare system states.
• It encompasses methods like dynamic batch-size control in DNN inference, selective delay in synchronous RL, and adaptive token batching in large language model serving.
• Tail batching optimizes system performance by balancing throughput gains against risks such as deadline violations and inefficient resource use across varied domains.

Tail batching denotes a set of batching and scheduling techniques that target the tail of system behavior rather than only mean throughput. In the cited literature, the “tail” may refer to the tail of an inference-latency distribution, the underfilled late iterations of batched LLM execution, the long-response stragglers that dominate synchronous reinforcement-learning rollouts, or the high-congestion overflow states of a batch-service queue. Taken together, these works suggest that tail batching is best understood as a family of tail-aware control policies for suppressing stragglers, reducing underutilization, or approximating rare but consequential tail states while retaining the efficiency gains of batching (Choi et al., 2020, Zheng et al., 2024, Gao et al., 25 Sep 2025, Xu et al., 4 Jan 2025).

1. Terminological scope

The term is not used uniformly across the literature. Some papers use it explicitly, while others present mechanisms that are tail-batching-like without adopting the phrase. The common pattern is selective batch formation or selective delay aimed at the regime where naïve batching becomes pathological: long-tail latency, decode-dominated valleys, long rollout rounds, or truncated queue tails.

Context	What constitutes the tail	Representative mechanism
Real-time DNN inference	95th-percentile inference latency	Dynamic batch-size tuning under an SLO
Large-batch LLM inference	Underfilled late iterations and kernel tail effect	Decode-prefill mixing and memory-centric token batching
Synchronous RL post-training	Long-response rollout stragglers	Separation into short rounds and long rounds
Batch-service queues	Overflow “tail” states	Abstract tail-state cost in finite-state approximation
SLA-aware DNN serving	Late arrivals trapped behind an ongoing batch	Layer-boundary preemption and catch-up batching

A central distinction follows from this heterogeneity. Tail batching is not always “batching the tail of a request queue.” In some systems it is instead a latency-aware online batch-size controller, in others a throughput-oriented mitigation of late-stage GPU underfill, and in still others a mathematical device for handling truncated state tails or end-of-horizon postponement. This suggests that the term is best treated as domain-relative rather than as a single standardized algorithmic primitive.

2. Tail-latency-aware batching in inference services

In real-time GPU inference, tail batching is most directly associated with latency-constrained batch-size control. DNNScaler defines tail latency as the $95^{th}$ percentile of the inference latency distribution and formulates the objective as maximizing average throughput over time subject to $Latency^{t} \leq SLO$. Its batching mechanism is dynamic rather than static: it starts from $BS = 1$, monitors tail latency, and uses a pseudo binary search to find the largest batch size that still satisfies the latency target. The control rule uses $\alpha = 0.85$: if tail latency is less than $SLO \times \alpha$, batch size is increased; if tail latency is greater than $SLO$, it is decreased; otherwise it is kept unchanged. The system first decides whether batching or multi-tenancy is more beneficial for a given DNN, then tunes the corresponding control knob. Compared with Clipper’s AIMD policy, DNNScaler’s search is faster and more adaptive, and the reported throughput improvement reaches up to $14\times$, with $218\%$ on average, while meeting latency requirements (Nabavinejad et al., 2023).

LazyBatching addresses a different source of tail behavior: head-of-line blocking caused by coarse graph-level batching. Instead of waiting for a full graph-level batch, it batches at the granularity of individual graph nodes or layers. Execution can be preempted at layer boundaries, a newly arrived request can “catch up,” and sub-batches are merged once they reach the same next graph node. The scheduler is explicitly SLA-aware. Its slack formulation is

$$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$

and for a batch of $N$ requests it uses the conservative estimate

$Latency^{t} \leq SLO$0

The reported effect is a reduction of response-time spikes caused by late arrivals waiting behind an ongoing batch. Relative to graph batching, LazyBatching reports average $Latency^{t} \leq SLO$1 latency improvement, $Latency^{t} \leq SLO$2 throughput improvement, and $Latency^{t} \leq SLO$3 improvement in SLA satisfaction; for Transformer, a 99th-percentile latency example is $Latency^{t} \leq SLO$4 ms versus $Latency^{t} \leq SLO$5 ms for the best graph-batching baseline (Choi et al., 2020).

Fluid Batching extends tail-aware batching to early-exit neural networks on edge NPUs. Its starting observation is that early exits dynamically shrink the active batch size deeper in the network, so static model-level batching can leave later layers underutilized and create long-latency outliers. The scheduler is exit-aware and deadline-aware. At an early exit, it computes a residual batch $Latency^{t} \leq SLO$6, the available slack $Latency^{t} \leq SLO$7, and a refill amount $Latency^{t} \leq SLO$8. Preemption is admitted only if

$Latency^{t} \leq SLO$9

with

$BS = 1$0

On the hardware side, Fluid Batching changes how samples are packed into GEMM matrices on a per-layer basis, and Stackable Processing Elements reconfigure compute structure for small batches. The reported aggregate result is $BS = 1$1 improvement in average latency and $BS = 1$2 improvement in tail latency SLO satisfaction (Kouris et al., 2022).

3. Tail-effect mitigation in batched LLM serving

In large batched or offline LLM inference, the most prominent tail-batching interpretation is not percentile latency control but suppression of low-density late-stage execution. BatchLLM targets workloads in which throughput matters more than latency and where requests exhibit prefix sharing. It explicitly identifies common prefixes globally, schedules requests sharing the same prefix together, and reorders groups so that those with larger decode-to-prefill ratio are scheduled first. The scheduling logic uses three queues—common prefix queue, distinct prompt queue, and decoding queue—and forms each iteration’s token batch by fetching in the order: decoding queue, distinct prompt queue, then common prefix queue, subject to the constraint that a common prefix must be processed before its distinct prompts. This allows decoding tokens to be mixed with later prefill chunks, which keeps token batches dense in iterations that would otherwise become decode-dominated valleys (Zheng et al., 2024).

BatchLLM’s batching constraint is memory-centric rather than request-centric. It uses a token chunk size $BS = 1$3 and a KV memory threshold $BS = 1$4, and it continues adding prefill chunks while the current batch remains under the memory threshold. The system also optimizes the prefix-shared Attention kernel with horizontal fusion to reduce tail effect and kernel launch overhead. The reported token-batching ablation shows throughput improvement of up to $BS = 1$5, TECD reduction of up to $BS = 1$6, and compression of the number of steps from about $BS = 1$7 to $BS = 1$8. Across microbenchmarks, BatchLLM outperforms vLLM by $BS = 1$9 to $\alpha = 0.85$0 on A100 and up to $\alpha = 0.85$1 on MI200; in two industry case studies, the speedups are about $\alpha = 0.85$2 and $\alpha = 0.85$3. On a representative workload, the prefix-caching saving ratio is $\alpha = 0.85$4 for vLLM and $\alpha = 0.85$5 for BatchLLM (Zheng et al., 2024).

A distinct but related line appears in FairBatching, where the target tail is TTFT rather than decode-stage underfill. FairBatching criticizes rigid decode-prioritizing schedulers and replaces them with slack-aware, time-budgeted batch formation. It defines per-token deadlines as

$\alpha = 0.85$6

then uses

$\alpha = 0.85$7

as the key admission signal. Batch capacity is determined by an adaptive time budget rather than a static token budget, and requests are prioritized in the order urgent decode, prefill, then non-urgent decode. This shifts batching from a decode-prioritizing paradigm to a fairness-aware one. The reported results show TTFT tail-latency reduction of up to $\alpha = 0.85$8 while maintaining TPOT SLOs, alongside $\alpha = 0.85$9 improvement in single-node capacity and $SLO \times \alpha$0 improvement in cluster-level capacity (Lyu et al., 16 Oct 2025).

4. Tail batching in synchronous RL post-training

RollPacker introduces tail batching as an explicit rollout scheduling strategy for synchronous RL post-training of LLMs. The target pathology is the long-tail response-length distribution in rollout steps: rollout accounts for about $SLO \times \alpha$1–$SLO \times \alpha$2 of total step time, reward for $SLO \times \alpha$3–$SLO \times \alpha$4, and training for $SLO \times \alpha$5–$SLO \times \alpha$6, while some responses reach up to $SLO \times \alpha$7 tokens and can be $SLO \times \alpha$8–$SLO \times \alpha$9 longer than median-scale responses. Under synchronous execution, the entire batch waits for the slowest response, so the shorter rollouts create severe GPU bubbles (Gao et al., 25 Sep 2025).

The core mechanism is the separation of rollout steps into short rounds and long rounds. In a short round, RollPacker uses speculative execution with speculation factor $SLO$0; in the evaluation, $SLO$1. It launches $SLO$2 prompts and $SLO$3 responses per prompt, but retains only the first $SLO$4 prompts that finish and, for each prompt, only the first $SLO$5 responses that complete. The excluded prompts are appended to a long-prompt queue. Once that queue reaches size $SLO$6, a dedicated long round is executed with speculation disabled and full-length responses allowed to complete. The design addresses two constraints identified in the paper: the rollout batch must still contain enough prompts and responses, and excluding long prompts must not distort the training distribution (Gao et al., 25 Sep 2025).

The reported effect is to concentrate the stragglers into a small number of long rounds while keeping most rounds balanced. With $SLO$7, the maximum response length in short rounds is reduced by up to $SLO$8, and average rollout time improves by up to $SLO$9 over the no-tail-batching setting. Tail batching alone reduces rollout cost strongly, with up to $14\times$0 speedup in average rollout time in some settings. End to end, RollPacker reports training-time reduction of $14\times$1–$14\times$2 compared to veRL and up to $14\times$3 compared to RLHFuse for the Qwen2.5 family on up to $14\times$4 H800 GPUs, without sacrificing accuracy (Gao et al., 25 Sep 2025).

5. Queueing and optimization viewpoints

In queueing-theoretic work, tail batching can mean controlling or approximating the tail of the system state rather than the tail of execution latency. The SMDP-based dynamic batching framework models a batch service queue with Poisson arrivals of rate $14\times$5, batch-size-dependent service time $14\times$6, mean $14\times$7, and energy consumption $14\times$8. The objective is to minimize

$14\times$9

equivalently a long-run average cost combining response time and power consumption. The infinite state space is truncated at $218\%$0, and all states above that threshold are aggregated into an overflow state $218\%$1 representing the tail states. The key device is an abstract overflow cost

$218\%$2

which makes the finite approximation “feel” the holding costs of the tail. With $218\%$3, $218\%$4, and $218\%$5, the minimum $218\%$6 decreases from $218\%$7 when $218\%$8 to $218\%$9 when $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$0, yielding $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$1 space-complexity reduction and $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$2 time-complexity reduction. The paper also notes that optimal policies can exhibit a control-limit form,

$$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$3

This is a threshold view of batching in which tail-state handling changes both numerical tractability and the aggressiveness of service decisions (Xu et al., 4 Jan 2025).

A related but distinct usage appears in online linear programming with batching. There the planning horizon $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$4 is divided into $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$5 equal-length batches, $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$6, and decisions on customers arriving within a batch may be delayed to the end of that batch. The notable structural result is that the proposed algorithms do not delay decisions in every batch. In the single-resource known-$$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$7 case, the algorithm delays only in the first and last batch and achieves $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$8 regret, while the paper also proves a matching $$Slack=SLA_{target}-(T_{wait}+SingleInputExecTime_{Req1}),$$9 lower bound; in the multiple-resource setting, delay is only in the last batch after using historical data, and the regret bound remains $N$0. This is not a named tail-batching primitive in the paper, but it is a mathematically precise form of postponement concentrated at the horizon boundaries rather than throughout the whole process (Xu et al., 2024).

6. Distinctions, misconceptions, and recurring design tradeoffs

A frequent misconception is that tail batching always refers to queue-tail aggregation. The literature is more specific. DNNScaler is best characterized as dynamic, latency-aware serving-time batching, not queue-tail batching per se. BatchLLM uses the language of tail effect to denote underfilled batch iterations and small-kernel inefficiency near the end of batched execution. RollPacker uses the term literally for long-tail rollout responses. The SMDP formulation uses “tail” for overflow states in a truncated Markov model. These are related only at the level of control intent: each method reallocates batching effort toward the regime that dominates worst-case behavior (Nabavinejad et al., 2023, Zheng et al., 2024, Gao et al., 25 Sep 2025, Xu et al., 4 Jan 2025).

Another recurrent result is that there is no universal batch size. DNNScaler reports that batching works well for large, compute-heavy DNNs such as Inception-V4, ResNetV2-152, NAS-Large, and PNAS-Large, while the effect is negligible for smaller networks such as Inception-V1 and MobileNet variants. Its selected batch sizes vary widely, including $N$1 for Inception-V4, $N$2 for NAS-Large, $N$3 for TextClassif, and $N$4 for ResV2-50. BatchLLM reaches a similar conclusion at token granularity: request-count limits can be suboptimal, and memory availability, decode-to-prefill ratio, and global prefix structure determine whether a large token batch is beneficial (Nabavinejad et al., 2023, Zheng et al., 2024).

Across domains, the same tradeoff recurs. Larger or denser batches raise throughput, GPU or NPU utilization, or energy efficiency; the tail risk is longer waiting, deadline violation, or late-stage underutilization. Tail batching changes where the system spends its batching budget. In inference systems, this often means enlarging batches only until an SLO boundary is approached. In LLM serving, it means injecting prefill work into decode-heavy iterations and choosing admission by memory rather than by request count. In synchronous RL, it means concentrating long responses into designated long rounds. In queueing models, it means penalizing overflow states strongly enough that the finite approximation preserves the economics of rare congestion. This suggests that tail batching is less a single algorithm than a design principle: batch selectively where the tail is structurally dominant, and avoid paying the full cost of worst-case behavior in every iteration or every request.