Length-Aware Prefill Scheduling in LLM Systems

• The paper introduces a framework where prompt and residual work signals are used to optimize time-to-first-token and overall system efficiency in LLM serving.
• It details advanced runtime mechanisms—such as operator-level preemption and dual prefill queues—to prevent head-of-line blocking and manage heterogeneous prompt lengths.
• Empirical results show significant improvements, including up to 67% lower end-to-end latency, by coupling cost models with dynamic runtime scheduling for mixed workloads.

Length-aware prefill scheduling is the family of LLM-serving policies that treat prompt length, uncached prefix length, remaining prefill work, or prompt-derived latency estimates as explicit control signals for admission, batching, routing, preemption, and capacity allocation during the prefill phase. The topic arises because prefill processes the full prompt, constructs KV cache state, and typically dominates time-to-first-token (TTFT), while prompt-length heterogeneity creates head-of-line blocking, queueing variance, and SLO violations, especially in prefill-decode disaggregated systems where prefill contention is concentrated on a dedicated GPU pool (Hsieh et al., 18 Feb 2026).

1. Operational setting and the source of length heterogeneity

Modern autoregressive inference is organized around a two-phase execution model. In prefill, the server processes the full prompt in parallel to construct KV cache and emit the first token; in decode, it generates subsequent tokens autoregressively. Multiple systems emphasize the same asymmetry: prefill is compute-bound, whereas decode is memory-bound. This asymmetry is what makes prompt length particularly consequential. A long prompt can occupy a prefill device for “hundreds of milliseconds,” inflate its own TTFT, and, more importantly, delay later arrivals behind it; in prefill-decode disaggregation, removing prefill/decode interference therefore concentrates the remaining contention onto the prefill side (Hsieh et al., 18 Feb 2026).

The practical motivation is not merely that prompts differ, but that they differ by orders of magnitude in production traces. In the QwenTrace workload used for FlowPrefill, chatbot and image tasks have mean prompt lengths around 590 and 532 tokens, while web search and summarization have mean prompt lengths of 5976 and 6833 tokens, with P99 values reaching 16,635 and 22,390 tokens; the TTFT SLOs are likewise heterogeneous, with text at 0.25s, image at 0.5s, search at 4.0s, and file at 6.0s for Llama3-8B. This makes the problem inherently one of mixed urgency and mixed size, not just raw throughput maximization (Hsieh et al., 18 Feb 2026).

Long-context serving sharpens the same issue in a different way. L4 argues that with context windows exceeding 128K tokens, the dominant inefficiency is often intra-batch length heterogeneity in attention kernels rather than lack of batching per se. In its characterization, heterogeneity can inflate decoding forward-pass latency by $1.1\times$ to $2.1\times$ even at fixed total token count, because long sequences create more work blocks and fixed tiling strategies become suboptimal. Although L4 targets decode more directly than prefill, it reinforces the broader point that sequence length is not a secondary workload feature; it is a primary systems variable that shapes kernel efficiency, queueing, and tail latency (Yuan et al., 22 Dec 2025).

2. Control signals, objectives, and what “length-aware” means

Length-aware prefill scheduling is not synonymous with shortest-prompt-first. Different systems expose different cost proxies, and the operative signal may be raw prompt length, current sequence length, uncached prefix length, remaining prefill tokens, predicted TTFT, or a joint QoE surrogate. L4 explicitly models request quality as

$$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$

where $I$ is input prompt length and $L$ is current sequence length, so prefill contributes quadratically in prompt length while decode iteration contributes linearly in sequence length. HexAGenT does not publish a closed-form prefill-time equation, but it states that its estimator converts “per-call input and output length” into runtime estimates including prefill time, decode time, KV-transfer latency, and decode memory demand, and then scores ready calls by

$R_s(c,t)=\frac{(t-a_w)+\Delta_s(c,t)}{H_w(t)},$

with $\Delta_{\mathrm{P}}(c,t)$ including waiting time, prefill time, and KV-transfer latency. In both cases, prompt length enters through an explicit cost model rather than a fixed size-priority rule (Yuan et al., 22 Dec 2025, Peng et al., 15 May 2026).

Several serving systems make this dependence operational through slack or deadline formulas. FlowPrefill uses slack-aware earliest-deadline-first: $priority=\frac{\sgn(slack)}{deadline}, \qquad slack = deadline - current\_time - \widehat{\text{TTFT}},$ so prompt length matters through $\widehat{\text{TTFT}}$ and through the batching rule

$T_{remain}>L \quad \text{and} \quad n<G,$

where $2.1\times$0 and $2.1\times$1 is the batch token budget. Kairos similarly computes

$2.1\times$2

thereby combining TTFT slack with a shortest-job bias. TaiChi’s proxy-level prefill routing tests whether a request can meet TTFT on a candidate instance via

$2.1\times$3

with $2.1\times$4 the queued prefill time, $2.1\times$5 the estimated execution time, and $2.1\times$6 the KV-transfer time on P-heavy instances. PrefillOnly, in the prefill-only regime, defines an aging-adjusted score

$2.1\times$7

and reports that $2.1\times$8 is a strong proxy for JCT, with Pearson correlation 0.987 on Qwen-32B FP8 on 1×A100 (Hsieh et al., 18 Feb 2026, Wang et al., 4 May 2026, Wang et al., 4 Aug 2025, Du et al., 12 May 2025).

These formulations show that “length” often means effective or residual work rather than raw prompt size. PrefillOnly uses cache-miss length, PrfaaS uses incremental prefill length $2.1\times$9 “excluding any cached prefix,” and Gimbal uses remaining prefill tokens rather than admission-time prompt length because continuous batching and chunked prefill make original prompt size an unreliable estimate of unfinished work. A strict reading of the literature therefore distinguishes at least four notions: total prompt length, uncached prompt length, current sequence length, and remaining prefill work. Length-aware schedulers differ mainly in which of these they operationalize and what system objective they optimize—goodput, QoE, workflow horizon attainment, or TTFT/TPOT SLO attainment (Qin et al., 16 Apr 2026, Sun et al., 13 Jun 2026).

3. Runtime mechanisms that make length-aware control executable

A scheduler can only exploit length heterogeneity if the runtime exposes interruption, admission, or queueing granularity fine enough to react before long prefills monopolize the device. FlowPrefill’s central contribution is exactly this point. It argues that prior chunked-prefill systems conflate scheduling frequency with preemption granularity: small chunks improve responsiveness but reduce efficiency; large chunks preserve throughput but worsen blocking. FlowPrefill separates the two by combining event-driven scheduling with operator-level preemption. Scheduling is triggered only on request arrival or completion, while runtime interruption is allowed at boundaries between operators such as qkv_proj, attn, o_proj, gate_up_proj, and down_proj. The result is that preemption blocking time is bounded by one operator rather than one chunk or one full layer; in the reported experiment, all observed blocking latencies remain below $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$0 ms, and operator-level preemption reduces average blocking time by $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$1–$$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$2 relative to layer-level preemption (Hsieh et al., 18 Feb 2026).

Kairos and Gimbal pursue a different route. Kairos keeps chunked prefill but changes the queue discipline: it estimates FCFS completion times from remaining prefill tokens and current throughput, computes normalized urgency, and greedily fills the next chunk budget with the highest-priority requests. Gimbal splits the problem into inter-engine and intra-engine control. Across DP engines, it dispatches on

$$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$3

where pre_i^{rem} is remaining prefill work from running requests and wait_i is waiting prefill work in the local queue. Inside each engine, it applies SJF-style prompt-length ordering with aging, promoting any request whose waiting time exceeds $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$4 seconds. In both systems, the relevant innovation is not a new asymptotic size-based theorem but a runtime path from prompt-derived work estimates to concrete queue selection (Wang et al., 4 May 2026, Sun et al., 13 Jun 2026).

TaiChi makes the same point in a hybrid PD aggregation/disaggregation architecture. It deliberately creates two service classes—P-heavy instances with larger chunk sizes and D-heavy instances with smaller chunk sizes—and then uses length-aware prefill scheduling to “selectively degrade TTFT by assigning short prefill requests to slower instances when doing so does not violate SLO constraints.” The purpose is latency shifting: use surplus TTFT slack from short, safe requests to free fast-prefill capacity for long or risky ones. This is not pure shortest-job-first either; it is a feasibility-filtered reassignment mechanism that depends on queue state, transfer cost, and SLO headroom (Wang et al., 4 Aug 2025).

Mechanism	Length signal	Scheduling action
FlowPrefill	Predicted TTFT, token counts	Event-driven batching and operator-level preemption
Kairos	TTFT slack, input length	Urgency-based chunked prefill ordering
Gimbal	Remaining prefill work, prompt length	Engine dispatch and SJF-with-aging local queues
TaiChi	Estimated TTFT under instance type	TTFT-aware routing across P-heavy and D-heavy instances

4. Architectural patterns: specialization, routing, and multi-stage decomposition

A large fraction of recent work shifts from single-queue heuristics to architectural specialization. PLA-Serve explicitly disaggregates prefill workloads by prompt length. It classifies requests by a compute-memory boundary $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$5, maintains separate short and long prefill queues, and supports both temporal disaggregation on a single prefill instance and spatial disaggregation across multiple instances. The short-prefill path uses an Adaptive-Wait-Depth batching policy, a bounded waiting window, and CUDA Graph buckets over lengths $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$6 and batch sizes $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$7. In real multi-turn workloads, PLA-Serve reports over 30% lower prefill latency than vanilla SGLang under PD disaggregation, 28% fewer SLO violations in multi-instance deployments than vanilla data parallel configuration, 12% fewer SLO violations than the SGLang router with load balancing in multi-GPU settings, and 35% higher throughput under high concurrency and mixed-request scenarios for Qwen2.5-32B prefill serving (She et al., 4 Jan 2026).

L4 operates at cluster scope rather than within a single prefill pool. Its organizing principle is “global homogenization for local efficiency”: partition identical model-serving instances into length-specialized groups, route a request to the earliest stage whose range contains its initial sequence length, and migrate it as current length grows beyond the stage boundary. Offline dynamic programming chooses stage boundaries and instance allocation, while periodic online refinement updates boundaries using local telemetry. Although L4 is more decode-oriented than prefill-specific, it directly treats prompt length $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$8 as a first-class routing feature and treats prefill delay $$Q = \left(C_0 + C_1 I + C_2 I^2\right) + \left(C_3 + C_4 L\right),$$9 as part of the QoE objective. Under the same configuration, L4 reduces end-to-end latency by up to 67% and tail latency by up to 69%, while improving overall system throughput by up to 2.89 times (Yuan et al., 22 Dec 2025).

Arrow and PrfaaS generalize the same idea to capacity elasticity and inter-cluster offloading. Arrow does not prioritize requests by prompt length in a local queueing sense; rather, it profiles a quadratic TTFT predictor from input length and uses it to decide prefill dispatch and adaptive instance-pool reconfiguration between prefill and decode. This makes it a cluster-level length-aware prefill capacity controller rather than a shortest-job scheduler. PrfaaS, by contrast, uses a direct threshold rule on incremental uncached prefill length: $I$0 Its case study finds an optimal threshold of $I$1K tokens, routes 49.6% of requests to the remote prefill cluster, keeps cross-cluster egress at approximately 13 Gbps, and improves throughput by 54% over homogeneous PD and 32% over a naive heterogeneous baseline (Wu et al., 17 May 2025, Qin et al., 16 Apr 2026).

Higher-level schedulers extend the same logic beyond isolated requests. ConServe raises the scheduling unit from turn to conversation, observes that agentic workloads often decompose into a compute-bound first-turn prefill followed by a long memory-bound tail, and therefore routes the first-turn prefill using directly observable first-turn input length and binds the conversation thereafter to the decoder with the lowest active KV-cache occupancy. HexAGenT treats each agentic workflow as an online-revealed DAG and evaluates candidate prefill/decode pairs using projected prefill time, decode time, KV-transfer latency, and decode memory demand derived from per-call input and output length. Both systems remain length-aware, but the decision target is conversation-level or workflow-level completion rather than single-request TTFT alone (Ding et al., 1 Jun 2026, Peng et al., 15 May 2026).

Pattern	Representative system	Main length signal
Dual prefill lanes	PLA-Serve	Prompt length relative to $I$2
Length-specialized replica groups	L4	Input length $I$3, current sequence length $I$4
Adaptive PD-ratio control	Arrow	Predicted prefill time from input length
Cross-cluster selective offload	PrfaaS	Incremental uncached prefill length $I$5
Conversation-level first-turn routing	ConServe	First-turn input length
Workflow-aware pair selection	HexAGenT	Per-call input and output length

5. Algorithmic variants and special regimes

Some systems make length-aware prefill scheduling more accurate by changing what “length” means computationally. PrefillOnly targets prefill-only workloads, where output length is fixed to one token and job size is therefore observable before dispatch. Its key scheduling result is that the relevant size is not raw prompt length but effective uncached prompt length, estimated from $I$6 and continuously recalibrated as prefix-cache state changes. This is a particularly strong instance of prefill-aware scheduling because it removes decode uncertainty altogether and makes SRJF-style queueing viable in production-like settings (Du et al., 12 May 2025).

UniPrefill changes the cost model rather than the queue discipline. It introduces block-wise dynamic sparsification, where token importance is estimated at selected full-attention layers, low-importance blocks are dropped, and the reduced token set propagates through downstream attention and FFN computation. The scheduler implication is explicit in the paper’s discussion: once such an operator exists, retained block count is a better proxy for prefill cost than raw prompt length. UniPrefill reports up to 2.1x TTFT speedup, with gains increasing with both prompt length and concurrency, but it also reports that gains can be small or negative for short prompts on some hybrid models, which suggests a thresholded or predicted-benefit admission rule rather than uniform activation (Fan et al., 7 May 2026).

MOCAP addresses a different special case: prefill-only long-context inference on wafer-scale chips. There the relevant scheduling unit is not the request but the chunk. The paper shows that equal-sized chunks are not equal-cost chunks, because for chunk $I$7,

$I$8

Latency-Balanced Chunk Partitioning therefore makes later chunks smaller, while Memory-Balanced KV Reallocation compensates for the fact that earlier pipeline stages accumulate KV cache for longer. Compared with GPipe, MOCAP reports 76.4% lower end-to-end latency and $I$9 higher throughput on average, and it extends maximum supported sequence length by up to $L$0 compared with Terapipe (Wang et al., 22 Jun 2026).

The handover setting studied in joint KV-cache transfer and token prefill makes the control variable even more explicit: prefill length itself becomes a scheduling decision. For $L$1 UEs with context lengths $L$2, the target BS chooses a shared batch prefill length $L$3, so that

$L$4

The optimal worst-user transfer delay for fixed $L$5 is

$L$6

and the optimal rule is to choose $L$7 so that $L$8 when possible. Although this is a mobility setting rather than a serving queue, it is one of the clearest optimization-based treatments of prefill length as a first-class scheduling variable (Lee et al., 30 Mar 2026).

6. Empirical record, misconceptions, and open problems

Across the current literature, the empirical case for length-aware prefill scheduling is strong but heterogeneous in what it measures. FlowPrefill reports up to $L$9 higher maximum goodput than prior PD-disaggregated systems under heterogeneous TTFT SLOs. L4 reports up to 67% lower end-to-end latency, up to 69% lower tail latency, and up to 2.89 times higher throughput. Kairos improves TTFT SLO attainment by up to 23.9%, TPOT SLO attainment by up to 27.1%, end-to-end SLO attainment by up to 33.8%, and decode throughput by up to 19.3%. TaiChi improves goodput by up to 77% over state-of-the-art systems under balanced TTFT and TPOT SLOs. Gimbal reduces average TTFT by 42.9% and average TPOT by 33.3% compared with vLLM, while improving high-load request throughput by 3.0%. HexAGenT reduces the SLO scale required for timely workflow completion by an average of 20.1% at 95% attainment and 33.0% at 99% attainment, with maximum reductions of 45.0% and 80.5%, respectively (Hsieh et al., 18 Feb 2026, Yuan et al., 22 Dec 2025, Wang et al., 4 May 2026, Wang et al., 4 Aug 2025, Sun et al., 13 Jun 2026, Peng et al., 15 May 2026).

A recurring misconception is that length-aware prefill scheduling is simply shortest-job-first applied to prompt tokens. The literature is more precise. FlowPrefill is deadline- and slack-aware, not explicit SJF. Kairos combines TTFT slack with a shortest-job bias. TaiChi uses size mainly to decide which requests can safely absorb slower service. HexAGenT may intentionally prioritize a long prefill if it lies on a workflow-critical path. Even in the most size-centric cases, such as PrefillOnly, the operative size is usually uncached or residual work rather than raw prompt length. A second misconception is that a scheduler alone is enough. The strongest results consistently combine a policy with a runtime mechanism: operator-level preemption, dual prefill queues, asynchronous migration, live KV transfer, or architecture-specific compaction (Hsieh et al., 18 Feb 2026, She et al., 4 Jan 2026, Du et al., 12 May 2025, Fan et al., 7 May 2026).

The main open problems are therefore not whether length matters, but how to couple length with accurate cost models and low-overhead runtime control. ARES shows that future-aware length prediction can be lightweight enough for online use—its LLM-native predictor reports MAE 3873.21, 8.4M parameters, and 1.33 ms latency at batch 1—but its own results also show that migration overhead can dominate under low bandwidth, so better initial placement remains important (Wang et al., 15 Oct 2025). Other limitations are architectural: L4 assumes enough replicas for specialization, UniPrefill can regress on short prompts, MOCAP assumes wafer-scale communication, and FlowPrefill explicitly does not implement SRPT or a full size-based prefill theory. A plausible synthesis is the one already suggested in the literature: combine PD disaggregation with prompt-length-specialized prefill replicas, preserve decode-side specialization separately, and let prompt-derived cost models drive both admission and migration only where the runtime can enforce them cheaply (Yuan et al., 22 Dec 2025).