General Non-Clairvoyant KV-Cache Scheduling via Regime-Aware Routing

Abstract: We study non-clairvoyant scheduling for batched LLM inference under a hard Key-Value (KV) cache memory budget. Each request has a known prompt length but an unknown response length, and its memory footprint comprises a fixed prompt component together with a response component that grows with each decoded token. At each decoding round, the scheduler chooses a feasible batch of active requests; evicting a request discards its accumulated cache states, wasting prior computation. The goal is to minimize total completion time against the optimal clairvoyant schedule that knows all response lengths. We present the first constant-competitive algorithm for arbitrary prompt lengths and arbitrary response lengths with no additional assumptions. Rather than relying on a single universal scheduling policy, our algorithm is built on a novel regime-aware routing framework. Specialized sub-schedulers handle different memory-growth geometries, while a meta-scheduler time-shares the memory budget across them and dynamically routes each job as its execution progressively reveals its behavior. This framework also yields constant-competitive guarantees for makespan and for total completion time under online arrivals.

• The paper introduces a regime-aware routing algorithm that dynamically separates large, prompt-heavy, and response-heavy jobs, achieving a 996-competitive total-completion-time guarantee without knowing response lengths.
• The framework combines geometric-cap rectangle scheduling for large and prompt-heavy jobs with geometric slicing for response-heavy jobs, delivering sub-ratios of 36 and approximately 78.67, respectively.
• The results show that no single priority rule can balance memory utilization and area-order completion, while specialized routing also extends constant-competitive guarantees to makespan and online arrivals.

Problem setting and motivation

The paper studies replica-level scheduling for batched LLM inference under a hard Key-Value (KV) cache memory budget $M$. Each request (job) has a known prompt length $s_i$ and an unknown response length $o_i$; its memory footprint is $s_i + u_{i,t} + 1$ while active, where $u_{i,t}$ is the number of tokens decoded in the current uninterrupted attempt. Time proceeds in discrete decoding rounds; at each round the scheduler picks a feasible batch, and killing a job discards all accumulated KV state. The objective is total completion time against an optimal clairvoyant schedule that knows all response lengths. The model captures continuous batching and block-based cache management as deployed in modern serving stacks, and inherits from classical non-clairvoyant scheduling the crucial twist that resource demand grows during execution: uncertainty in response length is directly uncertainty in future memory demand.

Prior theoretical work had established guarantees only under restrictions—clairvoyance (2607.09248), identical prompts, or large-memory regimes ($s_i + o_i = o(M)$). This paper removes all such restrictions and proves the first constant-competitive non-clairvoyant algorithm for arbitrary prompt lengths and arbitrary response lengths, with no additional assumptions.

Main result

The central theorem states that for every feasible batch instance there exists a fully non-clairvoyant algorithm whose total completion time satisfies $\mathrm{ALG}(J) = O(1) \cdot \mathrm{OPT}(J)$; the explicit instantiation via the routing meta-scheduler with cutoff $\tau = M/4$ yields a competitive ratio of 996. The algorithm runs in polynomial time (an event-driven implementation has $\mathrm{poly}(n, \log M)$ events), and the same framework gives constant-competitive guarantees for makespan and for total completion time under online arrivals.

The analysis rests on a memory-time area lower bound: defining $A(s,o) = s\cdot o + o(o+1)/2$, the optimum is at least $s_i$0 when jobs are ordered by nondecreasing area. Two sufficient conditions follow—high memory utilization and area-order completion—and the entire algorithmic design targets these simultaneously.

Impossibility of a single priority rule

A key structural finding is that no single global priority rule achieves a constant approximation, even clairvoyantly. With job types $s_i$1 of area $s_i$2 and $s_i$3 of area $s_i$4 on budget $s_i$5: prioritizing $s_i$6 leaves memory utilization at $s_i$7 (ratio $s_i$8), while prioritizing $s_i$9 inverts the area order and blocks many small-area jobs (also ratio $o_i$0). These two failure modes impose opposite pressures, so high utilization and area-consistent ordering cannot be reconciled by one ordering criterion. This impossibility motivates the regime decomposition that structures the whole algorithm.

Three geometric regimes

Jobs are partitioned by a prompt cutoff $o_i$1 into:

• Large jobs ($o_i$2): each nearly fills memory alone.
• Small prompt-heavy jobs ($o_i$3, $o_i$4): footprint stays within a constant factor of the prompt.
• Small response-heavy jobs ($o_i$5, $o_i$6): footprint grows well beyond the prompt; here $o_i$7, so response-length order approximates area order.

Each regime admits a constant-competitive sub-scheduler in isolation, but the partition itself is unknown to a non-clairvoyant scheduler—the prompt-heavy/response-heavy distinction depends on the unobserved response length.

Rectangle strip scheduling for large and prompt-heavy jobs

For the first two regimes, memory growth is within a constant factor of the prompt length, reducing the problem to non-clairvoyant rectangle strip scheduling: fixed width $o_i$8, unknown processing time $o_i$9, restarts lose progress. The authors note this variant appears not to have been studied before and may be of independent interest.

The scheduler maintains attempts with geometrically increasing length caps $s_i + u_{i,t} + 1$0 and area budgets $s_i + u_{i,t} + 1$1, greedily admitting the minimum-budget pending attempt that fits. The analysis hinges on a queue monotonicity lemma: the minimum pending budget is nondecreasing over time, and every active attempt's budget is bounded by every pending attempt's. The proof exploits that replacement attempts have the same width as their expired predecessors, so released memory immediately accommodates them before older, larger-budget attempts can claim it. Combined with a width-range utilization bound $s_i + u_{i,t} + 1$2, this yields per-job bounds $s_i + u_{i,t} + 1$3 and hence a constant guarantee whenever widths lie in a bounded range. Applied with $s_i + u_{i,t} + 1$4, both the large branch and the prompt-heavy branch achieve a 36-competitive ratio.

The paper is explicit that this guarantee is fragile: even a few response-heavy jobs break it, because queue monotonicity fails once effective widths grow, allowing large-area jobs to finish before small-area ones.

Geometric slicing for response-heavy jobs

For response-heavy jobs, the scheduler extends the geometric-slicing approach of prior identical-prompt work to heterogeneous prompts. Phases use doubling response caps $s_i + u_{i,t} + 1$5; in phase $s_i + u_{i,t} + 1$6, all jobs with $s_i + u_{i,t} + 1$7 are eligible and are scheduled by the Staggered Pipeline Scheduling (SPS) subroutine using a common proxy prompt $s_i + u_{i,t} + 1$8. Feasibility holds because real memory never exceeds proxy memory within a phase, and each job completes in the first phase whose cap reaches $s_i + u_{i,t} + 1$9, preserving approximate area order. For $u_{i,t}$0 the competitive ratio is 236/3 ≈ 78.67, proved by charging phase durations to processing and class-area lower bounds. Symmetrically, even a few prompt-heavy jobs make this scheduler's ratio unbounded, since SPS then underutilizes memory and response order diverges from area order.

Meta-scheduling and routing

Two meta-schedulers combine the branches. The black-box version assumes a known partition and round-robin time-shares sub-schedulers over doubling stages of length $u_{i,t}$1, completing completed jobs as dummies; it loses at most a factor $u_{i,t}$2 for $u_{i,t}$3 branches. The routing version resolves the unknown partition online: small jobs start in the prompt-heavy branch, which either completes them or certifies $u_{i,t}$4 upon reaching $u_{i,t}$5 tokens, at which point they move to the response-heavy branch. A routing-loss lemma shows certification and completion both fit within constant factors ($u_{i,t}$6 per job), because a job certified in stage $u_{i,t}$7 is available to the response branch in the same stage. Combining branch guarantees (36, 36, 236/3) with the meta-scheduler overheads yields the final 996-competitive bound.

Extensions

The framework yields several complementary results. In the clairvoyant setting, a direct rectangle-packing algorithm (using Steinberg's 2-approximation strip packing as a black box) achieves a 16-approximation for general instances, improving the analogous $u_{i,t}$8 bound from capacitated scheduling; area-order greedy gives $u_{i,t}$9 in the large-memory regime, matching concurrent work. The same routing algorithm is constant-competitive for makespan. For online arrivals, an online variant remains $s_i + o_i = o(M)$0-competitive for total completion time, with arrival delays charged against the lower bound $s_i + o_i = o(M)$1. The paper notes that for flow time under adversarial arrivals, an $s_i + o_i = o(M)$2 lower bound precludes constant competitiveness, making total completion time the natural latency objective here.

Limitations and open questions

Several caveats bear directly on the results. The constant factors are large—the main theorem's 996 and the response-heavy branch's ≈78.67 are far from practical—and the analysis does not attempt to tighten them. The competitive guarantee is against a clairvoyant benchmark on the offline batch model; the online result covers total completion time but not flow time, where strong lower bounds apply. The regime decomposition depends on the specific cutoff structure ($s_i + o_i = o(M)$3, geometric caps with $s_i + o_i = o(M)$4); whether smaller constants or alternative decompositions improve the bounds is not addressed. The systems recommendation—routing regimes to dedicated replicas—implicitly relies on three-fold resource augmentation relative to the single-replica theory, and the paper does not evaluate empirical performance on real traces or trained models. Whether the non-clairvoyant rectangle strip problem admits better constants than the black-box combination given here remains open, as does closing the gap between the clairvoyant 16-approximation and the $s_i + o_i = o(M)$5 large-memory bound for general instances.

Conclusion

This paper establishes the first constant-competitive non-clairvoyant scheduling guarantee for batched LLM inference with arbitrary prompt and response lengths, via a regime-aware routing framework that decomposes jobs by memory-growth geometry, solves each regime with a specialized sub-scheduler, and routes jobs dynamically as execution reveals their class. Beyond the worst-case guarantee, the decomposition formalizes why separating global routing from local scheduling—a pattern already present in production serving stacks—is algorithmically sound: simple local rules, routed by geometric class, suffice for robust performance.