PolyServe: Multi-SLO LLM Scheduling

• PolyServe is a multi-SLO scheduling policy that categorizes heterogeneous LLM requests into precise TPOT tiers based on token-level deadlines.
• It integrates per-tier request binning, profiler-guided admission control, and load-gradient routing to optimize goodput while meeting strict DSLOs.
• Empirical results show that PolyServe achieves significant goodput gains and robust tail-latency control through fine-grained autoscaling and dynamic scheduling.

PolyServe is a multi-SLO scheduling policy for LLM serving that targets heterogeneous token-generation latency requirements at fleet scale. Introduced in "PolyServe: Efficient Multi-SLO Serving at Scale" (Zhu et al., 17 Jul 2025), it addresses the limitations of coarse latency-sensitive (LS) versus best-effort (BE) classification by organizing latency-sensitive traffic into finer service-level objective tiers defined primarily by time-per-output-token (TPOT) and time-to-first-token (TTFT). Its design combines per-tier request binning, profiler-guided admission control, load-gradient routing, fine-grained autoscaling, and explicit tail-latency controls to maintain high SLO attainment while maximizing goodput (Zhu et al., 17 Jul 2025).

1. Problem setting and performance model

PolyServe is motivated by the observation that latency-sensitive LLM workloads are not homogeneous. The motivating examples distinguish a real-time chat system targeting $TPOT=20$ ms/token, an interactive code assistant targeting $TPOT=50$ ms/token, and document summarization tolerating $TPOT=500$ ms/token (Zhu et al., 17 Jul 2025). In that setting, treating all non-BE traffic identically either forces already tight classes to become even tighter or mixes tight and loose requests on the same server, causing the entire server to run at the smallest per-token interval (Zhu et al., 17 Jul 2025).

The paper formulates latency service objectives at token granularity. Each request $r$ has a TTFT budget and a TPOT budget. Rather than enforcing TPOT as a hard maximum gap, PolyServe adopts a deadline-based SLO (DSLO) in which the $i$-th token of request $r$ must arrive by

$t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$

This formulation permits short delays in one iteration to be absorbed by slack in later iterations (Zhu et al., 17 Jul 2025).

Token-level SLO satisfaction is encoded by

$A_{r,i} = 1$

if token $i$ of request $r$ meets its DSLO deadline, and $TPOT=50$0 otherwise. The SLO-attainment rate is defined as

$TPOT=50$1

with percentile attainment used in reporting, such as “90% of tokens meet DSLO” (Zhu et al., 17 Jul 2025). Goodput is defined as the total tokens generated within SLOs divided by elapsed time:

$TPOT=50$2

Goodput gain of a policy $TPOT=50$3 over baseline $TPOT=50$4 is $TPOT=50$5 (Zhu et al., 17 Jul 2025).

Within this model, multi-SLO serving raises three specific challenges. First, mixing different TPOTs on one server forces all requests to use the tightest TPOT. Second, each SLO tier’s load fluctuates, requiring fine-grained autoscaling per tier. Third, tail latency must be controlled for every tier, because unlike LS/BE settings, requests with different latency-sensitive SLOs cannot tolerate prolonged delays and no request may be sacrificed to help others (Zhu et al., 17 Jul 2025).

2. System architecture and request lifecycle

PolyServe sits atop any LLM serving engine, explicitly including vLLM and DeepSpeed-Inference, and consists of a global router holding per-SLO queues or “bins,” a collection of homogeneous serving instances, and a profiler database mapping $TPOT=50$6 to iteration time $TPOT=50$7 via offline measurement (Zhu et al., 17 Jul 2025). Each serving instance is a cluster of GPUs running the same model.

The request lifecycle is defined as a sequence of five steps. On arrival, each request provides prompt length $TPOT=50$8, expected decode length $TPOT=50$9, TTFT, and TPOT. The request is then placed into the queue corresponding to its TPOT tier. During scheduling, the router repeatedly picks the highest-priority pending request, defined as the earliest DSLO deadline or the nearest TTFT deadline for prefill, and scans the servers in the matching tier to find the “highest-load but still SLO-attainable” instance. For each candidate server $TPOT=500$0, the router simulates adding the request’s next token to the next iteration, uses the maximum of current KV cache and predicted future KV size to look up $TPOT=500$1 from the profiler, adds the server’s current wait time $TPOT=500$2, and checks whether

$TPOT=500$3

If the condition holds, the server can admit the request. The request is then enqueued and the server updates its load and KV-cache estimate (Zhu et al., 17 Jul 2025).

The profiling subsystem is central to these decisions. At startup, PolyServe runs microbenchmarks sweeping batch sizes $TPOT=500$4 and KV-cache lengths $TPOT=500$5, recording the end-to-end iteration time, including GEMM, attention, and communication. At runtime, the router interpolates this table to predict $TPOT=500$6 (Zhu et al., 17 Jul 2025). This suggests that PolyServe treats admissibility not as a static queueing rule but as an online feasibility test conditioned on batch growth and KV-cache evolution.

3. Multi-SLO scheduling policy

The scheduling policy begins with a fixed small set of TPOT tiers, exemplified by 20 ms, 30 ms, 50 ms, and 100 ms. Each tier $TPOT=500$7 maintains its own queue $TPOT=500$8 and a dynamically sized subpool of servers $TPOT=500$9 (Zhu et al., 17 Jul 2025). Incoming tier-$r$0 requests may be placed eagerly on servers in $r$1, while under saturation additional sharing behavior becomes possible.

Routing within a tier uses a greedy “highest-load-but-still-safe” rule. For a request $r$2 in tier $r$3, PolyServe sorts $r$4 in descending order of current load, measured as the number of queued and in-flight tokens. For each server $r$5 in that order, it simulates admission with new batch size $r$6, predicted KV cache

$r$7

and profiled iteration time $r$8. If the admission condition $r$9 holds, the request is assigned to that server (Zhu et al., 17 Jul 2025).

This sequential filling produces what the paper terms a “load gradient”: heavily loaded servers are filled first and the last server in a tier remains lightly loaded (Zhu et al., 17 Jul 2025). The significance of this pattern is operational rather than merely heuristic. It creates structure that later simplifies autoscaling, because drained servers tend to appear in a predictable order.

PolyServe also includes lazy promotion for resource sharing. When all servers in $i$0 fail the admission test and the tier is saturated, lower-priority, looser-SLO requests in $i$1 may be lazily promoted into $i$2, but only when their own tier is full (Zhu et al., 17 Jul 2025). The policy therefore allows looser-SLO traffic to share tighter-SLO instances under saturation while preserving the primary mapping from each tier to its own server subset. A plausible implication is that this mechanism exploits residual headroom in tighter tiers without collapsing the tier structure into unrestricted co-location.

4. Fine-grained autoscaling and BE-pool interaction

PolyServe’s autoscaling operates at the granularity of individual TPOT tiers rather than at the level of aggregate request rate. At any moment, each tier $i$3 has $i$4 active servers, and its queue dynamics determine whether it borrows additional capacity from the BE pool or relinquishes capacity back to it (Zhu et al., 17 Jul 2025).

Scale-up occurs when $i$5 exceeds a threshold and idle servers are available in the BE pool; one is assigned to $i$6 (Zhu et al., 17 Jul 2025). The detailed description further states that when a tier’s pending queue grows, it “borrows” a server from the BE pool if available, reconfigures batch-size limits instantly, and adds it to $i$7, noting that prefill clusters require no KV transfer (Zhu et al., 17 Jul 2025). Scale-down is driven by the least-loaded server in $i$8: if it has no in-flight requests from tier $i$9, though it may still run promoted tier $r$0 work, it is marked “pending,” and if it remains pending long enough it is removed and returned to the BE pool (Zhu et al., 17 Jul 2025).

The interaction between load-gradient routing and autoscaling is explicit. Because routing always attempts to fill higher-load servers first, servers are drained in roughly LIFO order, making removals predictable and low-impact (Zhu et al., 17 Jul 2025). The paper presents this as a means of reducing the churn of adding and removing servers in lower tiers under fluctuating load, delaying scale-up until strictly needed and thereby reducing autoscaling overhead in stateful LLM serving (Zhu et al., 17 Jul 2025).

Lazy promotion complements autoscaling. Looser-SLO requests wait first for their own tier’s servers and only overflow into tighter-tier servers if all of $r$1 are full (Zhu et al., 17 Jul 2025). In effect, PolyServe combines partitioning with controlled overflow rather than adopting either rigid isolation or unrestricted sharing.

5. Tail-latency control mechanisms

PolyServe includes two tail-latency management mechanisms intended to preserve high DSLO attainment even near maximal batching. The first is wait-time-aware scheduling. When evaluating a server $r$2 for request $r$3, the router includes both the server’s current iteration wait time $r$4 and the predicted next iteration time $r$5, requiring

$r$6

The purpose is to avoid situations in which a long queue at decode time causes a token to miss its per-token deadline despite apparently acceptable steady-state iteration cost (Zhu et al., 17 Jul 2025).

The second mechanism combines dynamic chunking with continuous chunked-prefill prediction. Under prefill-decode disaggregation, PolyServe uses a dynamic chunk size for prefill so that near the end of the prefill it sends the remaining tokens in one chunk if the remainder is less than $r$7; this saves one iteration for long prompts while maintaining bounded wait times (Zhu et al., 17 Jul 2025). The system also factors queue wait into scheduling the first decode token to meet TTFT (Zhu et al., 17 Jul 2025).

For co-located chunked prefill, PolyServe predicts in advance how many chunks will be needed, using

$r$8

and ensures that every chunk’s iteration time remains below $r$9 as the KV cache grows (Zhu et al., 17 Jul 2025). If the router foresees that later chunks would exceed $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$0, it refuses admission on that server and searches another (Zhu et al., 17 Jul 2025). This design moves admission control from an immediate next-step test toward a limited horizon forecast over the request’s chunked execution.

A common misconception in multi-class LLM serving is that batch efficiency can always be improved by aggressive co-location. PolyServe’s tail-latency controls formalize why that is not generally valid under multiple latency-sensitive SLOs: admission feasibility depends on future KV-cache growth, queue wait, and tier-specific deadlines, not only on current utilization (Zhu et al., 17 Jul 2025).

6. Implementation and empirical evaluation

PolyServe is implemented in C++, and its large-scale experiments use a 1 ms-granularity simulator driven by NVIDIA H200 profiling of LLaMA3-8B (Zhu et al., 17 Jul 2025). Evaluation uses nine traces: uniform_4096_1024, uniform_512_512, MoonCake variants, LMSYS, ShareGPT, and Splitwise, with mixed $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$1 ms and TPOT distribution $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$2 ms $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$3 ms $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$4 ms $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$5 ms $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$6 (Zhu et al., 17 Jul 2025). Baselines include random routing, reported as PD-Random and CO-Random; min-cycle-time routing, reported as PD-Minimal and CO-Minimal; and a static chunked co-location chooser, reported as CO-Chunk (Zhu et al., 17 Jul 2025).

At 90% DSLO attainment, prefill-decode disaggregation achieves on average $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$7 more goodput than the best baseline and reaches 92.5% of the theoretical optimal goodput (Zhu et al., 17 Jul 2025). Co-location yields a $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$8 gain and 72.9% of optimal goodput (Zhu et al., 17 Jul 2025). The evaluation also reports that PolyServe maintains near-uniform per-tier attainment, whereas other policies sacrifice the tightest tiers under load (Zhu et al., 17 Jul 2025).

The burstiness test swaps TPOT distributions halfway through a 300K-request trace. Under that condition, PD and CO still outperform baselines by $t_{r,i} \le TTFT_r + (i-1)\cdot TPOT_r.$9 and $A_{r,i} = 1$0 goodput, respectively, at 90% attainment (Zhu et al., 17 Jul 2025). In a cost evaluation with an unlimited server pool, PD and CO use fewer instance-seconds per request, defined as instances multiplied by seconds, than the best static chunking baseline (Zhu et al., 17 Jul 2025). Under sensitivity analysis as total servers vary from 8 to 64, per-instance goodput rises for all policies, while PD and CO remain strictly ahead and scale more gracefully (Zhu et al., 17 Jul 2025).

The scheduler itself is reported to be lightweight enough for large deployments. A single router thread can place approximately 4,825 requests per second, enough to manage hundreds of servers, and additional threads scale linearly (Zhu et al., 17 Jul 2025). This result is relevant because PolyServe’s admission logic depends on repeated server scans and profiler lookups; the measurement indicates that those control-plane costs do not dominate at the tested scale.

7. Scope, limitations, and broader implications

The paper identifies several contributions: motivating SLO-tiered pricing beyond LS/BE, articulating the batching and utilization trade-offs of fine-grained TPOT tiers, and designing a system that bins requests by TPOT, routes to the highest-load yet safe server, uses lazy promotion to share resources across tiers, builds a natural load gradient to ease autoscaling, and controls tail latency via wait-time-aware scheduling and adaptive chunking (Zhu et al., 17 Jul 2025). These contributions position PolyServe as a scheduling framework for high-throughput, multi-SLO LLM serving rather than as a model-architecture optimization.

Its stated limitations are equally specific. PolyServe assumes a small fixed set of TPOT tiers and stable profiling data (Zhu et al., 17 Jul 2025). The paper notes that one could add dynamic percentile-based tiers or online profiling to adapt to new models or hardware (Zhu et al., 17 Jul 2025). It also states that extending to heterogeneous instances or multi-model fleets is straightforward: one can maintain separate TPOT bins per model or include hardware type in the profiling key (Zhu et al., 17 Jul 2025). A further proposed extension is price-aware admission control that optimizes bundling of high- and low-paying traffic or integrates request value into routing decisions (Zhu et al., 17 Jul 2025).

These extensions indicate the conceptual boundary of the present system. PolyServe’s core abstractions are per-tier binning, load-gradient scheduling, lazy resource sharing, and token-level deadline accounting (Zhu et al., 17 Jul 2025). This suggests that its main significance lies in recasting LLM serving from binary latency stratification into a structured multi-tier control problem where admission, placement, and autoscaling are all conditioned on explicit token-level SLO feasibility.