Flowing Decode Scheduling in LLM Inference

• Flowing Decode Scheduling is a family of scheduling mechanisms that control autoregressive decoding by managing request admission, batching, and migration.
• It mitigates bottlenecks like unbounded KV-cache growth and deadline heterogeneity, ensuring stable memory usage and reduced per-token latency.
• Implementations such as Kairos, HydraInfer, and TaiChi demonstrate enhanced TPOT SLO adherence and throughput improvements through adaptive, flow-controlled policies.

Flowing Decode Scheduling denotes a family of scheduling mechanisms that regulate the decode phase of autoregressive inference, most prominently in LLM serving, by controlling admission into decoding, forming sub-batches, prioritizing ongoing decode work, or migrating active decode requests across heterogeneous resources so that per-token latency, KV-cache pressure, and system stability remain bounded under variable request lengths and load. In recent LLM systems, the term covers slack-guided adaptive batching, decode-first stage-level batching, flow-controlled admission, and migration-based TPOT control; in coding theory, the same phrase has also been used for update-order scheduling in layered or iterative decoders, where the objective is pipeline efficiency or reduced unnecessary edge updates rather than LLM serving (Wang et al., 4 May 2026, Dong et al., 19 May 2025, Wang et al., 4 Aug 2025, Dong et al., 13 Apr 2026, Chang et al., 25 May 2025, Murayama et al., 2018).

1. Terminological scope and principal variants

The phrase is not tied to a single canonical algorithm. Across the cited literature, it names several distinct control policies that share a common structural intuition: decode work is treated as a regulated flow rather than as an undifferentiated continuously batched stream.

Variant	Mechanism	Primary control signal
Kairos (Wang et al., 4 May 2026)	Slack-guided adaptive batching on active decode requests	Minimum slack and LUT-estimated per-step time
HydraInfer (Dong et al., 19 May 2025)	Always include all ongoing decode tasks first; fill leftover budget with chunked-prefill	Token budget $\tau$ from $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$
Flow-controlled scheduling (Dong et al., 13 Apr 2026)	Smooth prompt admissions into the active set with token-bucket $B_t$ and LIFO eviction	Memory $M(t)$, capacity $C$, admission budget $B_t$
TaiChi (Wang et al., 4 Aug 2025)	Offload long-running decodes from D-heavy to P-heavy and backflow SLO-risky requests	HBM watermark $M$, approaching factor $\alpha$
Arrow (Wu et al., 17 May 2025) / FlexNPU (Gu et al., 3 Jun 2026)	Rebalance decode capacity at pool or time-slice level	Running token load, token interval, bandwidth utilization, queue ratios

This suggests that “flowing” is best understood as a family resemblance term. The shared design move is explicit control over the rate, placement, or composition of decode work, rather than unconditional inclusion of all active requests in every decode step.

2. Decode-stage bottlenecks that motivate the method

In autoregressive LLM serving, once a prompt enters the active set, its KV-cache memory footprint grows by one token at each decoding step, and because the total number of tokens to be generated is unknown on arrival, the per-request memory usage is stochastic and unbounded until completion. Under high stochastic load, many requests may join decoding simultaneously, leading to bursty peaks in GPU memory usage; if memory is exhausted, the system must evict or drop requests, which degrades latency and throughput and destabilizes the queueing dynamics (Dong et al., 13 Apr 2026).

A second bottleneck is deadline heterogeneity inside the decode set. In disaggregated serving, the decode stage holds each request’s growing KV cache in GPU memory and processes one token at a time across all active requests. Traditional continuous batching ties all active requests into one big batch each step, causing short-ahead requests to idle behind stragglers. Kairos formulates this as a slack-management problem: short requests should “water-fall through multiple steps rapidly,” while longer requests are only batched when strictly necessary so as not to hold others up (Wang et al., 4 May 2026).

A third bottleneck is phase interference. HydraInfer isolates decoding on D instances because decode is strictly memory-bound, has low arithmetic intensity, and benefits from large batched KV-cache lookups; preventing prefill or encode compute from evicting or stalling decode’s working set reduces TPOT. Related systems frame the same issue as PD imbalance or resource misallocation: static prefill/decode splits can leave compute idle, leave bandwidth idle, or amplify TTFT/TPOT violations when workload composition changes (Dong et al., 19 May 2025).

3. Algorithmic forms in LLM inference

Slack-guided adaptive batching. Kairos defines the decode-side slack of request $i$ at step $k$ by

$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$0

with the simplified single-step view $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$1. The scheduler computes the minimum slack across active requests, sorts candidates by increasing $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$2, and greedily adds request $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$3 only if $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$4 and

$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$5

If no request can be packed under these conditions, the policy falls back to decoding all together. The stated invariants are: never violate the smallest slack among active requests, and only expand a batch when doing so increases end-to-end throughput (Wang et al., 4 May 2026).

Decode-first token-budget scheduling. HydraInfer computes a token budget

$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$6

then fills each scheduling round with all ongoing decode tasks first. Only if token budget remains does it pack chunked-prefill, and only if no decode or prefill remains does a separate encode batcher run image encoding. The policy therefore always “flows” existing decode work first, up to a profiled token budget derived from the per-token latency SLO (Dong et al., 19 May 2025).

Admission smoothing under unknown decode length. In the flow-control formulation, the scheduler does not treat decode batching itself as the only degree of freedom. Instead, it limits how many new prompts enter decoding in each slot through a token-bucket style admission budget $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$7. New arrivals are enqueued into a waiting queue $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$8, up to $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$9 prompts are tentatively admitted, and if $B_t$0, the scheduler evicts requests by LIFO until the capacity constraint is restored, returning evicted requests to $B_t$1. The primary objective is not local batching efficiency but smoothing the rate at which prompts join the active set (Dong et al., 13 Apr 2026).

Migration-based flowing. TaiChi starts all decodes on D-heavy instances and then applies a degrading-flow rule on D-heavy nodes: select the smallest set $B_t$2 of longest-so-far requests such that $B_t$3, then migrate $B_t$4 to P-heavy instances. The dual backflow rule on P-heavy nodes is

$B_t$5

and all such requests are migrated back to D-heavy instances. This realizes per-request TPOT shifting without predicting final output length (Wang et al., 4 Aug 2025).

Resource-share feedback. FlexNPU expresses dynamic PD co-location as a resource-allocation problem with total resources $B_t$6 split into $B_t$7 and $B_t$8, maximizing

$B_t$9

subject to $M(t)$0. Its practical implementation uses a daemon-level feedback loop that adjusts decode share according to memory-bandwidth utilization and phase queue ratios, then dispatches kernel launches in time-sliced fashion across decode and prefill streams (Gu et al., 3 Jun 2026).

4. Formal objectives: stability, deadlines, and capacity control

The most explicit stability analysis appears in the flow-control model. At each slot $M(t)$1, arrivals are Poisson with rate $M(t)$2, each prompt draws an i.i.d. decode length $M(t)$3, active request $M(t)$4 has generated $M(t)$5 tokens, each token uses $M(t)$6 units of memory, and total memory consumption is

$M(t)$7

A first-order necessary condition for stability is

$M(t)$8

and if $M(t)$9, then even an ideal scheduler that runs work-conservingly with no preemptions will see memory demand grow unboundedly. Under a constant admission cap $C$0, the waiting queue evolves as

$C$1

and with Lyapunov function $C$2, the conditional drift satisfies

$C$3

where $C$4. The sufficient memory-capacity condition

$C$5

then implies no overflow will ever occur (Dong et al., 13 Apr 2026).

Other formulations emphasize SLO feasibility rather than positive recurrence. Kairos enforces

$C$6

so that none of the selected requests in $C$7 misses its next-token deadline. Arrow enforces two instance-level constraints for candidate decode placement:

$C$8

with

$C$9

and uses the rough fluid model $B_t$0 as an explanation for required decode capacity (Wang et al., 4 May 2026, Wu et al., 17 May 2025).

Adjacent deadline-aware theory appears in SLAI, where the last-schedulable time for decode iteration $B_t$1 is

$B_t$2

and any request with current time $B_t$3 is marked critical. This is not labeled “flowing decode scheduling” in the narrowest sense, but it formalizes the same idea that decode order should be driven by per-token deadline proximity rather than static batching rules (Bari et al., 1 Aug 2025).

Taken together, these formulations suggest three distinct mathematical lenses on the subject: queueing stability under unknown decode length, hard or soft TPOT deadline control, and online resource-allocation under phase asymmetry.

5. Systems, architectures, and reported empirical behavior

The reported performance gains vary with architecture and objective. In the flow-controlled scheduler, synthetic experiments with three classes $B_t$4 reported average latency of approximately $B_t$5 ms versus approximately $B_t$6 ms for greedy baselines, $B_t$7-tile latency of $B_t$8 ms versus $B_t$9 ms, request throughput of $M$0 req/s versus $M$1 req/s, token throughput of $M$2 tok/s versus $M$3 tok/s, and KV-cache occupancy variance reduced by approximately $M$4. Under unknown decode length with heuristic LIFO eviction, latencies and throughputs remained within $M$5 of the known-length case and outperformed other prediction-based heuristics by $M$6–$M$7. On Lmsys-Chat-1m, low-load average latency was $M$8 ms versus $M$9 ms and p95 was $\alpha$0 ms versus $\alpha$1 ms; at high load, request throughput reached $\alpha$2 req/s versus $\alpha$3 req/s and token throughput $\alpha$4 tok/s versus $\alpha$5 tok/s, with peak memory about $\alpha$6 below $\alpha$7 and roughly $\alpha$8 less volatility than greedy schemes (Dong et al., 13 Apr 2026).

Kairos reports that at QPS $\alpha$9, TPOT SLO attainment rises from $i$0 for DistServe to $i$1, a $i$2 percentage-point improvement, and that at QPS $i$3 median per-request decode throughput increases from $i$4 tok/s to $i$5 tok/s, a $i$6 relative gain. HydraInfer reports up to $i$7 higher inference throughput compared to state-of-the-art systems on a single-node $i$8H800 GPU cluster while meeting the $i$9th percentile request SLO; its detailed examples give goodput changes from approximately $k$0 to approximately $k$1 req/s on LLaVA-1.5 (TextCaps), approximately $k$2 to approximately $k$3 req/s on LLaVA-NeXT (VizWiz), and approximately $k$4 to approximately $k$5 req/s on Qwen2-VL (VizWiz), together with $k$6th-percentile TPOT of about $k$7 s versus $k$8–$k$9 s on vLLM and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$00th-percentile TTFT of about $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$01 s versus about $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$02 s for LLaVA-1.5 on TextCaps (Wang et al., 4 May 2026, Dong et al., 19 May 2025).

TaiChi places flowing decode scheduling inside a broader latency-shifting framework. Under balanced SLOs it reports goodput uplift of $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$03–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$04 versus PD aggregation and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$05–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$06 versus PD disaggregation for chatbot SLO2 on Qwen-14B, and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$07–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$08 versus PD aggregation and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$09–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$10 versus PD disaggregation for summarization SLO2 on Qwen-14B. It also reports P90 normalized TPOT reduced by $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$11–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$12 compared to PD aggregation, and that adding flowing decode alone improves SLO attainment by approximately $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$13 under summarization SLO1 (Wang et al., 4 Aug 2025).

Several adjacent systems attack the same decode-side imbalance from different control surfaces. Arrow dynamically adjusts the number of instances handling prefill and decode tasks and reports up to $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$14 and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$15 higher request serving rates compared to PD-colocated and PD-disaggregated baselines, with per-trace gains of $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$16, $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$17, and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$18 over vLLM in the listed workloads. FlexNPU uses transparent NPU virtualization to support dynamic PD co-location and reports throughput improvements of $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$19 and $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$20 over static PD disaggregation on a $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$21-card Ascend 910C deployment of DeepSeek-R1, while on Qwen2.5-7B it reports TTFT reductions of $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$22–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$23 with TPOT within $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$24. FlowKV, although framed as load-aware scheduling plus low-latency KV transfer rather than flowing decode scheduling, reports average KV transfer latency reduced from $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$25 s to $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$26 s, a $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$27 reduction, and end-to-end acceleration of $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$28–$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$29 on LongBench. SLAI reports a $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$30 reduction in median TTFT and a $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$31 increase in maximum serving capacity while meeting tail TBT latency constraints (Wu et al., 17 May 2025, Gu et al., 3 Jun 2026, Li et al., 3 Apr 2025, Bari et al., 1 Aug 2025).

6. Architectural debate, misconceptions, and non-LLM uses

A recurrent design dispute concerns whether prefill-decode aggregation or disaggregation is superior. TaiChi states that aggregation is optimal for tight TTFT and relaxed TPOT, while disaggregation excels for strict TPOT and relaxed TTFT, and that under balanced TTFT and TPOT SLOs neither approach delivers optimal goodput. This suggests that flowing decode scheduling should not be identified exclusively with disaggregated serving: some systems isolate decode, some dynamically co-locate it, and some explicitly unify aggregation and disaggregation through per-request migration or latency shifting (Wang et al., 4 Aug 2025).

A second misconception is that the term denotes only throughput optimization. In the cited LLM literature, the objective may instead be provable stability under unknown decode lengths, strict adherence to TPOT SLOs, TTFT preservation under decode-first scheduling, or dynamic balance between compute-limited prefill and bandwidth-limited decode. The commonality is not a single optimality criterion but an insistence that decode-phase decisions be made with explicit control signals rather than static FCFS or unconditional continuous batching.

Outside LLM serving, the phrase has been applied to different scheduling problems in decoding theory. In QC-LDPC layered decoding, scheduling is formulated as an asymmetric traveling-salesman problem with edge cost

$$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$32

then refined by grouping layers and forbidding or penalizing edges to preserve a degree/puncture ordering that improves decoding performance. The reported schedule reduces idle cycles per iteration from $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$33 or $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$34 to $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$35 or $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$36 at pipeline depth $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$37, and from $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$38 or $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$39 to $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$40 or $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$41 at $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$42, while recovering almost all of the BLER loss seen in the idle-only schedule (Chang et al., 25 May 2025).

In zigzag decodable fountain codes, scheduled bit-wise peeling replaces flooding with a three-stage procedure: Stage A discovers edges that activate at least once, Stage B orders them into a refined list, and Stage C iterates only those edges while pruning stale ones. The reported overall decoding-process count is reduced by more than $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$43 for $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$44, and decoding time at $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$45 falls from approximately $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$46 ms per frame to approximately $$T_{\text{decode}}(\tau)\le \text{TPOT}_{\text{SLO}}$$47 ms per frame, while decoding-erasure-rate curves coincide within simulation noise (Murayama et al., 2018).

These non-LLM uses reinforce the broader interpretation of the term: “flowing decode scheduling” is not one algorithm, but a recurring strategy of imposing structure on decode/update order so that scarce memory, bandwidth, or pipeline slots are consumed by the requests, layers, or edges that matter most at that moment.