Prefill-Decode Disaggregation in LLMs

• Prefill-decode disaggregation is a serving architecture that splits LLM inference into a compute-bound prefill stage and a memory-bound decode stage.
• It optimizes key performance metrics by isolating the prompt processing and token generation phases, enabling independent tuning and efficient KV cache transfers.
• Extensions in this design include multi-turn, multi-model, and hardware-specialized variants that adjust resource ratios to meet strict SLOs.

Prefill-decode disaggregation is a serving architecture in which the prefill stage and the decode stage of LLM inference are executed on distinct resources rather than inside one monolithic worker. The rationale is a phase asymmetry that recurs across the literature: prefill processes the prompt in parallel and builds the initial KV cache, so it is typically compute-bound, whereas decode generates tokens autoregressively and repeatedly accesses weights and KV state, so it is typically memory-bandwidth-bound (Li et al., 9 Mar 2026, Kanani et al., 16 Mar 2026, Li et al., 5 Mar 2026). This separation has become a standard or dominant architecture for modern LLM serving, but it is not a single fixed design. Published systems range from classical GPU-pool separation to multimodal Encode-Prefill-Decode decompositions, multi-turn and multi-model variants, hybrid aggregation-disaggregation schedulers, and phase-specialized hardware (Bai et al., 5 Jan 2026, Singh et al., 2024).

1. Core execution model and phase asymmetry

In the standard autoregressive pipeline, prefill consumes the full input prompt, computes the first token, and initializes the KV cache; decode then generates subsequent tokens one step at a time using that cache. The dominant service metrics therefore split naturally across phases: TTFT is driven primarily by prefill, while TPOT or TBT is driven primarily by decode. This decomposition is not merely a software convenience. Roofline and microarchitectural analyses report that the two phases stress hardware in opposite ways, which is the foundational argument for disaggregation (Kanani et al., 16 Mar 2026).

The asymmetry is explicit in both systems and hardware papers. One study states that “the prefill (prompt processing) phase is compute-bound and highly parallel, whereas the decode (token generation) phase is memory bandwidth-bound and inherently sequential due to intensive key-value (KV) cache accesses” (Kanani et al., 16 Mar 2026). Another treats P/D disaggregation as a deployment pattern in which request handling is split into prefill instances and decode instances so that TTFT and TPOT can be optimized independently (Li et al., 5 Mar 2026). For multi-turn workloads, this distinction becomes even sharper because historical context accumulation inflates prefill work while decode remains latency-sensitive (Li et al., 9 Mar 2026).

This phase split is now embedded in mainstream serving stacks. One multi-turn study describes PD disaggregation as the standard architecture for modern LLM inference engines and notes its use in frameworks such as vLLM, SGLang, TensorRT-LLM, LMDeploy, and NVIDIA Dynamo (Li et al., 9 Mar 2026). A plausible implication is that “prefill-decode disaggregation” now names both a performance principle and a deployment baseline against which newer systems are evaluated.

2. Canonical architecture and the centrality of KV-cache transfer

In the canonical arrangement, a prefill node computes the request’s KV states and then transfers the resulting KV cache to a decode node, which continues generation without prefill interference (Li et al., 9 Mar 2026). This isolates the two bottlenecks, but it also introduces a new dependency: the end-to-end benefit now depends on how quickly and efficiently KV state can move between resources. That is why many papers treat the P→D handoff, rather than the phase split alone, as the true systems problem.

Benchmarking work makes this dependency explicit by comparing transfer media. One study evaluates colocated baselines against disaggregated setups in which KV cache is transferred directly GPU-to-GPU over the shared PCIe bridge, offloaded through CPU DRAM with a Redis lookup table, or stored on disk via fs_connector with page cache bypass (Li et al., 14 Nov 2025). Its main conclusion is that performance benefits from prefill-decode disaggregation are not guaranteed and depend on request load and KV transfer medium; stage-wise independent frequency scaling also does not yield energy savings because disaggregated serving has inherently higher energy consumption in the evaluated setups (Li et al., 14 Nov 2025).

Recent systems therefore optimize the handoff itself. In EPD-Serve, the multimodal P–D boundary uses hierarchical grouped KV cache transmission rather than waiting for prefill completion and sending the whole KV in one burst. The reported effect is a rise in communication-computation overlap from 15.27% to 98.78% for 1024-token input and from 25.08% to 99.92% for 2048-token input, together with bandwidth utilization improvements from 7.98 GB/s to 12.58 GB/s at length 1024 and from 10.66 GB/s to 11.71 GB/s at length 2048 (Bai et al., 5 Jan 2026). The same paper frames this as “deep communication-computation overlap” for the P–D stage.

The importance of handoff overhead is especially visible under bursty load. On a production-style A100 cluster with 2 prefill and 2 decode nodes, Kairos reports that prefill execution accounts for only 2–23% of P95 TTFT; queueing and inter-node GPU-GPU KV-cache transfer account for the rest (Arun et al., 2 Jul 2026). This suggests that once phase interference is removed, queueing policy and state movement often dominate the tail.

3. Resource ratios, SLOs, and the aggregation–disaggregation design space

A distinct branch of the literature treats PD disaggregation as a resource-allocation problem. In one analytical model, total throughput is

$$TP_{total}=\frac{N_{req}(L_{in}+L_{out})}{T_{total}},$$

with phase times

$$T_{prefill}=\frac{N_{req}L_{in}}{TP_{prefill}N_{prefill}}, \qquad T_{decode}=\frac{N_{req}L_{out}}{TP_{decode}N_{decode}}.$$

Balancing the pipeline yields

$$R_{P/D}=\frac{N_{prefill}}{N_{decode}}=\frac{L_{in}TP_{decode}}{L_{out}TP_{prefill}},$$

so the prefill-to-decode ratio depends on input length, output length, and effective phase throughput rather than on total throughput target alone (Li et al., 5 Mar 2026). The same paper combines M/M/1 modeling for TTFT-constrained prefill with empirical TPOT-constrained decode benchmarking, and validates a 3P4D deployment for a 5 M TPM target with TTFT $=2$ s and TPOT $=20$ ms (Li et al., 5 Mar 2026).

Dynamic systems build on the same idea but treat the optimal ratio as time-varying. DOPD monitors workload, forecasts average input length, average output length, and request count with ARIMA, maps those forecasts to profiled TTFT and TPOT tables, and proactively adjusts the counts of P-instances and D-instances (Liao et al., 26 Nov 2025). Reported gains include up to $1.5\times$ goodput improvement, P90 TTFT reductions up to 67.5%, P90 TPOT reductions up to 22.8%, and over 99% SLO attainment while using less additional resources (Liao et al., 26 Nov 2025).

At the same time, several papers reject the premise that PD disaggregation is universally optimal. TaiChi argues that aggregation is optimal for tight TTFT and relaxed TPOT, while disaggregation excels for strict TPOT and relaxed TTFT; under balanced TTFT and TPOT SLOs, neither pure approach delivers optimal goodput (Wang et al., 4 Aug 2025). Its unified architecture combines differentiated-capability instances, flowing decode scheduling, and length-aware prefill scheduling, and reports up to 77% goodput improvement over state-of-the-art systems under balanced SLOs (Wang et al., 4 Aug 2025). This debate has shifted the field from “whether to disaggregate” toward “how much disaggregation is required for a given SLO regime.”

4. Multi-turn and multi-model extensions

Multi-turn serving exposes shortcomings that are less visible in single-turn benchmarks. Standard PD disaggregation still sends every new turn back through prefill, recomputes historical context, and performs another P→D KV transfer. “Not All Prefills Are Equal” identifies two inefficiencies in this setting: repeated prefilling of the new prompt plus prior response, and repeated KV-state transfers between prefill and decode nodes (Li et al., 9 Mar 2026). On real multi-turn data, up to 99% of prefill cost can come from recomputing historical tokens, which motivates distinguishing full prefill from append-prefill. The paper’s PPD system lets decode nodes perform append-prefill locally when appropriate, reducing Turn 2+ TTFT by 68% overall and lowering KV transfer volume by about 75% in multi-turn settings (Li et al., 9 Mar 2026).

AMPD addresses the same interleaved prefill-decode pattern from a coordination perspective. It binds each session to a decode worker, then adaptively decides whether incremental prefill should run locally on that decode worker or remotely on a prefill worker, and further reorders prefill queues to maximize TTFT SLO satisfaction (He et al., 16 Feb 2026). On ToolBench, GAIA, HotpotQA, and DuReader, it reports SLO attainment improvements of up to 967.54% over Dynamo and up to 3435.1% over vLLM; adaptive routing alone sends about 13.9%–31.7% of prefill tasks locally (He et al., 16 Feb 2026). Kairos pursues a closely related idea with proactive prefill deflection from saturated prefill nodes to decode nodes, searching for the largest chunk schedule that preserves decode TBT SLOs; on production-style traces it reports up to 81% lower P95 TTFT and up to 79% higher SLO attainment (Arun et al., 2 Jul 2026).

The decode side of multi-turn serving can also become imbalanced after requests have already been handed off. ARES observes that static prefill-to-decode scheduling fails when output lengths vary widely, and introduces decode-phase rescheduling based on a continuous remaining-length predictor derived from the target LLM’s last-layer hidden state (Wang et al., 15 Oct 2025). It reports a 49.42% MAE reduction for length prediction, a 93.28% reduction in predictor parameters, P99 TPOT reductions of 74.77%, and up to 2.24 times higher goodput (Wang et al., 15 Oct 2025).

In multi-model and agentic workloads, disaggregation removes intra-model phase interference but does not remove cross-model redundancy. PrefillShare addresses repeated processing of the same shared context by factorizing the model into a frozen shared prefill module and task-specific decode modules, then routing requests so that the same session stays pinned to the same prefill worker (Woo et al., 12 Feb 2026). It reports accuracy close to full fine-tuning and serving gains of up to 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads (Woo et al., 12 Feb 2026). The conceptual shift is important: the unit of sharing becomes not only hardware, but also a prefill-compatible representation.

5. Multimodal, retrieval-aware, and cross-cluster variants

For large multimodal models, the PD boundary is no longer sufficient because an additional Encode stage precedes text prefill. EPD Disaggregation therefore separates encoding, prefill, and decode onto dedicated resources, adds asynchronous multimedia token migration, and introduces intra-request parallelism for encoding (Singh et al., 2024). Reported benefits include up to 15x lower peak memory utilization, up to 22x larger batch sizes, up to 10x more images per request, up to 2.2x larger KV caches, TTFT reductions up to 71.9%, and up to 90–100% improvement in SLO attainment relative to systems that do not disaggregate (Singh et al., 2024). EPD-Serve extends this line on Ascend hardware with stage-level orchestration, asynchronous feature prefetching, grouped KV transmission, and multi-route scheduling, reporting 57.37–69.48% end-to-end throughput improvement over PD-disaggregated deployment under high concurrency while maintaining TTFT below 2000 ms and TPOT below 50 ms (Bai et al., 5 Jan 2026).

Retrieval-augmented generation introduces another stage whose resource profile differs from both prefill and decode. Trinity argues that vector search should not remain entangled with LLM inference or be co-located opportunistically with prefill or decode; instead, it consolidates retrieval into a dedicated shared vector-search GPU pool with continuous batching and stage-aware scheduling (Liu et al., 1 Dec 2025). The paper’s qualitative conclusion is that vector search can then coexist with PD-disaggregated serving without sacrificing SLOs for either prefill or decode.

The same logic has been extended across datacenter boundaries. Prefill-as-a-Service defines the KV throughput of a model instance as

$\Phi_{kv}(l)=\frac{S_{kv}(l)}{T_{prefill}(l)},$

with cluster egress requirement

$$B_{out}\approx \frac{N}{P}\cdot \Phi_{kv}(L_{avg}),$$

and argues that dense-attention models keep P and D tightly coupled inside one high-bandwidth network domain because KV throughput is too large (Qin et al., 16 Apr 2026). Hybrid-attention architectures reduce KV size enough to make cross-cluster transport plausible, but only if offloading is selective and bandwidth-aware. PrfaaS therefore offloads only long-context prefills to standalone compute-dense prefill clusters and keeps short requests local. In a case study with an internal 1T-parameter hybrid model, the reported result is 54% higher throughput than homogeneous PD and 32% higher throughput than a naive heterogeneous baseline, while mean TTFT drops from 4.44 s to 2.22 s and P90 TTFT falls by 64% (Qin et al., 16 Apr 2026).

6. Hardware specialization, intra-GPU disaggregation, and limits of the paradigm

Several recent works move PD disaggregation from a software scheduling technique to a hardware design principle. DUET argues that a homogeneous accelerator cannot satisfy all requirements of hybrid Mamba–Transformer inference and introduces a hardware-level disaggregation in which a Prefill package uses systolic-array chiplets with off-package GDDR7 and a Decode package uses vector-unit arrays with in-package HBM3e (Kanani et al., 16 Mar 2026). Both packages are runtime-configurable for mixed Mamba and attention layers. On Nemotron-H-56B, Zamba2-7B, and Llama3-8B across four workloads, DUET reports a geometric-mean 4x lower TTFT than B200, 1.4x higher decode throughput, and 1.5x lower TBT (Kanani et al., 16 Mar 2026).

SPAD makes a related “less-is-more” argument for specialized chips. Its Prefill Chips enlarge systolic arrays and replace HBM with GDDR7, while its Decode Chips retain high bandwidth and reduce compute capacity (Zhang et al., 9 Oct 2025). Relative to modeled H100s, the reported chip-level results are 8% higher prefill performance at 52% lower hardware cost for Prefill Chips and 97% of decode performance with 28% lower TDP for Decode Chips. Cluster-level simulations on production traces report 19%–41% lower hardware cost and 2%–17% lower TDP while matching baseline performance (Zhang et al., 9 Oct 2025). On edge FPGAs, PD-Swap time-multiplexes a reconfigurable attention partition between a compute-heavy prefill engine and a bandwidth-optimized decode engine, achieving up to 27.8 tokens/s decode throughput and 1.3x–2.1x speedup over prior work without extra area cost (Zhang et al., 12 Dec 2025).

Other systems attempt to recover disaggregation-level isolation without separate devices. DuetServe operates in aggregated mode by default and activates SM-level GPU spatial multiplexing only when predicted TBT degradation threatens the SLO, reporting up to 1.3x higher total throughput while maintaining low generation latency (Gao et al., 6 Nov 2025). Nexus similarly argues that the benefits of PD disaggregation can be achieved within a single engine by dynamically partitioning SMs between prefill and decode; it reports up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM, as well as 1.4x higher throughput than vLLM-disaggregation while using half the GPUs (Shi et al., 9 Jul 2025). These results do not negate classical PD disaggregation, but they show that “disaggregation” can be logical and resource-partitioned rather than strictly cross-device.

Phase awareness has also entered model optimization. A PD-specific pruning method performs iterative block removal independently for prefill and decode and prunes transmitted KV cache only for decode reuse, retaining full KV during prefill. Under the default settings, it reports a 20.56% inference speedup and a 4.95 times reduction in data transmission bandwidth consumption (Zhang et al., 29 Aug 2025). Beyond autoregressive Transformers, Prefilling-dLLM presents a training-free prefill-decode disaggregation framework for diffusion LLMs that caches chunked prefix KV once and reuses it across denoising steps; on LongBench and InfiniteBench it reports 9.1–28.0x speedup at 8K–32K contexts (Xiong et al., 9 Jun 2026).

The literature is correspondingly careful about limits. One systematic benchmarking study concludes that PD disaggregation is not inherently better than colocated serving and that its value depends on batch size, request load, and KV transfer medium; independent DVFS does not overcome the energy overhead of disaggregated serving in the evaluated configurations (Li et al., 14 Nov 2025). TaiChi likewise shows that pure aggregation can be preferable under TTFT-first SLOs, while disaggregation dominates under TPOT-first SLOs (Wang et al., 4 Aug 2025). This suggests that prefill-decode disaggregation is best understood not as a universal default, but as a broad design space for matching phase behavior, communication paths, and SLO priorities.