Prefill-Decode Disaggregation
- Prefill-Decode disaggregation is an LLM serving architecture that separates prompt processing and token generation into compute-bound and memory-bound stages, respectively.
- It leverages metrics like TTFT, TPOT, and goodput to guide resource allocation and manage the transfer of large key-value caches between distinct GPU pools.
- Dynamic routing strategies such as append-prefill address multi-turn conversation challenges by reducing redundant computation and optimizing KV-cache transfers.
Searching arXiv for the cited works on Prefill-Decode disaggregation and closely related systems. Prefill-Decode (PD) disaggregation is a LLM serving architecture in which the prompt-processing prefill phase and the autoregressive decode phase execute on separate resources, typically distinct GPU pools. In this architecture, prefill computes the prompt in parallel and materializes the key-value (KV) cache, while decode consumes that cache to emit output tokens one step at a time. The split exploits a structural asymmetry: prefill is compute-bound, decode is memory-bandwidth-bound. For that reason, PD disaggregation has become the standard architecture for modern LLM inference engines and the dominant architecture for large-scale LLM serving, because it reduces phase interference and enables independent scaling; at the same time, it makes KV-cache transfer, queue balance, routing policy, and service-level-objective (SLO) trade-offs central systems problems (Li et al., 9 Mar 2026, Liu et al., 1 Dec 2025).
1. Architectural model and serving semantics
In PD disaggregation, requests first enter a prefill-only stage. That stage processes the entire prompt, performs the full self-attention computation over the input context, and builds the KV cache. The resulting cache is then transferred to a decode-only stage, which performs autoregressive generation. A canonical formulation separates a compute-bound prompt pass from a memory-bandwidth-bound token loop; one representative description states that prefill processes the full prompt in parallel with attention and that decode then autoregressively generates output tokens, with the KV cache transferred between the two pools (Li et al., 9 Mar 2026).
The operational significance of the split is not merely functional decomposition. It changes the unit of resource management. Prefill resources are provisioned for prompt throughput and first-token latency; decode resources are provisioned for steady-state token streaming and KV-cache residency. This separation is reflected in the standard serving metrics. TTFT denotes time-to-first-token, TPOT denotes time-per-output-token, and goodput denotes the sustained rate of requests that jointly satisfy both SLOs per unit GPU time (Wang et al., 4 Aug 2025). In practice, PD disaggregation turns these metrics into stage-specific objectives rather than properties of a monolithic execution path.
The architecture also elevates the KV cache to a first-class systems object. In one concrete example, the network transfer can involve the entire KV cache over IP or NVLink, with an example size of approximately $512$ MB for $2$ K tokens on Llama-3-8B (Li et al., 9 Mar 2026). This coupling is fundamental: the prefill and decode stages are logically independent except for the cache handoff.
2. Phase asymmetry and the rationale for disaggregation
The principal rationale for PD disaggregation is the mismatch between the resource profiles of prefill and decode. Prefill is dominated by high-arithmetic-intensity matrix computation. Decode repeatedly loads model weights and the growing KV cache and is therefore primarily constrained by memory bandwidth. SPAD formalizes this contrast quantitatively: for BLOOM-176B at , the prefill arithmetic intensity is at least $2.5$ FLOP/byte, whereas for decode at , it is approximately $0.5$ FLOP/byte (Zhang et al., 9 Oct 2025). The same study reports that reducing a modeled H100’s $3.35$ TB/s HBM bandwidth by increases prefill latency by only , while halving GPU cores slows decode latency by approximately $512$0, directly illustrating that the two phases respond to different hardware bottlenecks (Zhang et al., 9 Oct 2025).
This asymmetry explains why monolithic execution causes interference. Co-locating the two phases on the same accelerator forces a compromise in batch shape, kernel mix, and memory allocation. PD disaggregation avoids that compromise by placing prefill on hardware optimized for FLOPS and decode on hardware optimized for memory bandwidth. It also permits the prefill-to-decode resource ratio to be tuned to workload composition rather than fixed by the capabilities of a single device class.
At the same time, disaggregation can strand complementary resources. Adrenaline reports that under PD disaggregation, prefill instances exhibit less than $512$1 HBM bandwidth utilization and less than $512$2 HBM capacity utilization, while decode instances exhibit less than $512$3 compute utilization (Liang et al., 26 Mar 2025). This establishes an important qualification: PD disaggregation isolates bottlenecks effectively, but isolation alone does not guarantee high overall utilization.
3. KV-cache transfer and the multi-turn problem
The classical PD design is most natural for single-turn requests. Under multi-turn conversations, however, two inefficiencies become dominant. First, turn $512$4 often triggers full prefill over the entire conversation history, including prior outputs and the new user prompt, even though most tokens are already “warm.” Second, each new turn requires another transfer of the full KV cache from prefill to decode, and that transfer grows with context length. One study reports that under realistic multi-turn loads such as ShareGPT, repeated KV transfers saturate network links, queuing delays spike, and service degrades or times out (Li et al., 9 Mar 2026).
A central refinement is append-prefill, in which only newly appended tokens are processed while cached KV states for earlier tokens are reused. The complexity difference is explicit. Full prefill over $512$5 tokens uses $512$6 attention, whereas append-prefill on $512$7 new tokens with $512$8 cached tokens uses $512$9; typically $2$0 (Li et al., 9 Mar 2026). The interference difference is equally explicit: microbenchmarks show that one full-prefill worker co-located with decode causes approximately $2$1 slowdown in TPOT, while one append-prefill worker co-located with decode causes approximately $2$2 slowdown (Li et al., 9 Mar 2026).
On that basis, PPD introduces dynamic routing for turn $2$3 requests. It benchmarks discretized workloads $2$4, measures
$2$5
then computes
$2$6
routing locally to decode via append-prefill when the score is positive (Li et al., 9 Mar 2026). The key claim is not that local append-prefill is always superior, but that no single fixed routing strategy satisfies all SLOs simultaneously. In extensive evaluation, PPD reduces turn $2$7 TTFT by $2$8 on average while maintaining competitive TPOT and reducing KV network traffic by up to approximately $2$9 (Li et al., 9 Mar 2026).
AMPD addresses the same interleaved prefill-decode pattern from a broader multi-round perspective. It combines an offline planner with an online coordinator that dynamically routes incremental prefill tasks either locally on the decode worker or remotely to prefill workers, then reorders a small head window of queued prefill jobs to maximize SLO attainment. In reported results, adaptive routing offloads 0 of incremental prefill to decoders, prefill reordering reduces TTFT tail by 1, and the combined design raises SLO attainment by 2 over naive PD disaggregation (He et al., 16 Feb 2026). A plausible implication is that multi-turn serving turns PD disaggregation from a static architecture into a continual routing problem.
4. Scheduling, routing, and resource allocation
Once prefill and decode are separated, the main systems question becomes how many resources each phase should receive and how requests should move between them. A compact analytical model expresses the resource counts needed to sustain a target token throughput 3: 4 with the implied ratio
5
In the same framework, the effective prefill throughput under a TTFT constraint is derived from an 6 model, while decode throughput is obtained empirically from the largest batch size satisfying the TPOT target (Li et al., 5 Mar 2026). This provides a formal method for converting SLOs and request lengths into P/D resource counts.
Static sizing is often insufficient under bursty or heterogeneous traffic. DOPD therefore monitors recent input length, output length, and request rate, forecasts the next interval with ARIMA, and dynamically adjusts the number of prefill and decode instances to maintain an optimal ratio. Reported results include up to 7 higher goodput, up to 8 lower P90 TTFT, and up to 9 lower P90 TPOT relative to representative aggregation- and disaggregation-based baselines (Liao et al., 26 Nov 2025). ARES addresses a different imbalance: drift within the decode pool caused by long-output requests. It predicts remaining generation length from the LLM’s hidden state with a $2.5$0-layer MLP and reschedules active decode tasks across GPUs; the predictor reduces MAE by $2.5$1, the rescheduler reduces P99 TPOT by $2.5$2, and the system achieves up to $2.5$3 times higher goodput (Wang et al., 15 Oct 2025).
The same orchestration problem appears in heterogeneous hardware settings. A multi-vendor P-D system can place prefill on one GPU vendor and decode on another, using a transmission module that flattens tensors before RDMA transfer and reconstructs them after transfer so that precision formats, memory layouts, and parallel-strategy granularities remain compatible (Chen et al., 22 Sep 2025). That work also formulates a joint search over data, tensor, pipeline, and expert parallelism degrees together with P/D instance counts to maximize end-to-end throughput at minimum cost (Chen et al., 22 Sep 2025).
A recurrent conclusion across these works is that PD disaggregation is not only a pipeline split. It is a control problem over queueing, handoff, migration, and ratio selection, with TTFT and TPOT acting as coupled but not identical objectives.
5. Hardware specialization and execution substrates
Because prefill and decode have structurally different bottlenecks, PD disaggregation has motivated phase-specialized hardware rather than only phase-specialized scheduling. SPAD replaces a homogeneous GPU fleet with specialized Prefill Chips and Decode Chips. The Prefill Chip uses four $2.5$4 systolic arrays, $2.5$5 GB of GDDR7 at $2.5$6 TB/s, and normalized hardware cost $2.5$7 H100; the Decode Chip uses four $2.5$8 systolic arrays, $2.5$9 GB of HBM3 at 0 TB/s, normalized hardware cost 1 H100, and TDP approximately 2 W (Zhang et al., 9 Oct 2025). In cluster-level simulations on production traces, SPAD achieves 3 lower hardware cost and 4 lower TDP than the best GPU-only disaggregated baselines while offering the same performance (Zhang et al., 9 Oct 2025).
PD disaggregation has also been adapted to reconfigurable and virtualized substrates. PD-Swap targets edge FPGAs by keeping the table-lookup ternary matrix multiplication and weight-buffering engines static while dynamically swapping between a compute-heavy token-parallel prefill attention engine and a bandwidth-optimized decode attention engine. The reconfigurable partition bitstream is approximately 5 MB and can be loaded in roughly 6 ms over PCAP; the system hides most of that cost by overlapping reconfiguration with remaining prefill computation, and reports up to 7 tokens/s decode throughput with 8 speedup over prior work (Zhang et al., 12 Dec 2025).
FlexNPU uses transparent user-space virtualization on Ascend NPUs to support dynamic PD co-location rather than fixed disaggregation. It interposes on AscendCL APIs, virtualizes NPU objects, and dispatches operations through per-device daemons, enabling phase-aware scheduling without modifying model code, frameworks, or drivers. Reported results show no measurable inference overhead relative to direct passthrough, throughput improvements of 9 and $0.5$0 over static PD disaggregation on a $0.5$1-card Ascend 910C deployment of DeepSeek-R1, and TTFT reductions of over $0.5$2 relative to static PD co-location on Qwen2.5-7B with nearly unchanged TPOT (Gu et al., 3 Jun 2026).
These systems show that PD disaggregation functions as an architectural interface between model execution and hardware design. The prefill/decode boundary can be realized through chip specialization, dynamic partial reconfiguration, or runtime virtualization, but in each case the basic phase asymmetry remains the organizing principle.
6. Extensions beyond dense single-turn text generation
PD disaggregation has been generalized in several directions. In retrieval-augmented serving, Trinity separates vector search from both prefill and decode, introducing a dedicated vector-search GPU pool. Prefill-stage and decode-stage servers issue RPCs to that pool, which uses continuous batching for ANN search and stage-aware scheduling that reserves capacity for prefill while draining decode with leftover slots. Reported gains include $0.5$3 vector-search throughput, $0.5$4 end-to-end RAG serving throughput, P95 TTFT reduction of $0.5$5, and P95 token-latency reduction of $0.5$6 (Liu et al., 1 Dec 2025). This suggests that once prefill and decode are separated, other heterogeneous stages that were previously entangled with inference can also be externalized.
In multimodal serving, EPD-Serve extends PD into Encode-Prefill-Decode disaggregation. It adds asynchronous feature prefetching between encode and prefill and hierarchical grouped KV-cache transmission between prefill and decode. Under high concurrency, the system improves end-to-end throughput by $0.5$7 compared to PD-disaggregated deployment while satisfying TTFT below $0.5$8 ms and TPOT below $0.5$9 ms (Bai et al., 5 Jan 2026). Here PD becomes a substructure inside a wider stage-level decomposition.
For mixture-of-experts models, decode routing must account not only for request count or queue length but also for expert locality. ELDR derives an expert signature from prefill expert activations, partitions signature space across decode workers with balanced K-means, and then performs locality-band routing online. On deployments of up to $3.35$0 GPUs, ELDR reduces median TPOT by $3.35$1 over the strongest of four load-balancing baselines while leaving model outputs unchanged (Choi et al., 1 Jul 2026). In this setting, prefill does more than produce KV states: it also exposes predictive information about future decode cost.
PD disaggregation has also been pushed beyond the single datacenter. Prefill-as-a-Service argues that hybrid-attention models reduce KV-cache size sufficiently that long-context prefill can be offloaded to remote compute-dense clusters over commodity Ethernet, provided offloading is selective and bandwidth-aware. In a case study with an internal $3.35$2T-parameter hybrid model, the resulting PrfaaS-augmented heterogeneous deployment achieves $3.35$3 and $3.35$4 higher serving throughput than homogeneous PD and naive heterogeneous baselines, respectively, while using only modest cross-datacenter bandwidth (Qin et al., 16 Apr 2026).
7. Performance limits, debates, and design space boundaries
A central debate concerns whether PD disaggregation is intrinsically superior to aggregation. TaiChi frames the issue explicitly: aggregation is optimal for tight TTFT and relaxed TPOT, disaggregation excels for strict TPOT and relaxed TTFT, and neither is optimal under balanced TTFT and TPOT SLOs (Wang et al., 4 Aug 2025). TaiChi therefore unifies both policies in a hybrid design with differentiated-capability GPU instances and per-request latency shifting; under balanced SLOs it improves goodput by up to $3.35$5 over state-of-the-art systems (Wang et al., 4 Aug 2025). This is not a rejection of PD disaggregation, but it directly contradicts the common simplification that disaggregation is universally best.
A second debate concerns energy efficiency. A systematic reevaluation of disaggregated serving across GPU-to-GPU, host-DRAM, and NVMe KV transfer paths finds that performance benefits are not guaranteed and depend on request load and transfer medium. The same study reports that stage-wise independent frequency scaling enabled by disaggregation does not lead to energy saving because disaggregated serving has inherently higher energy consumption (Li et al., 14 Nov 2025). This establishes that the latency advantages of PD do not automatically imply energy optimality.
A third boundary concerns how far disaggregation should go. Adrenaline argues that pure PD leaves complementary resources idle and improves throughput by disaggregating part of decode attention and offloading it to prefill instances, achieving $3.35$6 higher overall inference throughput compared to state-of-the-art systems (Liang et al., 26 Mar 2025). At the other end of the spectrum, attention-FFN disaggregation treats PD as only an intermediate level in a deeper hierarchy of decomposition. Under strict TTFT/TPOT SLOs on DeepSeek-V3.2, operator-level Attention-FFN Disaggregation sustains around $3.35$7k tokens/s of system throughput in regimes where non-AFD deployments are infeasible (Wu et al., 27 May 2026). This suggests that PD disaggregation is best understood as a stable but nonterminal point in a broader design space of stage and operator specialization.
Taken together, these results place PD disaggregation in a precise systems context. It remains the canonical architecture for separating the compute-bound and memory-bound phases of LLM inference, but its practical value depends on workload mix, conversational structure, hardware heterogeneity, routing sophistication, and the cost of KV movement. The modern literature therefore treats PD disaggregation less as a fixed recipe than as an extensible abstraction for balancing interference isolation against communication, utilization, and SLO satisfaction.