---
title: 'PDP: Partially Disaggregated Prefill'
url: https://www.emergentmind.com/topics/partially-disaggregated-prefill-pdp
type: topic
---

# PDP: Partially Disaggregated Prefill

Partially Disaggregated Prefill (PDP) is an architectural and algorithmic paradigm for large language model (LLM) inference and serving that lies at the midpoint between fully unified and fully disaggregated systems. PDP aims to isolate and optimize the prefill (or prompt encoding) and decode (autoregressive token generation) stages, exploiting the distinct computational and memory-bound characteristics of each phase, while minimizing inter-phase resource interference, data movement overheads, and inefficiencies in storage and scheduling. PDP is realized in various serving frameworks, hardware architectures, scheduling algorithms, and system deployments, demonstrating substantial gains in throughput, latency, hardware utilization, cost-effectiveness, and flexibility across homogeneous and heterogeneous compute clusters.

## 1. Definition and Core Principles

PDP refers to the selective disaggregation of the prefill phase from the decode phase in LLM inference, partitioning computation, memory, or data-movement resources to ensure phase isolation, while often unifying storage or minimizing costly inter-GPU transfer.

- In fully unified systems, both prefill and decode share GPU compute and GPU memory for weights and KV-cache, which leads to interference between TTFT (time-to-first-token) and TPOT (time-per-output-token).
- Fully disaggregated systems run prefill and decode on separate physical devices, eliminating interference but causing weight replication, storage imbalance, expensive KV cache transfers, and high-overhead resource partitioning.
- PDP adopts intermediate strategies:
  - Semi-PD [2504.19867]: Disaggregate compute resources at the streaming multiprocessor (SM) level, binding SM partitions to prefill/decode workers via NVIDIA MPS, but unify storage (model weights and KV cache) in a single CUDA address space, thereby eliminating KV cache transfer and storage waste.
  - Multi-vendor PDP [2509.17542]: Prefill and decode are executed on heterogeneous GPUs (high FLOPS vs. high memory bandwidth) with a compatibility module masking differences and a parallel-strategy joint optimization for deployment.
  - Instance-level or temporal PDP [2504.18154]: Time-slice instances into long prefill epochs and decode epochs, scheduled cyclically with rolling activation or macro-instance coordination to ensure continuous TTFT availability.
  - Chunked and intra-request PDP [2509.24381]: Only encoder and/or prefill phases are disaggregated; decode remains local. Chunking and embedding trackers maximize inter- and intra-request parallelism.
  - Partitioned PDP [2509.17357]: Prefill distributed across idle/low-end cluster GPUs for heavy FLOPs, decode centralized on high-end GPUs with minimal KV communication.
  - Dispatcher-level PDP [2401.09670, 2408.08147]: Fine-grained prefill worker pools with adaptive P/D ratios per scenario, optimized RDMA KVCache transfer, on-demand scheduling, and multi-tier cache residency.

## 2. System Architectures and Implementation Strategies

PDP patterns are instantiated in diverse system architectures:

| System        | Computation Disaggregation     | Storage Management                    | Scheduling/Control        |
|---------------|-------------------------------|---------------------------------------|--------------------------|
| semi-PD       | SM-level partition (CUDA MPS) | Unified CUDA space; no KV transfer    | Dynamic SM controller    |
| EcoServe      | Time-epoch slicing in instance| Single instance holds both KV/weights | Rolling activation, macro|
| Cronus        | Prefill distributed per token | KV gathered to decode pool GPU        | Per-request proportional |
| DistServe     | Prefill/Decode per-device     | Bandwidth-aware placement             | Parallelism optimized    |
| P/D-Serve     | xPU groups per phase/scenario | Bulk RDMA, scenario-wise KV caching   | Dynamic P/D ratio, on-pull  |
| DynaServe     | Request split at optimal s_r  | Flexible, adaptive HBM allocation     | Two-level (global/local) |
| Mooncake      | Prefill/Decode clusters, tiered KVCache  | Layer-wise streaming, DRAM/SSD | Early rejection, prediction|
| HydraInfer    | ED+P split on multimodal      | Overlap via CUDA IPC/NCCL             | Stage-level batch        |
| TPLA          | Aggregated prefill, TP decode | Orthogonal transforms for TP slicing  | MLA/TPLA scheduling      |
| SPAD          | Prefill Chips for partial offload| Fractional layer (α) offloading      | Hardware provisioning    |


Context:
- On homogeneous clusters, PDP often binds SM or device partitions, or time slices, so both phases have non-interfering compute windows but do not shuffle KV cache or model weights.
- On heterogeneous clusters, PDP exploits the strengths of device classes; e.g., prefill runs on high-FLOPS chips even if VRAM is modest, decode on bandwidth-heavy chips.
- On NPU clusters, PDP organizes per-scenario P/D groups, mapping RDMA RoCE IP tuples to minimize bottlenecks and enable dynamic ratio adjustment [2408.08147].

Pseudocode examples and algorithms are given for SM-partition switching [2504.19867], rolling activation [2504.18154], proportional chunk assignment [2509.17357], and dynamic P/D ratio adjustment [2408.08147]. Algorithms typically employ measurement-driven dynamic adjustment (windowed SLO violation detection, checkpointing, rolling performance metrics).

## 3. Optimization Objectives and Analytical Models

PDP systems formalize their objectives using throughput, latency, SLO (Service Level Objective) attainment, and resource/cost constraints.

Typical objective functions (examples shown for semi-PD and multi-vendor PDP):

- Maximize request throughput $R'$ s.t.
  \[
  \begin{aligned}
    TTFT_x(R') &\leq S_p \\
    TPOT_y(R') &\leq S_d \\
    x + y &= 100 \\
    x, y &\geq 0
  \end{aligned}
  \]
Where (x, y) is the SM (or instance or device) split.

- Joint optimization over parallel degrees (dp, tp, pp, ep), instance counts, subject to prompt encoding and decoding latency and memory SLOs:
  \[
    \max_{dp_p, tp_p, pp_p, ep_p}
    T_p(...)
    \;\text{s.t.}\;
    l_p(...) \leq L_{ttft},\;
    m_p(...) \leq M_p
  \]
  (same for decode).

Latency and throughput are modeled using empirical or regression fits to token counts, SM fractions, arrival rates ($\lambda$), per-device FLOPS ($C_i$), and memory bandwidths. Models account for communication time for KV transfer ($T_{comm} = \mathrm{KV}_\mathrm{size}/B_\mathrm{interconnect}$), and for pipelined overlap regions ($\Delta$).

## 4. Scheduling, Batching, and Load Balancing

PDP deployments leverage scheduler designs that optimize TTFT, TPOT, and goodput under real workload variability:

- SLO-aware dynamic partitioning [2504.19867]: SM allocation (x, y) updated every window W, delaying switches but never draining state.
- Macro-instance coordination, rolling activation (EcoServe [2504.18154]): ensure at least one instance is prefill-ready, bound TTFT by $\Delta = (\tau_p+\tau_d)/N$.
- Chunked pipeline parallelism (Mooncake [2407.00079], RServe [2509.24381]): Input chunking, token budgets, and embedding tracker overlap streams encoding, prefill, and decode for both intra- and inter-request optimization.
- Candidate migration / rescheduling (ARES [2510.13668]): Future token loads predicted via hidden state-driven MLP, requests migrated to minimize token-load variance.
- Stage-level batching in multimodal PDP (HydraInfer [2505.12658]): Separate batch sizes for encode, prefill, decode, scheduled per SLO and hardware profile.

Correctness and load balancing are validated via metrics such as SLO attainment ($P_{90}$, $P_{99}$ latencies), migration cost modelling, and load variance reduction.

## 5. Communication and Storage Efficiency

PDP approaches are characterized by:

- Unified storage with no inter-process KV cache transfer (semi-PD, EcoServe).
- Bulk contiguous-buffer RDMA transfer (P/D-Serve [2408.08147]), minimizing transfer time by aligning and batching PageAttention KV blocks.
- Tiered KVCache pools (Mooncake [2407.00079]): VRAM for active blocks, DRAM for warm, SSD for cold with block-level deduplication.
- Transmission-module compatibility (multi-vendor [2509.17542]): flattening tensors to 1D for heterogeneous GPU transfer, aligning parallel strategies.

Quantitative improvements include 0.1% KV transfer time on OPT-175B (DistServe [2401.09670]), up to 46% reduction in D2D KV transfer time (P/D-Serve), and elimination of "second-token penalty" due to standing unified memory maps.

## 6. Performance Evaluation and Comparative Analysis

Extensive benchmarks confirm the benefits of PDP:

- semi-PD [2504.19867]: End-to-end latency per request reduced by 1.27–2.58× (DeepSeek), throughput increased by 1.55–1.72× (Llama).
- EcoServe [2504.18154]: Goodput up to 127% higher than baseline systems on LLMs of 30B, 70B, 32×L20 cluster with commodity Ethernet.
- Cronus [2509.17357]: TTFT99 reduced by 35%, TBT99 improved by 20%, aggregate throughput up 1.7× on heterogeneous clusters.
- DistServe [2401.09670]: Up to 4.48× higher per-GPU goodput under SLO, 10.2× tighter TPOT SLOs, low KV transfer cost.
- P/D-Serve [2408.08147]: End-to-end throughput up 60%, TTFT SLO success rate up 42%, aggregate throughput up 6.7×.
- DynaServe [2504.09285]: Goodput improvements of up to 4.34×, balancing HBM memory utilization by 49%, serving capacity up to 3.07× at 100 ms ITL.
- HydraInfer [2505.12658]: Throughput (>4× over vLLM), TPOT90 reduced from 0.065 s (mono) to 0.038 s (PDP).
- Mooncake [2407.00079]: Throughput gains scale superlinearly with context length, topping 525% at 128k tokens compared to vLLM.
- SPAD hardware [2510.08544]: Prefill throughput +8% at 52% lower cost, decode 97% performance at 28% lower TDP, cluster savings 19–41% TCO.

A plausible implication is that PDP paradigms generally outperform both colocated and fully disaggregated models in realistic, variable workloads, especially as model and context sizes scale.

## 7. Limitations, Controversies, and Future Directions

Limitations identified in the literature include:

- PDP scheduling and tuning may incur overhead if parallel degree/global search spaces become large; multi-vendor PDP's search is tractable due to discrete space size [2509.17542].
- Communication bottlenecks persist in some designs, notably at the prefill-decode boundary (KV transfer), with ongoing proposals for layer-wise pipelined RDMA [2509.17542].
- Prediction-driven migration (ARES) depends on sufficient hidden state fidelity and is sensitive to batch frequency and migration cost.
- PDP architectures deployed at massive scale (e.g., P/D-Serve across tens of thousands of NPUs) rely on bulk RDMA performance and scenario-aware grouping; model and vendor diversity remains a subject of extended experimentation.

Controversially, the precise balance between compute and memory offload (e.g., the fraction α in SPAD [2510.08544]) governs cost and efficiency, with future work suggested to optimize this balance under dynamic workload shifts and hardware heterogeneity.

Future research directions focus on refining PDP's:
- Cost minimization algorithms (explicit cost terms in objective functions).
- By-layer pipelining to further overlap compute and communication.
- Integration with novel attention mechanisms (e.g., PDP separation in TPLA [2508.15881]).
- Adaptation to domestic hardware ecosystems and new NPU/GPU designs.
- Scaling to ultra-large model deployments and online adjustment under bursty, mixed workloads.

PDP thus represents an important and evolving class of design choices that enable LLM serving systems to match the rapidly diversifying technical landscape and workload demands.

Source: https://www.emergentmind.com/topics/partially-disaggregated-prefill-pdp