---
title: 'DistServe: Optimized Disaggregated LLM Serving'
url: https://www.emergentmind.com/topics/distserve
type: topic
---

# DistServe: Optimized Disaggregated LLM Serving

DistServe is an architectural framework for large language model (LLM) serving that explicitly disaggregates the prefill (prompt encoding) and decoding (autoregressive generation) stages across distinct GPU instances, enabling tightly optimized resource allocation, parallelism configuration, and latency SLO (service level objective) compliance. DistServe departs from conventional LLM serving systems that colocate prefill and decoding on the same resources and batch both phases together, addressing major interference and coupling limitations inherent in monolithic serving architectures [2401.09670].

## 1. Architectural Rationale: Disaggregation of Prefill and Decoding

Modern auto-regressive LLM inference consists of a compute-heavy prefill phase, which encodes the user's entire prompt in one Transformer pass to emit the initial output token, followed by an iterative decoding phase that generates each subsequent token autoregressively with an evolving KV-cache. Conventional architectures batch both phases on shared GPUs, aiming for maximum aggregate throughput; however, this policy introduces two fundamental drawbacks:

- **Prefill–decoding interference**: Large prompt batches (prefill) delay decoding requests, inflating time-per-output-token (TPOT), while queued decoding slows first-token emission (TTFT).
- **Resource coupling**: Shared allocation binds parallelism and GPU usage, though prefill and decoding exhibit disparate compute, memory, and SLO requirements.

DistServe segregates prefill and decoding onto dedicated GPU instances, eliminating phase interference. This permits phase-tailored selection of parallelism degrees (tensor/pipeline splits), and dynamic per-phase allocation to maximize per-GPU “goodput”—the sustainable request rate meeting both TTFT and TPOT SLOs [2401.09670].

## 2. Latency Objectives and Goodput Formulation

LLM application responsiveness is measured by two phase-specific latency metrics:

- **Time to First Token (TTFT)**: Wall-clock interval from request arrival to first output token emission.
- **Time Per Output Token (TPOT)**: Mean wall-clock latency per subsequent token.

Given target SLOs $\tau_1$ (TTFT), $\tau_2$ (TPOT), and an attainment goal $\alpha$ (fraction of requests compliant), DistServe maximizes sustainable request rate $R_{\text{req}}$ per GPU such that $\mathbb{P}(\text{TTFT} \leq \tau_1 \land \text{TPOT} \leq \tau_2) \geq \alpha$, formulated as:

$$
G(K_p, c_p, K_d, c_d) = \frac{\min\{g_p(K_p, c_p), g_d(K_d, c_d)\}}{K_p + K_d}
$$

Subject to:

- $K_p+K_d \leq N_{\text{max}}$ (GPU budget)
- Memory constraints per configuration
- $g_p$ and $g_d$ determined via phase-specific discrete event simulation under actual workload and SLO conditions

This optimization is solved by enumerating feasible parallelism, simulating phase throughputs, and selecting the allocation that maximizes goodput [2401.09670].

## 3. Placement, Parallelism Strategy, and Bandwidth Considerations

DistServe employs a two-phase placement and parallelism search:

- **Phase 1:** Enumerate all valid combinations of parallelism (tensor/pipeline degrees, GPU count) within memory limits for prefill and decoding, simulating per-batch latency using M/D/1 queue models.
- **Phase 2:** For clusters with high bandwidth (e.g., Infiniband $>200$ Gbps), arbitrary placement suffices. For low-affinity clusters (e.g., intra-node NVLINK), pipeline stages of prefill/decoding are co-located on the same node to ensure KV-cache traffic uses NVLINK ($\sim600$ GB/s), minimizing communication overhead.

The algorithm executes in $O(N \cdot M^2)$ time, enabling rapid replanning on commodity CPUs [2401.09670]. KV-cache transfer is shown to be negligible ($<0.1\%$ latency, $<30$ ms for $95\%$ of requests) when using NVLINK-aware placement.

## 4. Performance Evaluation and Comparative Metrics

DistServe was evaluated on a 32×A100-80GB, 4×8-GPU node platform, across diverse workloads:

| Workload                 | TTFT SLO ($\tau_1$) | TPOT SLO ($\tau_2$)   | SLO Attainment (%)  | Per-GPU Request Rate         |
|--------------------------|---------------------|-----------------------|---------------------|------------------------------|
| Chatbot (13B–175B)       | 0.2–0.4 s           | 0.1–0.2 s             | >90                 | Up to 4.48× state-of-art vLLM|
| Code completion (66B)    | 0.125 s             | 0.2 s                 | >90                 |                              |
| Long-text summarization  | 15 s                | 0.15 s                | >90                 |                              |

Notable quantitative improvements over monolithic systems (vLLM):

- Up to $4.48\times$ higher per-GPU sustainable request rate under identical SLO constraints
- Up to $10.2\times$ tighter SLO compliance (lower $\tau_1$, $\tau_2$ at $90\%$ attainment)
- KV-cache transfer latency $<0.1\%$ total, latency breakdown $<30$ ms for $95\%$ requests

## 5. Extensions, Tradeoffs, and Related Systems

Disaggregation in DistServe introduces KV-cache communication overhead, making sufficient intra-/inter-node bandwidth necessary. The system uses FCFS scheduling; more advanced preemptive or priority algorithms could further reduce tail latency.

Limitations:

- Lack of built-in fault-tolerance (e.g., decoding instance failure can block multiple prefill phases)
- Assumes stable workload; simulation fidelity reduces with highly variable input

Potential future directions include preemptive scheduling, replica-level fault recovery, adaptation to heterogeneous GPU clusters, and online auto-replanning to accommodate workload shifts [2401.09670].

DistServe’s architectural principles influenced subsequent systems:

- **P/D-Serve** [2408.08147] optimized grouping by scenario, dynamic P/D ratio adjustment, and high-throughput block-free KVCache transfer over RDMA, achieving 6.7× throughput versus aggregated serving and robust large-scale production deployment.
- **TokenScale** [2512.03416] resolved burst backpressure by introducing proactive token-level autoscaling (Token Velocity), and convertible decoders for rapid elasticity, with SLO attainment increases and cost advantage over DistServe, BlitzScale, and AIBrix.

## 6. Load Balancing, Distributed Dispatching, and Theoretical Foundations

DistServe’s design reflects broader principles of distributed load balancing under partial information, as studied in the parallel server model [2008.00793]. Naive schemes can degrade under multi-dispatcher settings due to herding and incast effects. The Tidal Water Filling (TWF) approach coordinates dispatch probabilities based on collective water-level minimization, yielding lower mean response time and thinner latency tails than classical JSQ(d), JIQ, and LSQ policies.

A “DistServe” load balancer, by aggregating partial queue-length information and employing stochastic water-filling, approaches centralized optimality while retaining strong stability and throughput guarantees, demonstrating that informed distributed algorithms can outperform both information-poor and naive policies in large-scale LLM serving [2008.00793].

## 7. Significance and Implications

DistServe establishes the viability and necessity of phase-disaggregated LLM serving to decouple latency objectives, eliminate phase interference, and rigorously optimize resource utilization. Its simulator-driven placement and explicit goodput maximization enable substantial improvements in strict SLO environments. The framework’s influence is evident in subsequent large-scale production deployments, as well as in theoretical and practical advancements in load balancing and autoscaling for disaggregated AI infrastructure.

Source: https://www.emergentmind.com/topics/distserve