---
title: Disaggregated LLM Serving Infrastructure
url: https://www.emergentmind.com/topics/disaggregated-llm-serving-infrastructure
type: topic
---

# Disaggregated LLM Serving Infrastructure

Disaggregated LLM serving infrastructure partitions the major computational and memory-intensive phases of large language model (LLM) inference—such as prefill (prompt encoding) and decode (autoregressive token generation)—onto distinct machine pools, accelerators, and memory tiers. This architectural paradigm addresses the inherent resource heterogeneity in LLM workloads, unlocks high hardware utilization, and enables scalable, cost-effective deployment at production scale, as demonstrated by systems such as DistServe, HexGen-2, BanaServe, DéjàVu, Mooncake, MegaScale-Infer, and xDeepServe [2401.09670][2502.07903][2510.13223][2403.01876][2407.00079][2504.02263][2508.02520]. The design combines optimized communication strategies, adaptive scheduling, and fault-tolerant mechanisms to meet stringent service-level objectives (SLOs) for both latency and throughput over large, distributed, and often heterogeneous accelerator clusters.

## 1. Architectural Principles and Taxonomy

Disaggregation in LLM serving has four primary forms:

- **Phase (Prefill–Decode) Disaggregation:** Splits prompt computation (prefill) and step-wise decoding onto separate compute pools or device types, matching phase resource needs (compute-bound vs. memory-bandwidth-bound) [2401.09670], [2510.13223], [2502.07903].
- **Compute–Memory Disaggregation:** Offloads the large, growing key–value (KV) cache to external memory nodes (host DRAM, SSD, or remote RDMA-attached memory) while retaining compute on accelerators [2407.12391], [2407.00079].
- **Accelerator Pooling and Heterogeneous Deployments:** Schedules distinct workload phases onto specialized hardware, e.g., high-FLOPS GPUs for prefill and high-HBM-bandwidth GPUs for decode [2509.17542], [2502.07903].
- **Module/Operator Disaggregation (MoE/Attention):** Decouples transformer attention, feed-forward, and Mixture-of-Experts modules, allowing them to scale and run independently [2504.02263], [2508.02520].

These approaches are architected via a multi-tier serving stack. Typical layers include (a) a parameter store for model weights, (b) a cache tier for activations, (c) distributed compute engines (grouped by phase), and (d) high-speed network fabrics (NVLink, InfiniBand, RoCE) coordinating phase transitions and cache transfer.

## 2. Core Mechanisms and Dataflows

A canonical disaggregated serving workflow operates as follows [2401.09670][2502.07903]:

1. **Request Admission:** Scheduler receives a prompt, determines input (prefill) and expected output (token count).
2. **Prefill Execution:** Request assigned to prefill pool; full prompt executed in parallel with customized parallelism (tensor/model/pipeline), producing initial token(s) and the KV cache for all layers.
3. **KV-cache Transfer:** KV cache, typically tens to hundreds of MB, is serialized and transferred (over NVLink/RDMA/Ethernet) to the decode pool, either via collective or one-sided direct access (e.g., GPUDirect RDMA) [2501.14743][2407.00079].
4. **Decoding Execution:** Decode pool iteratively runs small-batch, memory-bound autoregressive steps, streaming output tokens. Continuous batching and fine-grained scheduling optimize for time per output token (TPOT) [2407.12391][2401.09670].
5. **Cache Management:** Disaggregated systems offload inactive or intermediate KV cache to secondary tiers (CPU RAM, SSD) [2407.00079][2510.13223].

For module-level disaggregation or MoE, further splitting is applied at the transformer layer, decoupling attention and feed-forward paths, which are realized across heterogeneous pools with tailored communication (e.g., ping-pong pipelining, expert-parallel routing) [2504.02263][2508.02520].

## 3. Scheduling, Load Balancing, and Autoscaling

Optimal efficiency in disaggregated settings requires fine-grained scheduling, continuous adaptation, and hardware-aware resource allocation:

- **Static and Adaptive Partitioning:** Early systems (DistServe, vLLM-disaggregated) statically partitioned GPU pools for prefill and decode. Adaptive schedulers (Arrow, HeteroScale) react to real-time metrics (queue length, utilization, SLO attainment) to elastically allocate instances to roles and automatically rebalance under bursts or workload skew [2505.11916][2508.19559].
  
- **Mathematical Models:** Placement is formalized as a mixed-integer linear program (MILP) or as network flow optimization, maximizing throughput subject to GPU and network constraints:
  $$
  \max_{A,t,s,f} \sum_{g, h} f_{g\to h}
  $$
  Subject to per-GPU and per-link bandwidth/memory limits, and phase-specific SLOs [2502.07903].

- **Cluster-Wide Scaling:** Coordinated autoscaling (HeteroScale) uses a primary metric (e.g., decode TPS) for proportional control of joint prefill and decode pools, aligned to network topology to minimize data transfer bottlenecks and keep P/D ratios optimal [2508.19559].

- **Module-level Load Balancing:** For MoE or attention disaggregation, algorithms such as expert load balancing, ping-pong pipeline parallelism, or attention-head migration are used to allocate resources dynamically, reduce stragglers, and saturate device throughput [2504.02263][2510.13223][2508.02520].

## 4. Communication and Fault Tolerance

Disaggregation by definition introduces network overhead and necessitates optimized communication primitives:

- **High-Performance RDMA and Direct Memory Access:** Custom communication layers (e.g., MegaScale-Infer's M2N, xDeepServe's XCCL, DéjàVuLib) eliminate GPU–CPU copies and expose tensor-centric, point-to-point and all-to-all primitives, realizing up to 4.2× throughput vs. NCCL for small messages and stable tail latencies as system scale grows [2504.02263][2508.02520][2403.01876].

- **KV Cache Strategies:** Pull-based transfer (KVDirect) removes decode-side idling and reduces total latency compared to push mode; CPU/SSD-backed global KV stores enable cache load-insensitive routing and dynamic prefix reuse [2501.14743][2510.13223][2407.00079].

- **Microbatch Swapping and Memory Tiering:** KV cache for idle microbatches can be asynchronously swapped to host memory (as in DéjàVu), significantly reducing GPU RAM requirements (e.g., 2× RAM reduction, 1.8× batch size increase) [2403.01876][2407.00079].

- **Fault Tolerance:** State replication (per-microbatch and per-token) between neighbor workers allows sub-second recovery from device failures, with recovery time $T_{\mathrm{recover}} = T_{\mathrm{fetch\_replica}} + T_{\mathrm{replay}}$, achieving 1.24× slowdown vs. 1.89× in non-replicated systems [2403.01876].

## 5. Heterogeneous and Modular Deployment

Disaggregated LLM serving unlocks previously unattainable heterogeneity and modularity in large clusters:

- **Heterogeneous Accelerator Support:** Workload phases are scheduled to the hardware tier (GPU/CPU/NPU) best matched to their resource profile; e.g., high-TFLOPS devices for prefill, high-HBM for decode, and optimal cost-performance ratios via joint search over parallelism settings and instance counts [2509.17542][2502.07903][2508.02520].

- **Cross-vendor and Multi-generation Compatibility:** Systems such as HexGen-2 and P-D (multi-vendor) remove GPU-vendor lock-in by flattening/retile tensor formats and accommodating different parallelisms in KV cache alignment and control [2502.07903][2509.17542].

- **Unified, Programmable APIs:** LLM microserving exposes sub-request-level REST endpoints (prep_recv, remote_send, start_generate) and a programmable router, unifying data- and model-parallel, phase-disaggregated, and prefix-migration patterns in a consistent interface [2412.12488].

- **Autoscaling and Fine-Grained Elasticity:** Stateless instance pools (Arrow) and partially disaggregated rolling activation (EcoServe) permit efficient, SLO-aware scaling of capacity with superlinear gains in request serving rates over static architectures [2505.11916][2504.18154].

## 6. Empirical Performance and Limitations

Disaggregated serving infrastructures consistently report large improvements in utilization, throughput, and SLO compliance:

- **Throughput Gains:** Up to 2× overall throughput and order-of-magnitude increases in strict SLO rates are observed across OPT, Llama, and Mixture-of-Expert models [2403.01876][2401.09670][2504.02263][2510.13223].
- **Latency Reductions:** P99 time-to-first-token (TTFT) and time-per-output-token (TPOT) reduce by 20–80% in production microbenchmarks, even under high-load and long-context scenarios [2501.14743][2510.13223][2407.00079].
- **Cost Savings:** Heterogeneous and modular partitioning reduce cost per token by 30–40%, and coordinated scaling saves hundreds of thousands of GPU-hours daily at cloud scale [2502.07903][2508.19559][2509.17542].
- **Resource Utilization:** Adaptive scheduling and balanced parallelism increase GPU utilization from 60% (static colocated) to 85% (fully adaptive disaggregated) [2505.11916][2501.14743].

Limitations persist in the need for very high-bandwidth networks to avoid cache-transfer bottlenecks in fully disaggregated settings (minimum 40+ GB/s per node recommended for SLO compliance) [2504.18154][2401.09670], and in the increased system complexity and interdependence introduced by modular, cross-pool policies. Systems such as EcoServe and semi-PD propose compromise between full and partial disaggregation to better accommodate commodity settings or cluster-level storage constraints [2504.18154][2504.19867].

## 7. Synthesis and Forward Directions

Disaggregated LLM serving infrastructures are now considered foundational to large-scale, high-throughput, and cost-sensitive generative AI deployment. They enable independent scaling of phase- or operator-specific compute, memory, and interconnect resources, transparent support for multivendor and multi-generational hardware, and robust, SLO-aware cluster operation. Key open areas for future development include:

- Predictive, ML-based workload-driven phase and module scheduling [2510.13223][2502.07903].
- WAN–optimized KV cache synchronization for geo-distributed and hybrid-cloud deployments [2510.13223].
- Integration of adaptive compression, quantization, and storage tiering in cache-centric architectures [2407.12391][2407.00079].
- Hierarchical scheduling and intelligent placement for nested disaggregation (e.g., intra-phase, expert routing) [2508.02520][2504.02263].
- Robust, zero-downtime upgrades and fault-tolerance through further modularization and stateless orchestration [2505.11916][2510.13223][2403.01876].

With an expanding ecosystem of production-grade frameworks and research systems, and as both model and infrastructure complexity scale, disaggregation is poised to remain the dominant pattern for next-generation LLM inference services.

Source: https://www.emergentmind.com/topics/disaggregated-llm-serving-infrastructure