---
title: Disaggregated Serving Systems
url: https://www.emergentmind.com/topics/disaggregated-serving-system
type: topic
---

# Disaggregated Serving Systems

A disaggregated serving system is an architectural paradigm in which heterogeneous compute, memory, and other resources are logically and often physically separated, then orchestrated together using highly optimized scheduling and data movement mechanisms. The principal motivation is to address resource skew, phase interference, and inflexibility inherent in monolithic and tightly coupled systems—allowing independent scaling, improved resource utilization, and strict adherence to service-level objectives (SLOs) for complex, high-throughput workloads such as large language model (LLM) inference, recommendation serving, and multimodal pipelines. Modern disaggregated serving systems underpin production-scale AI, cloud, and telecom infrastructures, employing dynamic orchestration, rack-scale shared memory, profile-based simulation, and fine-grained migration mechanisms.

## 1. Foundational Concepts and Motivation

Disaggregated serving systems decompose monolithic server designs into distinct resource pools—compute nodes (CPUs, GPUs, NPUs), memory nodes (DRAM, CXL-attached memory), storage nodes, and network components—connected by high-speed interconnects or fabrics. Inference pipelines are further decomposed along phase or operator boundaries (e.g., prefill, decoding, and expert stages for LLMs). This decoupling allows each resource type to be managed, scheduled, and scaled independently to match workload demands and hardware heterogeneity [2212.00939], [2401.09670], [2602.23036], [2512.18194].

In the context of transformer-based LLM serving, disaggregated deployment most commonly refers to the separation of the prefill phase—which is highly compute-bound, as it requires a full forward pass over a long prompt—and the decode phase—which is memory-bandwidth bound, as it involves autoregressive generation using a progressively growing key/value (KV) cache. Disaggregated serving eliminates the negative interference between these phases previously observed in colocated systems, where prefill jobs would monopolize compute, starving latency-sensitive decoding and causing tail SLO violations [2401.09670].

Disaggregation extends beyond LLMs. In large-scale recommendation systems, separation of compute engines and high-capacity memory nodes for embedding table sharding produces significant Total Cost of Ownership (TCO) reductions, resource pooling flexibility, and robust failure isolation [2212.00939]. In modern datacenters and edge/fog deployments, resource boards for computation, memory, storage, and network are individually manageable, with composable allocation and fine-grained maintenance [2104.04060], [1904.01311].

## 2. Key Architectural Patterns

Across domains, several recurring patterns characterize disaggregated serving systems:

**a. Pooling and Composable Resource Fabrics:**  
Resources of each type (compute, memory, storage) are provisioned in pools and aggregated through software-defined control planes or hardware fabrics (e.g., PCIe, InfiniBand, CXL). Work is mapped to components based on available capacity, real-time load, and data dependencies. Pools may be physically dispersed but present a unified logical namespace [2512.18194], [2212.00939].

**b. Phase-Oriented Separation (e.g., Prefill–Decode):**  
LLM serving disaggregates the prefill and decode phases, either at the server level (distinct GPU workers), inside a single device via SM or CU masking (e.g., semi-PD [2504.19867], RAPID-Serve [2601.11822]), or temporally (EcoServe's epochs [2504.18154]). Decode workers operate on precomputed KV caches, achieving high memory-bandwidth efficiency and batch utilization without compute-heavy interference from prefill jobs.

**c. Layer/Operator Disaggregation:**  
Full “Transformerless” architectures (e.g., xDeepServe [2508.02520]) go further by splitting attention, feedforward, and MoE operator execution into modular, schedulable subgraphs across distributed NPUs, with dedicated communication libraries (XCCL) and inter-module load-balancing and fault isolation.

**d. Disaggregated Caching and Data Management:**  
KV caches and embedding tables can reside in rack- or cluster-wide shared memory, host DRAM/SSD, or pooled memory nodes, with direct GPU or CPU access mediated via custom memory management, two-tier synchronization, and prefix-aware indexing (e.g., TraCT [2512.18194], Mooncake [2407.00079], BanaServe [2510.13223]).

**e. Dynamic Orchestration and Scheduling:**  
Router and scheduler logic must coordinate resource allocation dynamically, often via load-aware, SLO-driven, or prefix-contention-agnostic policies. Techniques include global KV store with overlapped transmission, per-task micro-request splitting, proactive phase rolling, bottleneck-based batch composition, and online/offline planning [2510.13223], [2504.09285], [2511.21862], [2504.18154].

## 3. Operational Mechanisms: Synchronization, Scheduling, and Data Movement

Disaggregated serving requires explicit orchestration of inter-phase transitions, data movement, and fine-grained resource balance:

**a. Synchronization and Consistency:**  
Modification of global or distributed caches introduces metadata races and visibility issues. Systems such as TraCT address non-coherent CXL shared memory by introducing two-tier (local/global) lock hierarchies: mutexes on each node, and a lock manager thread that arbitrates access to shared lock slots in memory. All metadata updates are followed by explicit cacheline flushes to guarantee coherence semantics for other nodes, while large KV transfers exploit DMA engine semantics for direct, cache-bypassed data movement [2512.18194].

**b. Cache and State Management:**  
KV caches are indexed by prefix hashes in fixed-size, cache-aligned hash tables or managed via LRU lists with reference counting. Dynamic eviction, layer-wise overlapping, or hot-spot replication (as in Mooncake) are employed to optimize memory capacity and minimize stale fetches. Global KV stores (e.g., BanaServe) decouple compute allocation from state, enabling routers to perform load-aware scheduling independent of local cache content [2510.13223], [2407.00079].

**c. Adaptive Routing and Load Balancing:**  
AMDP [2602.14516] and related frameworks use real-time statistics (TTFT, ITL) at both prefill and decode workers to adaptively route incremental workloads, optimizing for SLO fulfillment. Adaptive reordering of prefill queues maximizes SLO rate by permuting tasks within a look-ahead window. Macro-schedulers at the instance or cluster level orchestrate rolling phase transitions (EcoServe [2504.18154]), elastically adjusting capacity through mitosis-like splitting and merging.

**d. Data Movement Acceleration:**  
KV transfers, traditionally over RDMA with host-NIC-DRAM hops, become the dominant performance bottleneck as model and context sizes scale. TraCT eliminates this by leveraging CXL shared memory for direct, single-hop load/store and GPU DMA between devices (640 ns per hop, ~10 GB/s), yielding 4.2× speedup for 1 MB blocks and a 9.8× reduction in TTFT over RDMA baselines [2512.18194]. P/D-Serve [2408.08147] implements block-free, contiguous-buffer RDMA with per-scenario dynamic prefill/decode ratio tuning to further reduce bandwidth and latency overhead.

## 4. Performance Modeling, Evaluation, and Trade-Offs

Performance analysis in disaggregated serving systems rigorously models compute, memory, and network components, and is tightly linked to workload-aware scheduling:

**a. Latency and Throughput Models:**  
For phase-wise disaggregation (DistServe [2401.09670], semi-PD [2504.19867]), TTFT and TPOT are modeled as queueing processes (e.g., M/D/1, M/M/1), decomposing request latency into queuing delay, compute, and network transfer components:
\[
TTFT = T_{\mathrm{prefill}} + T_{KV\,transfer} + T_{\mathrm{decode}}
\]
Prefix-aware caching and fast, in-fabric KV transfers deliver transformative reductions in TTFT (up to 9.8×) and P99 latency (up to 6.2×) [2512.18194].

**b. Resource Utilization and Efficiency:**  
Fine-grained disaggregation enables independent scaling and utilization maximization. BanaServe [2510.13223] achieves both compute and memory utilization within 5 percentage points of 95%, maintaining linear throughput scaling as load increases, in contrast to static disaggregated and monolithic designs that saturate early or waste resources.

**c. Trade-Offs:**  
Disaggregation introduces overheads (e.g., KV cache transfers, synchronization costs, increased average hop counts and marginal power/network traffic [1904.01311]) that must be balanced against gains in SLO adherence, concurrency, and cost savings. Notably, design choices such as block size in RDMA, layer-wise versus full-KV transfer, CXL capacity limits, and local cache pinning each influence aggregate system efficiency.

## 5. Extensions: Multimodal, Multi-Model, and Heterogeneous Serving

Recent advances generalize disaggregated serving across modality, workload, and hardware boundaries:

**a. Multimodal and Stage-Level Disaggregation:**  
vLLM-Omni [2602.02204] and EPD-Serve [2601.11590] present “stage abstraction” and three-way “EPD” decomposition (Encode, Prefill, Decode), enabling construction of any-to-any pipelines for complex multimodal models. Each stage functions as an independent execution engine with individualized resource allocation, batching, and data connectors. Stage-level disaggregation supports flexible deployment, horizontal and vertical scaling, and optimized SLO management for mixed text, audio, and vision workloads.

**b. Multi-Model and Multi-Agent Sharing:**  
PrefillShare [2602.12029] factorizes the prefill module as a frozen, shared component, allowing multiple specialized decode modules to reuse a computed KV cache for identical prompt prefixes. This eliminates redundant KV computation and memory allocation across models, reducing p95 tail latency by up to 4.5× and enhancing throughput and hit ratios in multi-agent serving.

**c. Heterogeneous Hardware and Simulation:**  
LLMServingSim 2.0 [2602.23036] provides a unified simulation environment to jointly analyze disaggregated serving and hardware heterogeneity (e.g., GPUs, NPUs, DRAM, CXL, PIM). The simulator models runtime scheduling, data movement, network contention, and power, achieving sub-1% error vs. real deployments—a tool for co-designing next-generation infrastructure.

## 6. Limitations, Challenges, and Future Directions

Despite its benefits, disaggregated serving poses significant challenges:

- **Shared memory scale and coherence:** Current CXL devices offer TB-scale capacity but lack hardware cache coherence, requiring explicit synchronization and incurring CPU overhead [2512.18194].
- **System bottlenecks and scalability:** Single lock managers, metadata contention, and central schedulers may become critical bottlenecks in very large racks or multi-rack deployments, necessitating hierarchical or distributed approaches.
- **Trade-off selection and tuning:** Determining optimal prefill/decode ratios, buffer sizes, eviction policies, and communication protocols requires workload-aware modeling and online adaptation [2408.08147], [2504.18154].
- **Applicability to new workloads:** Stage-wise, multi-agent, and multi-modal extensions (e.g., vLLM-Omni, EPD-Serve) introduce fragmentation, tail-latency risk, and the need for automated graph optimization across stages [2602.02204].

Open questions include the integration of hardware-coherent memory for simplified synchronization, cross-platform peer DMA for KV sharing, extension to hybrid cloud/edge or federated environments, and the generalization of disaggregated orchestration mechanisms to other data-intensive domains (databases, analytics engines).

## 7. Representative Quantitative Outcomes

The operational impact of disaggregated serving systems is substantial across application areas. Select performance results include:

| System            | Key Gains vs. Baseline                                                      |
|-------------------|-----------------------------------------------------------------------------|
| TraCT [2512.18194]| TTFT up to 9.8× lower; P99 latency up to 6.2× lower; 1.6× higher throughput |
| DistServe [2401.09670] | 2–4.5× peak goodput; 7.4–10.2× tighter SLO attainable                 |
| BanaServe [2510.13223] | 1.2–3.9× throughput, 1.4–70.1% lower latency vs. leading baselines      |
| EcoServe [2504.18154] | 65–85% goodput gain vs. non-disaggregated on commodity clusters          |
| Mooncake [2407.00079] | Up to 525% throughput gain in long-context workloads                    |
| DisaggRec [2212.00939] | 21–49.3% TCO reduction, near-monolithic throughput, with flexible scaling   |
| PrefillShare [2602.12029] | 4.5× lower p95 latency, 3.9× higher throughput for multi-model agents         |

These results demonstrate the transformative effect of disaggregation on both resource efficiency and user-perceived latency, especially under tight SLOs and scalable, heterogeneous deployments.

---

In sum, the disaggregated serving system paradigm is a foundational architectural principle for next-generation large-scale AI serving, characterized by logical and physical phase/operator separation, dynamic orchestration, and high-performance intra/inter-node communication. It enables robust, cost-effective, and SLO-compliant deployment of LLMs, recommendation engines, and multimodal inferencing in both datacenter and cloud-edge environments [2512.18194], [2212.00939], [2401.09670], [2602.23036], [2504.18154], [2510.13223], [2408.08147], [2602.12029].

Source: https://www.emergentmind.com/topics/disaggregated-serving-system