--- title: Disaggregated Serving Models url: https://www.emergentmind.com/topics/disaggregated-serving-models type: topic --- # Disaggregated Serving Models Disaggregated serving models are system architectures for large-scale language model (LLM) inference that physically separate the major phases of an LLM workload—most classically, the compute-bound prefill (prompt encoding) and the memory-bound decode (autoregressive token generation) steps—onto distinct compute resources, such as GPU or NPU pools or even individual kernel engines. The primary objective of disaggregation is to decouple the heterogeneity in resource demand between stages, mitigate phase interference, and unlock phase-specific optimizations in batching, scheduling, parallelism, and system provisioning. A variety of disaggregation approaches have now emerged, ranging from phase-level to operator-level to generalized modular graphs, each tuned to the QOS demands of multi-agent, multi-LLM, and multimodal workloads. ## 1. Motivations and Principles of Disaggregation Disaggregation addresses three major bottlenecks in LLM serving. First, colocated serving models—which handle prefill and decode within the same GPU process—suffer from prefill–decode interference: compute-bound prefill epochs degrade TPOT (time-per-output-token) for concurrent decode streams, while batched decodes delay time-to-first-token (TTFT) for new requests [2401.09670, 2506.05508, 2510.13223]. Second, phase-coupled designs constrain both phases to the same parallelism and resource limits, resulting in inefficient hardware utilization and over-provisioning to meet tight per-phase SLOs [2401.09670]. Third, the recent rise of multi-model, multi-tenant, and multimodal serving has spotlighted further inefficiencies: redundant prefill computation and KV cache duplication for shared prompts, load imbalance between phases, and scheduling rigidity in the face of traffic bursts and agentic multi-round workflows [2602.12029, 2602.14516, 2510.13223]. Disaggregated serving models introduce architectural separation to treat each phase—and even each component—independently, addressing these challenges through (i) isolated resource pools, (ii) explicit handoff of intermediate state (typically the KV cache), and (iii) phase- or module-targeted scaling and scheduling. ## 2. Architectural Decompositions The central organizing principle of disaggregated serving is to map the inference pipeline into independently managed modules. The canonical split is phase-level: prefill versus decode [2401.09670, 2504.18154, 2408.08147, 2510.13223]. More advanced schemes further refine: | Decomposition Level | Example System | Unit of Disaggregation | |--------------------------|-----------------------------|---------------------------------| | Phase (Prefill–Decode) | DistServe, BanaServe | Prefill and decode GPUs | | Operator (Attn–FFN–MoE) | xDeepServe, AFD [2605.28302]| Attention, FFN, Experts | | Modular Stage Graph | vLLM-Omni [2602.02204] | Arbitrary model components | | Storage/Adapter | InfiniLoRA [2604.07173] | LoRA adapters, KV cache server | Phase-level disaggregation assigns prompt encoding to prefill GPU/NPU pools and token emission to decode pools, with a handoff of the prompt's completed KV cache. Operator-level ("Attention-FFN Disaggregation") splits at the level of transformer submodules: for example, MoE experts and attention can be mapped to different device groups and synchronized via all-to-all exchanges [2508.02520, 2605.28302]. Arbitrary stage-graph abstraction generalizes this, allowing serving pipelines of interconnected LLMs, diffusion blocks, and domain-specific modules, each scheduled and batched independently [2602.02204]. Distinctions may also be made at the memory or storage layer, e.g., decoupling LoRA adapter execution or introducing disaggregated, multi-tier KV cache services [2604.07173, 2407.00079]. ## 3. Scheduling, Routing, and Load Balancing Decoupling introduces new degrees of freedom and complexity in scheduling and routing. Classical phase-level systems (e.g., DistServe, P/D-Serve) statically allocate prefill and decode device pools, with requests routed in FIFO to the next-free prefill instance, then decode [2401.09670, 2408.08147]. More advanced models introduce: - **Model-agnostic decode pooling**: SUN [2603.02599] shares a single frozen decode module across tasks and models, routing decode requests model-agnostically and batching cross-model, raising decode GPU utilization from ∼30–50% to ∼80–90%. - **Cache-locality-aware routing**: Mooncake [2407.00079] and PrefillShare [2602.12029] incorporate prefix/KV cache hit-rates and optimized early rejection to maximize cache locality and reduce redundant compute, with cache-hot content routed to maximize reuse. - **Dynamic, load-aware dispatch**: BanaServe [2510.13223] employs purely load-driven routing, decoupling cache placement via a global store. Adaptive rebalancing mechanisms, including layer/attention-level migration of state, enable fast response to workload skew. - **Multi-stage and multi-modal orchestration**: EPD-Serve [2601.11590] and vLLM-Omni [2602.02204] generalize routing for multimodal and any-to-any model graphs, organizing requests over stage-specific per-instance load and demand, including modality-aware path selection. - **Game-theoretic and feedback-driven adaptation**: Recent work introduces regime-aware controllers using Price of Anarchy (PoA) analysis to detect when queueing or cache-affinity strategies degrade, dynamically adapting routing parameters (e.g., overlap weights, randomness, queue sensitivity) to maintain bounded inefficiency under saturation [2606.17081]. ## 4. Performance, Energy, and Scalability Disaggregation unlocks new performance regimes, but also introduces transfer overheads and system bottlenecks. - **Latency and Throughput**: Disaggregated serving, when paired with phase- or operator-specific optimizations, substantially reduces TTFT and TPOT under tight SLOs. For example, DistServe increases per-GPU goodput by 2–4× and supports up to 10.2× tighter latency constraints [2401.09670]. Operator-level disaggregation in AFD sustains up to 4k tokens/s throughput under configurations where baseline deployments are infeasible [2605.28302]. - **Transfer Overheads**: The principal cost is the KV cache transfer from prefill to decode. Empirical studies indicate that with high-bandwidth interconnects (PCIe/NVLink), the transfer duration is <0.1% of end-to-end latency [2401.09670, 2601.08833], but can become a bottleneck for ultra-long contexts or low-bandwidth deployments. - **Energy Considerations**: Benchmarking (e.g., [2601.08833]) shows that although per-phase DVFS (dynamic voltage/frequency scaling) is possible under disaggregation, overall energy per token typically increases versus a fair colocated baseline, except in rare large-batch settings. BiScale [2602.18755] achieves up to 39%/48% prefill/decode energy reductions via joint phase-aware placement and two-level DVFS. - **Scalability**: Systems such as P/D-Serve [2408.08147] and BanaServe [2510.13223] have demonstrated reliable deployment at the scale of tens of thousands of devices, with dynamic organization of P/D pools, auto-scaling, and MLOps integration. PaDG (partially disaggregated) approaches like EcoServe [2504.18154] leverage rolling temporal disaggregation and macro-instance scheduling for cost-effective scalability on commodity hardware, without requiring ultra-high-bandwidth interconnects. ## 5. Extensions: Multi-Model, Multi-Tenancy, and Multimodality Disaggregation enables architectural flexibility for advanced use cases beyond the single-model, single-tenant regime. - **Multi-LLM and agentic serving**: SUN [2603.02599] and PrefillShare [2602.12029] independently show how decoder or prefill modules can be universally shared across tasks/models, via prefill-only or decode-only fine-tuning, yielding 2x+ throughput gains and greatly improved utilization under Zipf-skewed or agentic (multi-module, multi-round) workloads. These designs reduce redundant compute and memory for shared prefixes, with PrefillShare reporting up to 4.5x tail-latency reduction. - **LoRA and adapter scalability**: Disaggregation at the fine-tuning layer decouples adapter execution from base-model inference. InfiniLoRA [2604.07173] scales multi-LoRA and MoE serving, increasing SLO-compliant request rates by 3.05x and LoRA SLO-attainment by 54% versus coupled designs. - **General Graphs, Multimodality, and Arbitrary Stages**: vLLM-Omni [2602.02204] introduces the stage-graph approach, abstracting inference as a directed acyclic graph of model–engine pairs, supporting arbitrary combinations across modalities (text, image, audio), and any-to-any pipelines. This delivers up to 91.4% reduction in job completion time and high realized batching efficiency per stage. - **Temporal and Partial Disaggregation**: Approaches such as EcoServe [2504.18154] (PaDG), DynaServe [2504.09285], and semi-PD [2504.19867] explore intra-instance temporal partitioning, elastic resource sharing at the SM level, and token-level micro-request splits, further bridging the gap between rigid aggregation and full disaggregation for higher goodput and tail-latency improvements. ## 6. System Design, Coordination, and Practical Guidelines Designing Pareto-optimal disaggregated serving systems requires careful attention to bottleneck identification, rate-matching, resource elasticity, and coordination across layers. - **Dynamic Rate Matching and Elastic Scaling**: The optimal ratio of prefill to decode resources is governed by workload mix, context lengths, SLOs, and device bandwidths. Dynamic control policies allocate pools to track traffic shifts, with scaling triggers reflecting observed utilization and queue backlogs [2506.05508]. - **Unified Storage and Memory Management**: Semi-PD [2504.19867] and BanaServe [2510.13223] address the inefficiency of replicated weights/KV cache and storage imbalance by introducing unified memory management, atomic SM-level partitioning with fast switching, and global cache stores, supporting high utilization and ultra-fast reconfiguration. - **Load-aware and Queue-free Scheduling**: Gateways and routers adopt queue-free or immediate-admission (on-demand forwarding) policies to minimize queuing delay and SLO violation exposure, notably in P/D-Serve [2408.08147] and BanaServe [2510.13223]. - **Optimized Data Movement and Communication**: Block-free and overlapping layerwise transfers via RDMA/NVLink, all-to-all and point-to-point data routing (XCCL, CloudMatrix384 in xDeepServe [2508.02520]), and staged or grouped KV transfers (EPD-Serve [2601.11590]) are critical to sustaining low transfer overhead and high concurrency at scale. Practical deployment guidelines include preferring phase disaggregation only for prefill-heavy or large models; using colocated serving for small models or decode-heavy traffic; tailoring phase-specific parallelism and batching; implementing dynamic scaling loops; and dimensioning interconnect for worst-case transfer demands [2506.05508]. ## 7. Limitations, Challenges, and Future Directions While disaggregated serving delivers marked improvements, several limitations persist: - **Energy Costs**: Stage isolation increases energy per token unless combined with sophisticated DVFS and phase-aware provisioning [2601.08833, 2602.18755]. - **Coordination Overheads**: As modules proliferate (operator-level, LoRA, multi-agent), queuing, data transfer, and scheduling complexity grows, requiring advanced prediction, feedback and in some cases formal game-theoretic control to avoid inefficiency at saturation [2606.17081]. - **Architectural Assumptions**: Many sharing strategies such as SUN [2603.02599] require a common backbone architecture for KV cache compatibility; support for heterogeneous or hybrid models (e.g., mixing Qwen and Llama) is an open area, with distillation and alignment layers proposed as possible remedies. - **Autoscaling and Foresight**: Most current systems rely on reactive policies (load-aware, windowed statistics); future work is needed on workload forecasting, reinforcement learning-driven scheduling, and elastic scaling under price constraints. Open questions remain on efficient deployment over WAN/multi-region clusters, integration of predictive scheduling and routing policies, and extending the paradigm to next-generation workloads—including trillion-parameter models, extremely long-context, and fully dynamic modular inference. --- References: - [2401.09670], [2407.00079], [2408.08147], [2504.18154], [2504.09285], [2504.19867], [2506.05508], [2508.02520], [2510.13223], [2601.08833], [2601.11590], [2602.02204], [2602.12029], [2602.14516], [2602.18755], [2603.02599], [2604.07173], [2605.28302], [2606.17081] Source: https://www.emergentmind.com/topics/disaggregated-serving-models