Disaggregated Serving Models

Updated 22 June 2026

Disaggregated serving models are architectures that separate prefill and decode phases, enabling targeted optimization of compute-bound and memory-bound tasks.
They reduce phase interference and improve resource utilization by assigning specific hardware pools to each inference stage.
This modular approach supports multi-model, multi-tenant, and multimodal serving, allowing dynamic scaling and tailored scheduling for varied workloads.

Disaggregated serving models are system architectures for large-scale LLM inference that physically separate the major phases of an LLM workload—most classically, the compute-bound prefill (prompt encoding) and the memory-bound decode (autoregressive token generation) steps—onto distinct compute resources, such as GPU or NPU pools or even individual kernel engines. The primary objective of disaggregation is to decouple the heterogeneity in resource demand between stages, mitigate phase interference, and unlock phase-specific optimizations in batching, scheduling, parallelism, and system provisioning. A variety of disaggregation approaches have now emerged, ranging from phase-level to operator-level to generalized modular graphs, each tuned to the QOS demands of multi-agent, multi-LLM, and multimodal workloads.

1. Motivations and Principles of Disaggregation

Disaggregation addresses three major bottlenecks in LLM serving. First, colocated serving models—which handle prefill and decode within the same GPU process—suffer from prefill–decode interference: compute-bound prefill epochs degrade TPOT (time-per-output-token) for concurrent decode streams, while batched decodes delay time-to-first-token (TTFT) for new requests (Zhong et al., 2024, Mitra et al., 5 Jun 2025, He et al., 15 Oct 2025). Second, phase-coupled designs constrain both phases to the same parallelism and resource limits, resulting in inefficient hardware utilization and over-provisioning to meet tight per-phase SLOs (Zhong et al., 2024). Third, the recent rise of multi-model, multi-tenant, and multimodal serving has spotlighted further inefficiencies: redundant prefill computation and KV cache duplication for shared prompts, load imbalance between phases, and scheduling rigidity in the face of traffic bursts and agentic multi-round workflows (Woo et al., 12 Feb 2026, He et al., 16 Feb 2026, He et al., 15 Oct 2025). Disaggregated serving models introduce architectural separation to treat each phase—and even each component—independently, addressing these challenges through (i) isolated resource pools, (ii) explicit handoff of intermediate state (typically the KV cache), and (iii) phase- or module-targeted scaling and scheduling.

2. Architectural Decompositions

The central organizing principle of disaggregated serving is to map the inference pipeline into independently managed modules. The canonical split is phase-level: prefill versus decode (Zhong et al., 2024, Du et al., 25 Apr 2025, Jin et al., 2024, He et al., 15 Oct 2025). More advanced schemes further refine:

Decomposition Level	Example System	Unit of Disaggregation
Phase (Prefill–Decode)	DistServe, BanaServe	Prefill and decode GPUs
Operator (Attn–FFN–MoE)	xDeepServe, AFD (Wu et al., 27 May 2026)	Attention, FFN, Experts
Modular Stage Graph	vLLM-Omni (Yin et al., 2 Feb 2026)	Arbitrary model components
Storage/Adapter	InfiniLoRA (Chen et al., 8 Apr 2026)	LoRA adapters, KV cache server

Phase-level disaggregation assigns prompt encoding to prefill GPU/NPU pools and token emission to decode pools, with a handoff of the prompt's completed KV cache. Operator-level ("Attention-FFN Disaggregation") splits at the level of transformer submodules: for example, MoE experts and attention can be mapped to different device groups and synchronized via all-to-all exchanges (Xiao et al., 4 Aug 2025, Wu et al., 27 May 2026). Arbitrary stage-graph abstraction generalizes this, allowing serving pipelines of interconnected LLMs, diffusion blocks, and domain-specific modules, each scheduled and batched independently (Yin et al., 2 Feb 2026). Distinctions may also be made at the memory or storage layer, e.g., decoupling LoRA adapter execution or introducing disaggregated, multi-tier KV cache services (Chen et al., 8 Apr 2026, Qin et al., 2024).

3. Scheduling, Routing, and Load Balancing

Decoupling introduces new degrees of freedom and complexity in scheduling and routing. Classical phase-level systems (e.g., DistServe, P/D-Serve) statically allocate prefill and decode device pools, with requests routed in FIFO to the next-free prefill instance, then decode (Zhong et al., 2024, Jin et al., 2024). More advanced models introduce:

Model-agnostic decode pooling: SUN (Woo et al., 3 Mar 2026) shares a single frozen decode module across tasks and models, routing decode requests model-agnostically and batching cross-model, raising decode GPU utilization from ∼30–50% to ∼80–90%.
Cache-locality-aware routing: Mooncake (Qin et al., 2024) and PrefillShare (Woo et al., 12 Feb 2026) incorporate prefix/KV cache hit-rates and optimized early rejection to maximize cache locality and reduce redundant compute, with cache-hot content routed to maximize reuse.
Dynamic, load-aware dispatch: BanaServe (He et al., 15 Oct 2025) employs purely load-driven routing, decoupling cache placement via a global store. Adaptive rebalancing mechanisms, including layer/attention-level migration of state, enable fast response to workload skew.
Multi-stage and multi-modal orchestration: EPD-Serve (Bai et al., 5 Jan 2026) and vLLM-Omni (Yin et al., 2 Feb 2026) generalize routing for multimodal and any-to-any model graphs, organizing requests over stage-specific per-instance load and demand, including modality-aware path selection.
Game-theoretic and feedback-driven adaptation: Recent work introduces regime-aware controllers using Price of Anarchy (PoA) analysis to detect when queueing or cache-affinity strategies degrade, dynamically adapting routing parameters (e.g., overlap weights, randomness, queue sensitivity) to maintain bounded inefficiency under saturation (Georgiou, 11 Jun 2026).

4. Performance, Energy, and Scalability

Disaggregation unlocks new performance regimes, but also introduces transfer overheads and system bottlenecks.

Latency and Throughput: Disaggregated serving, when paired with phase- or operator-specific optimizations, substantially reduces TTFT and TPOT under tight SLOs. For example, DistServe increases per-GPU goodput by 2–4× and supports up to 10.2× tighter latency constraints (Zhong et al., 2024). Operator-level disaggregation in AFD sustains up to 4k tokens/s throughput under configurations where baseline deployments are infeasible (Wu et al., 27 May 2026).
Transfer Overheads: The principal cost is the KV cache transfer from prefill to decode. Empirical studies indicate that with high-bandwidth interconnects (PCIe/NVLink), the transfer duration is <0.1% of end-to-end latency (Zhong et al., 2024, Li et al., 14 Nov 2025), but can become a bottleneck for ultra-long contexts or low-bandwidth deployments.
Energy Considerations: Benchmarking (e.g., (Li et al., 14 Nov 2025)) shows that although per-phase DVFS (dynamic voltage/frequency scaling) is possible under disaggregation, overall energy per token typically increases versus a fair colocated baseline, except in rare large-batch settings. BiScale (Basit et al., 21 Feb 2026) achieves up to 39%/48% prefill/decode energy reductions via joint phase-aware placement and two-level DVFS.
Scalability: Systems such as P/D-Serve (Jin et al., 2024) and BanaServe (He et al., 15 Oct 2025) have demonstrated reliable deployment at the scale of tens of thousands of devices, with dynamic organization of P/D pools, auto-scaling, and MLOps integration. PaDG (partially disaggregated) approaches like EcoServe (Du et al., 25 Apr 2025) leverage rolling temporal disaggregation and macro-instance scheduling for cost-effective scalability on commodity hardware, without requiring ultra-high-bandwidth interconnects.

5. Extensions: Multi-Model, Multi-Tenancy, and Multimodality

Disaggregation enables architectural flexibility for advanced use cases beyond the single-model, single-tenant regime.

Multi-LLM and agentic serving: SUN (Woo et al., 3 Mar 2026) and PrefillShare (Woo et al., 12 Feb 2026) independently show how decoder or prefill modules can be universally shared across tasks/models, via prefill-only or decode-only fine-tuning, yielding 2x+ throughput gains and greatly improved utilization under Zipf-skewed or agentic (multi-module, multi-round) workloads. These designs reduce redundant compute and memory for shared prefixes, with PrefillShare reporting up to 4.5x tail-latency reduction.
LoRA and adapter scalability: Disaggregation at the fine-tuning layer decouples adapter execution from base-model inference. InfiniLoRA (Chen et al., 8 Apr 2026) scales multi-LoRA and MoE serving, increasing SLO-compliant request rates by 3.05x and LoRA SLO-attainment by 54% versus coupled designs.
General Graphs, Multimodality, and Arbitrary Stages: vLLM-Omni (Yin et al., 2 Feb 2026) introduces the stage-graph approach, abstracting inference as a directed acyclic graph of model–engine pairs, supporting arbitrary combinations across modalities (text, image, audio), and any-to-any pipelines. This delivers up to 91.4% reduction in job completion time and high realized batching efficiency per stage.
Temporal and Partial Disaggregation: Approaches such as EcoServe (Du et al., 25 Apr 2025) (PaDG), DynaServe (Ruan et al., 12 Apr 2025), and semi-PD (Hong et al., 28 Apr 2025) explore intra-instance temporal partitioning, elastic resource sharing at the SM level, and token-level micro-request splits, further bridging the gap between rigid aggregation and full disaggregation for higher goodput and tail-latency improvements.

6. System Design, Coordination, and Practical Guidelines

Designing Pareto-optimal disaggregated serving systems requires careful attention to bottleneck identification, rate-matching, resource elasticity, and coordination across layers.

Dynamic Rate Matching and Elastic Scaling: The optimal ratio of prefill to decode resources is governed by workload mix, context lengths, SLOs, and device bandwidths. Dynamic control policies allocate pools to track traffic shifts, with scaling triggers reflecting observed utilization and queue backlogs (Mitra et al., 5 Jun 2025).
Unified Storage and Memory Management: Semi-PD (Hong et al., 28 Apr 2025) and BanaServe (He et al., 15 Oct 2025) address the inefficiency of replicated weights/KV cache and storage imbalance by introducing unified memory management, atomic SM-level partitioning with fast switching, and global cache stores, supporting high utilization and ultra-fast reconfiguration.
Load-aware and Queue-free Scheduling: Gateways and routers adopt queue-free or immediate-admission (on-demand forwarding) policies to minimize queuing delay and SLO violation exposure, notably in P/D-Serve (Jin et al., 2024) and BanaServe (He et al., 15 Oct 2025).
Optimized Data Movement and Communication: Block-free and overlapping layerwise transfers via RDMA/NVLink, all-to-all and point-to-point data routing (XCCL, CloudMatrix384 in xDeepServe (Xiao et al., 4 Aug 2025)), and staged or grouped KV transfers (EPD-Serve (Bai et al., 5 Jan 2026)) are critical to sustaining low transfer overhead and high concurrency at scale.

Practical deployment guidelines include preferring phase disaggregation only for prefill-heavy or large models; using colocated serving for small models or decode-heavy traffic; tailoring phase-specific parallelism and batching; implementing dynamic scaling loops; and dimensioning interconnect for worst-case transfer demands (Mitra et al., 5 Jun 2025).

7. Limitations, Challenges, and Future Directions

While disaggregated serving delivers marked improvements, several limitations persist:

Energy Costs: Stage isolation increases energy per token unless combined with sophisticated DVFS and phase-aware provisioning (Li et al., 14 Nov 2025, Basit et al., 21 Feb 2026).
Coordination Overheads: As modules proliferate (operator-level, LoRA, multi-agent), queuing, data transfer, and scheduling complexity grows, requiring advanced prediction, feedback and in some cases formal game-theoretic control to avoid inefficiency at saturation (Georgiou, 11 Jun 2026).
Architectural Assumptions: Many sharing strategies such as SUN (Woo et al., 3 Mar 2026) require a common backbone architecture for KV cache compatibility; support for heterogeneous or hybrid models (e.g., mixing Qwen and Llama) is an open area, with distillation and alignment layers proposed as possible remedies.
Autoscaling and Foresight: Most current systems rely on reactive policies (load-aware, windowed statistics); future work is needed on workload forecasting, reinforcement learning-driven scheduling, and elastic scaling under price constraints.

Open questions remain on efficient deployment over WAN/multi-region clusters, integration of predictive scheduling and routing policies, and extending the paradigm to next-generation workloads—including trillion-parameter models, extremely long-context, and fully dynamic modular inference.

References: