---
title: Disaggregated LLM Inference
url: https://www.emergentmind.com/topics/disaggregate-llm-inference
type: topic
---

# Disaggregated LLM Inference

Disaggregate LLM Inference is a systems-level paradigm that decomposes the canonical monolithic inference pipeline for transformer-based Large Language Models (LLMs) into physically and logically independent computational microservices, each specialized for a substage such as tokenization, embedding, prompt prefill (KV-cache construction), and autoregressive decode. This architecture contrasts sharply with traditional single-replica or multi-replica designs and is motivated by marked phase heterogeneity in compute/memory requirements, network characteristics, and latency-sensitivity within the LLM inference workflow. The principal goal of disaggregation is to maximize hardware utilization, minimize tail latency, and optimize resource allocation for production LLM serving at scale [2407.12391].

## 1. Architectural Principles and Patterns

The disaggregated inference pipeline decomposes into distinct microservices mapped to logical stages:

- **Tokenization Service:** A CPU-based microservice that converts input text into fixed-length arrays of token IDs. Typical latency $T_{\text{tok}}$ is negligible but critical as the first step.
- **Embedding Lookup Service:** Runs on CPU or GPU depending on the model. Acts as a vector-gather RPC over the embedding table, with lookup latency $T_{\text{emb}}$ often hidden by pipelining.
- **Prefill Service (Prompt Encoding):** Processes one-time prompt encoding over large batches on high-memory GPUs. Responsible for building the complete KV-cache for each request.
- **KV-Cache Management:** Specialized attention memory handlers (e.g., RingAttention, PagedAttention, vAttention) manage per-request key/value pages, shard state across devices, or support demand paging to optimize cache locality.
- **Decode Service:** Autoregressive token generation, mapped to low-latency GPUs or even CPUs, providing one-token-at-a-time output with token-level batching.
- **Output Projection Service:** Optionally isolated as a thin GEMM microservice.

Microservice-pipeline disaggregation (prefill/decode split) is foundational, but further disaggregation includes per-layer/component fanning (pipeline parallelism), expert-service routing in MoE models, and sharded attention-head execution [2407.12391].

### Example Disaggregated Pipeline Steps

| Substage                | Typical Execution Target    | Key Service Characteristics             |
|-------------------------|----------------------------|-----------------------------------------|
| Tokenization            | CPU                        | Fixed-latency, handoff to embedding     |
| Embedding Lookup        | CPU/GPU                    | Batched RPCs, pipelined                 |
| Transformer Layer(s)    | GPU cluster/node           | Blockwise pipeline/fan-out/fan-in       |
| KV-Cache Management     | Specialized memory service | Paged, sharded, virtual-contiguous      |
| Output Projection       | Thin GPU                   | Often fused, but can be specialized     |
| Decoding                | Low-latency GPU            | Token-granular, continuous batching     |

Patterns such as pipeline parallelism, Mixture-of-Experts dynamic routing, fan-out/fan-in across attention heads or layers, and cluster-wide KV-cache sharding support further scale-out and fine-grained modularity [2407.12391].

## 2. Latency, Throughput, and Resource Optimization

Disaggregation introduces inter-service RPC, network-hops, and serialization boundaries, increasing $T_{\text{comm}}$ per substage. However, it enables substantial system-level gains:

- **Throughput** increases as microservices batch more aggressively over homogeneous work units (large prompt batches during prefill, small dynamic batches during decode).
- **Tail Latency** is dramatically reduced by phase isolation; for instance, TetriInfer reports up to 40% reduction in tail token-level latency using a two-level scheduling mechanism [2401.11181].
- **Resource Utilization** is enhanced by specialization: high-memory bandwidth or larger HBM GPUs are reserved for prefill, while low-latency, cost-efficient GPUs are assigned to decode [2407.12391, 2411.05555].
- **Cost-Efficiency** is improved; Splitwise achieves up to 2× better GPU cost efficiency by specializing prefill on A100s and decode on T4s [2407.12391].

A system-level latency model for a fully disaggregated inference request:

\[
T_\text{total} = T_\text{tok} + T_\text{emb} + \sum_{i=1}^N (T_{\text{trans}_i} + T_{\text{comm}_i}) + T_\text{proj} + T_\text{dec}
\]

Batching and scheduling are optimized under throughput and latency SLOs by trading off batch size $B$ and per-token computation/communication overheads [2407.12391]. Empirically, throughput and tail latency improvements are observed across production-like workloads [2407.12391, 2506.21901, 2411.05555].

## 3. Algorithmic Mechanisms and Practical Implementations

Several systems exemplify distinct algorithmic choices for disaggregated LLM inference:

- **KVDirect**: Implements distributed disaggregation by decoupling prefill and decode across nodes with a tensor-centric, RDMA-based KV cache transfer layer. Pull-based KV transfer allows decode workers to fetch cache data on-demand, reducing per-request latency by 55% compared to vLLM. Effective bandwidth utilization saturates the network link capacity, and decode-side GPU idle time is minimized [2501.14743].
- **AcceLLM**: Utilizes redundancy in KV-cache copies across paired accelerators to balance workload, tolerating stragglers and smoothing tail TBT. Redundancy factor $r$ is dynamically set to fit available memory. Experimental results demonstrate up to 30% improvements in latency and efficiency, with near-ideal hardware utilization [2411.05555].
- **Harli**: Addresses decode-phase underutilization by co-locating parameter-efficient finetuning tasks with inference decode, balancing memory and compute demands with unified memory management and QoS-constrained scheduling. Achieves up to 92% higher finetune throughput while maintaining strict decode latency constraints [2511.11729].
- **TetriInfer**: Enforces fixed-size, chunked prompt prefill and a two-level, resource-aware scheduler for prefill and decode assignment. Systematically partitions workloads, uses length-prediction buckets for anticipating decode resource needs, and reduces tail latency and overall resource cost [2401.11181].
- **TD-Pipe**: Temporally disaggregates pipeline parallelism, executing extended prefill and decode bursts to eliminate phase-switch bubbles. AI-based prefill, greedy memory simulation, inter-batch work stealing, and spatial-temporal intensity metrics yield throughput gains up to 2.7× over traditional pipeline approaches [2506.10470].

These systems employ microservice orchestration via Kubernetes, workload-aware scheduling, pull- vs. push-mode KV transfer, dynamic batching, speculative decode, and both hardware-level (e.g., RDMA, NVLink) and software-level (e.g., paged attention) optimizations.

## 4. Microarchitectural and Performance Analysis

Systematic GPU profiling and queueing analysis reveal the core rationale for disaggregation:

- **Prefill** is compute-bound: high GPU SM utilization (80–90%), high arithmetic intensity ($>50$ FLOPs/byte), large batched GEMMs, and good cache reuse.
- **Decode** is memory-bound: SM utilization falls to ≈30%, DRAM bandwidth saturates ($200–300$ GB/s), and L2 hit rate drops (<40%), as per studies on Llama-3, Qwen2.5, and others [2512.01644].
- Network and memory architecture become decisive bottlenecks under disaggregation, especially as KV caches are transferred at multi-GB/s rates and require careful attention management (paging/sharding vs. monolithic residency) [2501.14743, 2407.12391].
- Disaggregated systems enable phase-aware placement, such as placing prefill on compute-optimized (FLOP-rich) GPUs and decode on memory-rich or latency-optimized GPUs [2407.12391, 2511.07422].
- Energy consumption is decode-dominated; strategies such as output-projection quantization or cache locality grouping directly target the memory wall in decode [2512.01644].

The move to decoupled microservices also facilitates fine-grained fault isolation, enables elastic scaling per phase, and permits queueing-theoretic load balancing algorithms (power-of-two-choice, cache-aware assignment) [2407.12391, 2506.21901].

## 5. Case Studies and Comparative Evaluation

Empirical results across public systems and benchmarks illustrate the concrete benefits of disaggregation:

- **TetriInfer**: Yields up to 97% reduction in TTFT and 47% lower JCT on mixed prefill/decode workloads versus monolithic baselines, with 38% resource cost reduction [2401.11181].
- **KVDirect**: Demonstrates 55% latency reduction (P90) on ArXiv workloads, with KV transfer constituting only 0.5–1.1% of total latency. Pull-mode transfer reshapes decode GPU idling dynamics, enhancing queue discipline [2501.14743].
- **AcceLLM**: Achieves up to 30% higher throughput and 300% lower tail TBT than Splitwise/vLLM, with memory overhead kept to ≈5 GB/instance [2411.05555].
- **TD-Pipe**: 1.91–2.73× throughput increase over tensor/pipeline parallel approaches on PCIe clusters, by removing pipeline bubbles via temporally disaggregated switching [2506.10470].
- **Harli**: Maintains <40 ms decode time-per-output-token (TPOT) even while co-locating PEFT jobs, with near-theoretical GPU occupancy [2511.11729].

Distinctive mechanisms such as per-layer KV streaming, unified memory allocation, and chunked prefill batching support these results. For long-context or high-throughput serving scenarios, sharded KV management (RingAttention/Infinite-LLM) and dynamic cache paging (vAttention/PagedAttention) enable cluster-scale scalability with minimal software overhead [2407.12391].

## 6. Open Challenges and Future Directions

Despite their operational advantages, disaggregated LLM inference systems face significant unresolved problems:

- **Network Overhead:** High-volume KV cache transfers (often tens of MB/request) stress PCIe/NVLink fabrics; sustained scaling demands further protocol and hardware refinement [2506.21901].
- **Adaptive Autoscaling:** Jointly tracking SLOs and optimally adjusting prefill/decode fleet sizes remains unsolved at production scale; current practice relies on custom orchestration [2506.21901, 2511.07422].
- **Fault Tolerance:** Exactly-once semantics, stateful failover, and cache-consistency protocols are non-trivial when prefills and decoders fail independently [2506.21901].
- **Cache Persistence and Sharing:** Indexing and managing KV entries across multi-tenant workloads and requests challenge cache management primitives and raise safety/isolation considerations [2506.21901, 2511.07422].
- **Phase Prediction and Load Estimation:** Highly accurate output-length and resource-prediction models are critical for effective scheduling but generalize poorly across prompt and model heterogeneity [2407.12391, 2401.11181].
- **Integration with Vertical Scaling and Serverless:** Smooth migration between disaggregated, monolithic, and serverless deployments remains an ecosystem bottleneck [2506.21901].

Further optimization opportunities exist in hardware-software co-design for RDMA fabrics, speculative decode to overlap phases, and hierarchical multi-tier caching regimes for KV management at trillion-parameter scales.

---

Disaggregate LLM inference represents a paradigm shift in LLM serving, delivering order-of-magnitude improvements in latency, throughput, and cost by aligning compute/memory/network characteristics of each inference substage to its optimal hardware/software microservice [2407.12391, 2501.14743, 2411.05555, 2511.11729, 2506.21901, 2511.07422, 2512.01644]. Modularity, fine-grained scheduling, dynamic load balancing, and resource-specialized allocation are now established as necessary foundations for efficient, scalable, production-grade LLM inference.

Source: https://www.emergentmind.com/topics/disaggregate-llm-inference