HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators

Abstract: LLM inference comprises a compute-bound prefill phase and a memory-bound decode phase, and recent systems disaggregate them onto separate hardware. Yet today's datacenter GPUs rely on costly HBM whose bandwidth sits almost entirely idle during prefill. LLM serving across memory-heterogeneous accelerators (MemHA) pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, promising lower cost without sacrificing performance. Pushed to its most economical form, MemHA serving is inherently cross-vendor, since the best-suited chip for each phase may come from a different vendor. This breaks two assumptions that single-vendor disaggregation takes for granted -- a KV format both ends consume natively, and a shared software stack. We present \textbf{HMA-Serve}, a MemHA-centric disaggregated serving system pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode efficiently. HMA-Serve achieves this through (1) phase-wise quantization, applying vendor-native low precision for high-throughput prefill while keeping decode in high-precision BF16, (2) a compute-transfer pipeline that overlaps each layer's KV cache transfer with later-layer prefill to reduce time-to-first-token (TTFT), and (3) deferred dequantization, shipping raw quantized bytes and reconstructing them lazily on the decode GPU to reduce network bandwidth and HBM usage. Across four Qwen3 models (4B--32B) and three production traces, HMA-Serve delivers up to $3.2\times$ higher goodput than state-of-the-art memory-homogeneous methods and $4.8\times$ higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks.

• The paper introduces HMA-Serve, a system that disaggregates LLM serving by pairing low-cost GDDR-based accelerators for compute-bound prefill with HBM-based GPUs for memory-bound decode.
• It employs phase-wise quantization and pipelined compute-transfer strategies that achieve up to 3.2× higher throughput and 4.8× improved cost efficiency over homogeneous systems.
• The architecture maintains accuracy via deferred dequantization, offering a scalable and cost-effective blueprint for deploying large language models in heterogeneous environments.

Efficient Disaggregated LLM Serving on Memory-Heterogeneous Accelerators: An Analysis of "HBM Is Not All You Need" (2606.29986)

Introduction

The deployment of LLMs at scale presents ongoing hardware cost and efficiency challenges, particularly due to the divergent computational demands of their prefill and decode stages. The paper "HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators" (2606.29986) introduces HMA-Serve, a disaggregated serving system that pairs GDDR-based accelerators for the compute-bound prefill stage with HBM-based GPUs for the memory-bound decode stage. This architecture leverages cross-vendor, memory-heterogeneous accelerator (MemHA) designs to optimize both performance and cost, proposing a set of systematic solutions for cross-vendor KV cache transfer and precision format mismatches. The following analysis summarizes the technical contributions, empirical results, implications, and prospective directions highlighted by the paper.

Background and Motivation

LLM inference pipelines are dominated by two contrasting phases. The prefill phase is highly parallel and compute-bound, efficiently utilizing dense matrix multiplication that is well-suited for accelerators with high arithmetic throughput but moderate memory bandwidth. By contrast, the decode phase is iterative and memory-bound, stressing memory bandwidth due to repeated KV cache access and weight streaming. Commodity datacenter GPUs, such as NVIDIA's A100/H100 or AMD MI300X, rely predominantly on expensive high-bandwidth memory (HBM) to satisfy decode requirements, but this premium bandwidth is almost entirely underutilized during prefill—A100 GPUs show over 97% HBM bandwidth idleness for prefill workloads in practical settings.

Figure 1: The roofline model situates GDDR-based accelerators (e.g., Tenstorrent p150) as ideal for prefill, while HBM-based GPUs are optimal for decode, motivating a heterogeneous pairing for cost-efficiency.

Emerging GDDR-based accelerators, such as the Tenstorrent Blackhole p150, offer compute capabilities at a fraction of the price of flagship HBM GPUs. However, these devices lack the necessary bandwidth for decode and do not share a common hardware or software stack with Nvidia GPUs, challenging system integration.

HMA-Serve Architecture: Design and Mechanisms

HMA-Serve operationalizes MemHA serving by integrating a scheduling and data transfer pipeline across GDDR-based Tenstorrent accelerators (for prefill) and an HBM-based Nvidia A100 GPU (for decode), connected via a 100 Gbps RoCE RDMA fabric. Three primary mechanisms address the challenges posed by cross-vendor serving:

Figure 2: HMA-Serve architecture, outlining the cross-vendor scheduling, phase-native quantization, compute-transfer pipeline, and deferred dequantization along the serving path.

1. Phase-wise Quantization

Prefill can tolerate low-precision arithmetic without accuracy loss, so it is executed using vendor-native BFP8 on Tenstorrent. Decode, however, remains sensitive to numerical error buildup and is thus performed in BF16 on the A100. This design ensures each accelerator phase operates at peak efficiency.

2. Compute-Transfer Pipelining

To mitigate added time-to-first-token (TTFT) from cross-device KV cache transfer, HMA-Serve overlaps per-layer KV egress from the prefill device with computation in subsequent layers. Specialized device-to-host DMA and batched RDMA minimize transfer latency, and pipelined scheduling ensures that only a minimal fraction of the transfer is exposed as request overhead.

3. Deferred Dequantization

Raw quantized BFP8 KV data is transferred over the network and stored directly in HBM. Conversion to BF16 and the necessary memory/layout transformations for FlashAttention are performed within a fused kernel on the A100 during decode. Critically, this bit-manipulation-based conversion leverages idle integer ALUs, minimizing hardware contention with GEMM-heavy decode kernels.

Empirical Evaluation and Results

HMA-Serve is evaluated under real world conditions—pairing four Tenstorrent p150b accelerators for prefill with a BF16 A100-80GB for decode—across four Qwen3 model scales (4B–32B) on three production workloads. Its performance is benchmarked against DistServe-Homo (homogeneous phase disaggregation using two A100s) and Sarathi-Hetero (oracle colocation baseline).

Figure 3: Comparison of HMA-Serve, DistServe-Homo, and Sarathi-Hetero on throughput and latency metrics across four model scales, showing HMA-Serve’s superior goodput at all but the smallest model.

Performance

HMA-Serve achieves up to 3.2× higher goodput and 4.8× greater goodput-per-dollar than state-of-the-art memory-homogeneous systems under tight SLOs (joint TTFT/TPOT). Its throughput advantage increases with model size, as prefill becomes increasingly compute-heavy and parallelizable on GDDR silicon, emphasizing the limitations of homogeneous HBM-based scaling.

Cost Efficiency

Figure 4: Goodput per hardware budget normalized to DistServe-Homo, illustrating HMA-Serve’s widening advantage in cost efficiency as model and workload scale increases.

HMA-Serve’s heterogeneous integration enables much higher throughput per unit cost, especially for larger models where HBM-based prefill becomes a bottleneck. The cost normalization shows decisive superiority except in the smallest 4B regime, where performance parity can be reached by scaling out additional low-cost GDDR prefill replicas.

Accuracy

Quantitative results on reasoning benchmarks (MATH500, AIME24/25) confirm that HMA-Serve's asymmetric quantization (low-precision prefill, high-precision decode) preserves output fidelity, as fully quantized pipelines (all BFP8) result in accuracy degradation on certain tasks.

Theoretical and Practical Implications

HMA-Serve demonstrates that the optimal cost-performance regime for LLM serving is likely to lie at the intersection of heterogeneous, cross-vendor hardware, rather than within a single-vendor, monolithic stack. Phase-specific hardware provisioning makes it feasible to reduce both capex and opex for LLM deployment without compromising latency or generation quality.

From a systems perspective, HMA-Serve offers concrete design principles for decoupling phase-specific format and hardware constraints via pipeline and kernel co-design, suggesting immediate impact for LLM system integrators and hardware manufacturers. As GDDR-based accelerators evolve and further special-purpose silicon appears, the flexibility and extensibility of the HMA-Serve approach is likely to increase.

Future Prospects

The paper opens up several avenues for further work:

• Broader Hardware Integration: Exploring additional GDDR-based accelerators or SRAM-based architectures and supporting broader interconnect standards could enable larger and more flexible heterogeneous clusters.
• Precision/Format Extensions: As emerging vendors (AMD, Intel) introduce novel low-precision formats, advanced runtime scheduling could further optimize split execution and conversion.
• Granular Phase Specialization: Sub-phase specialization, such as separate hardware for RoPE or other attention mechanisms, may yield additional performance gains.
• Autoscaling and Orchestration: Integration with advanced scheduling and autoscaling policies will be necessary to realize the full cost benefits in dynamic serving workloads.

Conclusion

HMA-Serve provides a concrete, silicon-demonstrated framework for cost-effective, high-throughput LLM inference by pairing phase-specialized, memory-heterogeneous accelerators and resolving cross-vendor format mismatches through pipelined transfer and deferred quantization. The demonstrated throughput and cost-efficiency gains indicate a paradigm shift away from all-HBM or single-vendor solutions. This work substantiates the practical case for fine-grained hardware specialization in neural inference serving and presents a compelling blueprint for future AI systems infrastructure.