HMA-Serve: MemHA LLM Inference

• HMA-Serve is a MemHA-centric serving system that assigns compute-bound prefill to GDDR accelerators and memory-bound decode to HBM GPUs for cost-effective LLM inference.
• It leverages phase-wise quantization, a compute-transfer pipeline, and deferred dequantization to reconcile hardware heterogeneity and improve throughput.
• Empirical evaluations show up to 3.2× higher goodput and 4.8× better goodput-per-dollar compared to homogeneous serving systems.

HMA-Serve is a MemHA-centric disaggregated serving system for LLM inference that pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, and is designed explicitly for memory-heterogeneous, often cross-vendor deployments (Wei et al., 29 Jun 2026). Its central premise is that LLM inference is phase-asymmetric: prefill is compute-bound, whereas decode is memory-bound. On that basis, HMA-Serve assigns prompt processing to lower-cost GDDR accelerators and autoregressive generation to HBM GPUs, then bridges the resulting cross-vendor KV-cache, layout, and software-stack incompatibilities through phase-wise quantization, a compute-transfer pipeline, and deferred dequantization (Wei et al., 29 Jun 2026).

1. Motivation and problem setting

HMA-Serve emerges from the observation that prefill and decode stress fundamentally different hardware resources. For a sequence of $T$ prompt tokens, prefill amortizes weight fetches across many tokens and becomes compute-bound; decode processes one new token at a time against the full KV history and is memory-bound (Wei et al., 29 Jun 2026). The paper reports that on an A100 running dense BF16 prefill at 4K tokens across Qwen3 4B–32B, model-flop utilization exceeds $70\%$ while memory-bandwidth utilization drops below $10\%$ around 1K and under $1\%$ at 16K. At $L=4\text{K}$, the A100 leaves $96.8$–$97.2\%$ of HBM bandwidth idle, rising above $99\%$ at 16K (Wei et al., 29 Jun 2026).

This hardware mismatch motivates memory-heterogeneous accelerators, or MemHA: GDDR devices for compute-bound prefill and HBM GPUs for bandwidth-bound decode. The economic argument is explicit. The Tenstorrent Blackhole p150 delivers $664$ TFLOPS of BFP8 at about \$1,300 and is described as roughly an order of magnitude cheaper per chip than an A100, while still achieving competitive prefill throughput on 4–8K-token prompts (Wei et al., 29 Jun 2026). HMA-Serve therefore rejects the assumption that both phases should execute on identical HBM hardware.

A second motivation is interoperability. In its most economical form, MemHA is inherently cross-vendor, which breaks two assumptions used by single-vendor disaggregation: a KV format both ends consume natively, and a shared software stack (Wei et al., 29 Jun 2026). HMA-Serve is specifically the system proposed to make such cross-vendor disaggregation practical.

2. System architecture and data path

HMA-Serve executes prefill on a pool of GDDR-based accelerators, concretely a Tenstorrent p150 four-chip mesh, and decode on HBM GPUs, concretely an NVIDIA A100 80 GB (Wei et al., 29 Jun 2026). A scheduler assigns each request to a prefill worker and a decode worker, and the KV cache produced layer by layer on the prefill device is streamed over a 100 Gbps RoCE fabric to the decoder.

The producer side runs native BFP8 kernels. As each attention layer completes, it emits K/V tiles in Tenstorrent’s 32×32 BFP8 tiled layout, stages them in device DRAM, pushes them to pinned host memory via device-driven DMA at about $6.7$ GB/s, and then sends them over RDMA. The network path uses RoCE v2 with a measured effective throughput of about $70\%$0 Gb/s. On the consumer side, the A100 receives layer pages into pinned host buffers, copies them to HBM, stores them as raw BFP8 pages, and reconstructs them lazily inside a fused paged-attention kernel (Wei et al., 29 Jun 2026).

The architectural novelty is not merely phase split, but phase split under format mismatch. The producer emits Tenstorrent-native tiled BFP8; the consumer expects BF16 paged row-major buffers compatible with FlashAttention-style decode paths. HMA-Serve resolves this through a fused consumer-side path that performs BFP8-to-BF16 reconstruction, tiled-to-row-major layout conversion, and key reordering after RoPE during the first read of each page (Wei et al., 29 Jun 2026).

3. Core mechanisms

The system is organized around three co-designed mechanisms (Wei et al., 29 Jun 2026).

Mechanism	Operation	Effect
Phase-wise quantization	Prefill uses vendor-native low precision; decode stays in BF16	Higher prefill throughput without sacrificing decode fidelity
Compute-transfer pipeline	Each layer’s KV transfer overlaps with later-layer prefill	Reduces the transfer contribution to TTFT
Deferred dequantization	Raw quantized bytes are shipped and reconstructed lazily on the decode GPU	Reduces network bandwidth and HBM usage

Phase-wise quantization means that prefill runs in Tenstorrent-native low precision, with weights in mixed BFP8/BFP4 depending on operator path and activations and KV cache in BFP8, while decode runs entirely in BF16 on the A100 (Wei et al., 29 Jun 2026). The rationale is phase-specific: prefill is compute-bound and tolerant to low precision, whereas decode is memory-bound and more sensitive to cumulative numerical error over long contexts. The reconstruction relation used for BFP8 is described as

$70\%$1

with a shared block exponent $70\%$2 for each 32×32 tile (Wei et al., 29 Jun 2026).

The compute-transfer pipeline overlaps device-to-host push, RDMA, and host-to-device copy with ongoing prefill of later layers. For layer $70\%$3, the per-layer KV size is modeled as

$70\%$4

and transfer time is decomposed into

$70\%$5

Without overlap, the first-token time is approximated by

$70\%$6

whereas with pipelining it becomes

$70\%$7

The transfer tail is small whenever later-layer prefill provides enough slack to hide earlier-layer KV egress (Wei et al., 29 Jun 2026).

Deferred dequantization keeps the KV cache in raw BFP8 form inside HBM and reconstructs BF16 only at consumption time inside the attention read path. This reduces both network and H2D traffic by about $70\%$8 relative to BF16 transport and reduces HBM footprint by the same factor, because the decoder stores compressed BFP8 KV pages rather than full BF16 KV tensors (Wei et al., 29 Jun 2026). The paper reports about $70\%$9 ms for fused reconstruction of a full request’s KV versus about $10\%$0 ms for an unfused elementwise pass.

4. Scheduling, interoperability, and serving semantics

HMA-Serve is a disaggregated serving system rather than a single-kernel optimization. Requests are routed to a prefill worker and a decode worker; decode uses vLLM’s paged attention to batch active sequences, while prefill runs per-request traces on the Tenstorrent mesh (Wei et al., 29 Jun 2026). For example, the prefill mappings are reported as DP=4 for Qwen3 4B and 8B, DP=2 and TP=2 for 14B, and TP=4 for 32B, while decode always runs on one BF16 A100 (Wei et al., 29 Jun 2026).

Cross-vendor interoperability is handled at the serialization level. The producer ships raw tile bytes together with per-layer metadata containing shape $10\%$1, tile geometry, dtype/layout tag, per-tile shared exponents, and head-order information for RoPE-adjusted keys (Wei et al., 29 Jun 2026). On receipt, the consumer performs asynchronous H2D copies into preallocated HBM pages and marks them ready for decode.

Flow control is provided through RDMA send-side credits and per-request buffer pools; if consumer-side queues fill, producer egress is throttled at layer granularity (Wei et al., 29 Jun 2026). This design is important because HMA-Serve assumes no GPUDirect peer-to-peer path across vendors. Instead, it relies on rapid device-to-host push on the producer and overlapped host-to-device intake on the consumer.

A plausible implication is that HMA-Serve shifts the primary serving abstraction from “GPU-resident request” to “cross-device request pipeline,” where readiness is defined by full-layer KV arrival rather than co-residence of all inference phases on a single accelerator.

5. Empirical performance and cost model

Evaluation uses Qwen3 4B, 8B, 14B, and 32B, and three production-style traces: ShareGPT chat, LongBench QA, and arXiv summarization (Wei et al., 29 Jun 2026). The key serving metric is goodput at 90% SLO attainment, denoted $10\%$2, defined as the maximum throughput in requests per second at which at least 90% of requests meet both TTFT and TPOT SLOs. SLOs are set to $10\%$3 the no-load latency of a strong homogeneous disaggregation baseline called DistServe-Homo (Wei et al., 29 Jun 2026).

Across these models and traces, HMA-Serve delivers up to $10\%$4 higher $10\%$5 than state-of-the-art memory-homogeneous serving and up to $10\%$6 higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks (Wei et al., 29 Jun 2026). The cost model normalizes an A100 to $10\%$7 cost unit and a p150 to about $10\%$8 of an A100. Under that accounting, an HMA-Serve box consisting of one A100 plus four p150s costs $10\%$9 units, versus $1\%$0 units for a homogeneous one-prefill, one-decode A100 design (Wei et al., 29 Jun 2026).

The reported inference rates on an RTX-4080 for related model variants underline the latency trade-off between discrete and soft heads, but for HMA-Serve’s actual serving path the decisive microresults are elsewhere: device-pushed DMA achieves about $1\%$1 GB/s versus about $1\%$2 GB/s for standard host-driven readback, and fused deferred dequantization keeps decode throughput within about $1\%$3 of a dequant-free baseline at the largest viable batch (Wei et al., 29 Jun 2026).

Quality preservation is verified on MATH500 and AIME24/25. The system’s BFP8-prefill plus BF16-decode configuration tracks full BF16 with no measurable loss, whereas fully BFP8 execution degrades the hardest cases, especially for long reasoning chains (Wei et al., 29 Jun 2026). This result is central: HMA-Serve’s precision split is not incidental but the condition under which cross-vendor MemHA remains quality-neutral.

6. Limitations, design trade-offs, and broader significance

HMA-Serve’s benefits are strongest when prompts are long enough for prefill to be compute-bound and for transfer to be largely hidden under compute. For very short prompts, the paper notes that prefill may be too small to hide transfer, in which case a strong single HBM GPU can outperform a small GDDR mesh on raw prefill latency (Wei et al., 29 Jun 2026). Similarly, if network bandwidth is constrained—for example, well below the 100 Gbps RoCE setup used in evaluation—the transfer tail can re-enter the TTFT critical path (Wei et al., 29 Jun 2026).

The system also depends on vendor-specific runtime hooks: layer-complete events, device-pushed DMA, and explicit knowledge of low-precision tile format and layout conversion semantics (Wei et al., 29 Jun 2026). This makes HMA-Serve a systems co-design rather than a portable middleware layer. A common misconception would be to treat it as merely “distServe plus cheaper prefill hardware.” The paper suggests instead that MemHA requires a new serving substrate because layout, precision, and transfer orchestration are part of the algorithmic design, not just the deployment environment.

Within disaggregated LLM serving, HMA-Serve is positioned against memory-homogeneous disaggregation such as DistServe, single-vendor heterogeneous systems such as Splitwise, and phase-asymmetric precision schemes such as Mix-Quant (Wei et al., 29 Jun 2026). Its distinctive contribution is to show that cross-vendor, memory-heterogeneous serving can be both feasible and advantageous when the KV bridge is designed around raw-byte transport, overlapped egress, and fused decode-side reconstruction. In that sense, HMA-Serve is best understood not simply as a deployment recipe, but as a systems formulation of phase-specialized LLM inference under hardware heterogeneity (Wei et al., 29 Jun 2026).