Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics

Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk. Agentic workloads hammer it: many sub-agents query one large codebase, reusing the same blocks. When that corpus outgrows one GPU it is partitioned across instances, so a query and the blocks it selects often sit on different GPUs: answering it means attention across instances. The reflex of prior cross-instance KV systems is to move the cache: pull the selected blocks to the requester. Multi-head Latent Attention inverts the arithmetic, compressing each token's key and value into one narrow vector, so a routed query row is only ~1 KB, smaller than the chunk it attends; routing the query is then often cheaper than moving the cache. Which primitive wins, over which fabric and request shape, is uncharted, least of all on device-initiated RDMA that makes per-request cross-node transfers cheap. We characterize cross-instance MLA attention on a real multi-node H100 cluster, distilling two reusable artifacts: a topology-aware cost model (probe / transfer / compute / return / merge) and a closed-form route/fetch/local predicate, whose constants we measure on real IBGDA, where the model tracks batched round-trips to within ~7%. At decode it routes the query, trading the cost of moving the cache (a ~3 ms re-adaptation splice for a contiguous chunk, or a scattered gather under selection) for a tens-of-microsecond round trip, and picks the fabric by probe latency, not peak bandwidth. We instantiate the cost model and predicate for MLA, but neither is MLA-specific: they apply wherever compression or sparse selection shrinks attention to small chunks (DeepSeek-V3.2, V4, and GLM-5.1 today). Extending them to a new architecture requires measuring just two coefficients: the routed payload and fetch's move-the-cache cost.

• The paper develops a topology-aware cost model and decision rule showing that routing a query to the cache holder can deliver a roughly 116 μs round trip at 1,024 query rows versus about 3 ms to fetch and splice the cache, while matching attention outputs to bf16-level accuracy.
• The paper finds that device-initiated RDMA outperforms host-mediated proxies for attention-sized payloads above roughly 1 KB, identifies about eight concurrent requesters as the holder-side capacity elbow, and shows that route remains favorable under substantial congestion and tensor parallelism.
• The paper demonstrates that routing is especially effective for sparse attention because holders process scattered selected entries at canonical positions without positional re-rotation, avoiding the 25–56% divergence caused by applying contiguous-cache rotation to sparse selections.

Motivation: cross-instance attention as a steady-state operation

The paper addresses a transport question in LLM serving: when a decoding query must attend over compressed KV state (c^KV) that resides on another GPU instance, should the system move the cache to the query or the query to the cache? The authors motivate this with an emerging reuse pattern at the provider-curated content layer — canonical corpora (case law, code repositories, documentation snapshots) prefilled once into MLA latent form and shared across tenants and requests. Because local prefix caching cannot capture reuse when each request precedes the shared chunk with different content, and because the hot tail of such corpora exceeds one GPU's HBM, cross-instance attention becomes routine rather than exceptional. Agentic workloads sharpen this further: many concurrent sub-agents fork one large immutable document prefix, so the shared c^KV dwarfs any agent's own suffix.

The structural argument is byte asymmetry. Under Multi-head Latent Attention, each token's key and value collapse into a single $d_{qk}{=}576$-wide vector, so a routed query row is roughly 1 KB on the wire while the chunk it attends holds thousands of equally wide vectors — about three orders of magnitude more data. Since MLA is deployed across DeepSeek V2–V4, Kimi K2.x, and GLM-5.1, the authors argue this asymmetry is broadly shared rather than idiosyncratic. The routing primitive itself is not new (DistAttention, Helix, context parallelism); what the paper claims is new is its economic case under MLA, across serving instances, on commodity fabric, using device-initiated RDMA.

The three primitives and their cost shapes

The paper frames the choice among three redistribution primitives distinguished by what crosses the fabric:

• fetch pulls the remote chunk's c^KV, re-rotates its positional encoding to the requester's offset, and splices it into the local paged cache.
• route ships the query batch to the holder, computes partial attention there, and merges the returned $(o, m, \ell)$ statistic via online softmax.
• local recomputes the chunk from scratch (re-prefill).

The load-bearing finding is that these differ in cost shape, not just magnitude. Fetch carries a flat position-adaptation splice of approximately 2.8–3.1 ms per chunk on DeepSeek-V2-Lite/H100, essentially independent of chunk size (~10% growth from 55 to 4096 tokens) because ~80% of it is the per-layer $\delta$-rotation kernel, which is launch-bound. Local scales linearly into tens of milliseconds for kilo-token chunks. Route pays neither: the holder's c^KV stays at its canonical position, so route is a latency-bound round trip rather than a bandwidth-bound transfer. The splice tax is not MLA-specific — position-independent caching on GQA/MHA models pays the analogous carved-prefix recompute (EPIC-style) — but under sparse selection neither applies, since selected entries are attended at their canonical positions.

A topology-aware cost model

The paper distills the characterization into an additive decomposition — probe / transfer / compute / return / merge — instantiated for Q-routing as $$T_{\mathrm{route}} = T_{\mathrm{probe}} + M_q(q{+}p)/BW + T_{\mathrm{compute}} + T_{\mathrm{merge}}$$. Calibrated on real H100 SXM5 nodes over NDR-200 InfiniBand with NVSHMEM IBGDA, the model yields two constants: a payload-independent probe of ≈16 μs and an effective bandwidth of ≈25 GB/s, both verified invariant across a 10× payload span (900–8736 B/row). With no refit, the model tracks batched round trips to ≈7% MAPE for $M_q \ge 512$ (≈3% for $M_q \ge 2048$); the small-batch residual is a fixed ~9 μs kernel turnaround the linear term omits. The same affine form refits four other fabrics (NVLink 3.0/4.0, PCIe Gen4/Gen5) to 2–4% MAPE by changing only its two constants, establishing the model as fabric-general in structure.

At the real MLA payload, a routed round trip at $M_q{=}1024$ measures ≈116 μs — roughly 26× cheaper than fetch's ≈3 ms splice, rising to ~125× at $M_q{=}1$. The practical consequence is that the selection predicate can be evaluated arithmetically per request from two measured fabric constants, without online profiling.

A caveat the authors state plainly: the cluster runs the legacy closed NVIDIA driver (the open kernel module destabilized the nodes), disabling GPUDirect Storage and placing IBGDA on the legacy GPUDirect-RDMA path. Absolute latencies are therefore configuration-specific and likely conservative; the conclusions rest on the relative route/fetch/local ratio, which favors routing even more strongly under a faster driver.

The device-initiated RDMA regime

Prior transport work reported NVSHMEM IBGDA slower than host-mediated proxies for MoE all-to-all dispatch (e.g., 24.3 μs vs 18.0 μs round trips in NCCL GIN measurements), raising doubt about device-initiated RDMA for fine-grained traffic. This paper shows the result does not carry to attention: at attention's kilobyte-scale payload ($q{+}p \approx 2$ KB), toggling only the transport on a fixed chunk-prefetch workload gives IBGDA a decisive win — per-fetch p50 of 6.0 ms versus 8.4 ms (+40% for the proxy) and a steady-state floor +53% higher, with wall-clock throughput tied. The guideline is a payload threshold: above ~1 KB, device-initiated RDMA wins; below it, a proxy. The throughput tie also localizes the binding constraint to the host pipeline, not the wire — a point that resurfaces as the prototype's main limitation.

Two capacity elbows bound holder-side fan-in. A pool of $K{=}8$ CUDA streams for staging incoming c^KV into the symmetric heap is the measured elbow (tail p50 −7%, floor −9% versus serialized; $(o, m, \ell)$0 regresses from oversubscription), setting the number of requester instances one holder backs before copy-engine contention caps throughput. Symmetrically, holder-side partial attention — measured with DeepSeek's production FlashMLA decode kernel — is flat up to $(o, m, \ell)$1 concurrent requesters (15–37 μs at decode scale, ~100× below the splice), then turns linear once the GPU saturates; even fully saturated at $(o, m, \ell)$2 it stays ≤0.4 ms. For the agentic fan-in workload, $(o, m, \ell)$3 is thus the point where a second replica becomes worthwhile.

Primitive selection and the crossover map

On wire bytes alone, route moves $(o, m, \ell)$4 bytes against fetch's $(o, m, \ell)$5, breaking even near $(o, m, \ell)$6. At a 2k-token chunk, routing moves ≥76% fewer bytes at decode-typical batches ($(o, m, \ell)$7). The closed-form predicate reduces to rules of thumb: default to route at decode; fetch only to amortize a chunk over many subsequent local steps; local only below ~75–220 tokens; cap concurrent flows per holder near 8; choose fabric by probe latency, not peak bandwidth.

The most consequential qualification is that wire bytes do not decide end-to-end latency at the current prototype. Measured through the Python client, route's time-to-first-token scales as ≈3.5 ms + 12.5 μs·$(o, m, \ell)$8: a fixed host overhead dominates the microsecond-scale wire cost, so a splice-free bytes-back fetch still wins end-to-end below $(o, m, \ell)$9 of a few hundred. The authors attribute this to three named implementation reductions — collapsed-response put, holder-compute amortization, dispatcher batching — and are explicit that the gap is engineering, not a defect in the model. They also distinguish it carefully from the splice tax, which burdens the semantic move-the-cache operation itself and which route avoids regardless.

Sparse selection: route as distributed indexing

The predicate extends beyond MLA because compression and selection both shrink the routable unit. Where a sparse indexer (DeepSeek-V3.2/V4 DSA, GLM-5.1, NSA) selects top-$\delta$0 scattered blocks per query, route is "that selection made distributed": the holder attends the chosen entries in place at their canonical positions, requiring no re-rotation, while fetch must gather a non-contiguous set across holders. The merge is verified exact — reproducing single-instance attention to bf16 noise floor (max-absolute 0.0012–0.0014, a 36× margin inside the 0.05 floor) on the reference implementation and to ≤0.002 on FlashMLA dense, FlashInfer paged, and FlashMLA bf16 sparse kernels across $\delta$1. Notably, applying the contiguous-reuse $\delta$2-rotation to a scattered selection diverges from the reference by 25–56%, confirming splice is a property of contiguous reuse, not selection.

Measured scatter behavior favors route sharply: gathering a 2048-entry selected set spread across $\delta$3 holders grows from ≈1.3 to ≈3.9 ms per layer as $\delta$4 goes 1→7 (scattering defeats bulk coalescing), while the route fan-out stays flat at tens of microseconds. Holder compute under selection tracks the selection budget, not store size (within ~15% across a 16× corpus range), with the indexed gather costing 1.1×–2–3× the dense kernel at matched $\delta$5 — still 17–60 μs, ~50× below the splice. Across the released selection budgets (top-512 to top-2048), the break-even spans ≈270–1080 query rows, above every decode batch, so route wins at decode across the whole family.

Sensitivity: fabrics, congestion, and tensor parallelism

A striking measurement result is that at decode the five tested fabrics cluster within 1.5× (≈31–48 μs at $\delta$6): a single thread block issues the whole transfer and can push only ≈18–25 GB/s, so the dispatch is issue-bound, not bandwidth-bound. The same H100 routes over its 900 GB/s NVLink 4.0 mesh at only ≈21 GB/s for one block — a hair below its cross-node IBGDA rate — and an A100 on NVLink 3.0 is slower than an RTX Pro 6000 on nominally 6× lower-peak PCIe Gen5. Peak bandwidth therefore does not even order the fabrics at decode; multi-block dispatch recovers true link peaks but buys headroom the route/fetch decision does not need, since route already sits >60× below fetch at decode.

Congestion robustness follows from cost structure: route stays cheapest until effective bandwidth falls below ≈0.2 GB/s (a ~125× degradation), and empirically the round trip is flat through $\delta$7 concurrent flows, rising only at full link subscription ($\delta$8; $\delta$9: 114→250 μs, +119% on the tight NDR-200 link, +36% on cross-socket PCIe Gen4, +9% on PCIe Gen5) — never approaching the splice. Topology placement within the fat-tree (same-leaf versus spine-traversing) changes nothing measurable. Under TPLA tensor parallelism, rank-paired routing cuts per-rank inter-node bytes by $$T_{\mathrm{route}} = T_{\mathrm{probe}} + M_q(q{+}p)/BW + T_{\mathrm{compute}} + T_{\mathrm{merge}}$$0 with ranks proceeding in parallel, so routing composes with TP rather than conflicting with it.

Limitations and open questions

The paper is candid about several constraints. The legacy-driver configuration makes absolute numbers a conservative operating point whose mechanism-level explanation (open-driver headroom) is asserted rather than measured. The end-to-end host-overhead gap — route losing to splice-free fetch below a few hundred query rows despite winning wire bytes — remains open pending the three named transport reductions being implemented and validated inside a live serving stack; the serving system consuming the predicate is explicitly out of scope. The idle-path warm-up artifact on isolated A40 cross-socket flows is reported but its mechanism is not isolated. Congestion is characterized only under self-congestion on otherwise-idle fabrics; contention from unrelated tenants is deferred to the serving layer. Finally, the FP8 production sparse-decode kernel is not measured directly, though the authors argue it would only narrow the sparse gather premium.

Conclusion

This paper converts the route-versus-fetch question for cross-instance MLA attention from a design reflex into a measured decision rule. Its reusable artifacts are a topology-aware five-term cost model fit to ≈7% MAPE on real IBGDA and a closed-form route/fetch/local predicate whose inputs — routed payload and fetch's move-the-cache cost — reduce to two measurable coefficients per architecture. The central results are quantitative: a ≈116 μs routed round trip against a flat ≈3 ms splice, ≥76% fewer wire bytes at decode batches, IBGDA beating a CPU proxy by 40% at attention payloads where prior work found the opposite for MoE dispatch, and holder capacity elbows near 8 concurrent requesters on both copy and compute axes. The honest boundary of the contribution is end-to-end: until the identified host-overhead reductions land, the wire-byte case for routing is necessary but not yet sufficient at small batches, and demonstrating the wall-clock win inside a production serving stack is the question this work leaves open.