Cross-Datacenter KVCache Transfer

• Cross-datacenter KVCache transfer is a technique that decouples LLM prefill from decode by moving key-value cache states across geographically distributed data centers.
• It employs hybrid-attention models, hardware-accelerated media codecs, and pipelined operations to reduce time-to-first-token and improve bandwidth utilization.
• Advanced scheduling, load balancing, and multicast strategies ensure consistency and meet deadline requirements despite network latency and heterogeneous hardware.

Cross-datacenter KVCache transfer denotes the set of systems, algorithms, and protocols employed to move the Key-Value (KV) Cache state—typically generated during transformer-based LLM inference—across geographically distributed data centers. KVCache transfer enables LLM serving deployments to decouple compute and storage, accelerate prefill phases via cache reuse, provide elasticity across heterogeneous clusters, and maintain consistency among replicas, all while contending with WAN bandwidth constraints, network latencies, and divergent hardware capabilities. The literature covers a wide spectrum, from protocol-level innovations exploiting hardware video codecs for lossless tensor transfer to network-level optimizations using multicast trees or deadline-aware traffic scheduling and systems-level solutions balancing load, cache hit rates, and real-time network characteristics.

1. Architectural Principles of Cross-Datacenter KVCache Transfer

Conventional LLM serving architectures tightly couple the prefill (prompt encoding and KVCache construction) and decode (autoregressive token generation) phases within a single, RDMA-connected data center, primarily due to the high volume and bursty nature of dense-attention model KVCache outputs. In this architecture, every decode phase must wait for the entire KVCache to be available with sub-second latency, constraining deployments to sites with 400 Gbps–class RDMA fabrics.

Recent advances leverage model-side innovations, such as hybrid-attention transformers (where only a fraction $\alpha$ of layers use full attention), to achieve substantial KVCache size reductions—by 3×–10×—making cross-datacenter transport over commodity Ethernet (25–100 Gbps) feasible in certain configurations. In PrfaaS ("Prefill-as-a-Service"), compute-intensive prefill is selectively offloaded to specialized remote clusters, and only the (possibly reduced) KVCache is sent cross-site to local decode clusters. This design decouples prefill and decode scaling, reduces overprovisioning, and improves resource utilization in hybrid or heterogeneous accelerator environments (Qin et al., 16 Apr 2026).

A similar motivation underlies distributed cache-routing and load-balancing architectures, such as GORGO, which jointly consider network latency and cache hit overlap to decide the optimal serving region for any given request, exposing cross-region KVCache reuse as a critical primitive for low tail-latency serving (Toniolo et al., 12 Feb 2026).

2. Algorithms and Techniques for Efficient KVCache Transport

Efficient transmission of the KVCache must address two core challenges: minimizing the time-to-first-token (TTFT) when serving LLM inference, and containing network and compute resource consumption. Recent systems such as KVFetcher (Mi et al., 10 Feb 2026) employ hardware-accelerated media codecs for this purpose:

Codec-Friendly KVCache Representation:

• KVFetcher maps the four-dimensional KVCache tensor $[N_{tok},L,H,D]$ into groups of video frames compatible with H.265/HEVC intra/inter-frame prediction. The mapping prioritizes slicing along the token dimension to maximize redundancy removal and compacts layer-triplets into frame blocks that preserve spatial and head-wise token order, supporting lossless compression.
• Empirical results indicate that under such layout, lossless H.265 compression achieves $\approx 11.9\times$ reduction, outperforming generic tensor coders (4–6×). Frame shape auto-tuning further optimizes bitrate for given GPU models and sequence lengths.

Pipelined Fetch and Decode:

• Transmission, decoding (NVDEC ASIC), and restoration (CUDA kernel) stages are overlapped. As each frame arrives, the on_frame_probe callback asynchronously reconstructs the tensor region, hiding restoration cost under the decode step and ensuring decode throughput matches or exceeds network receive rate.
• Adaptive chunk sizing and resolution selection minimize pipeline bubbles on fluctuating links, while fetching-aware schedulers and queue management prevent resource contention for both reuse and non-reuse workloads.

Bandwidth-Aware Offloading and Scheduling:

• PrfaaS explicitly selects which requests to offload cross-datacenter by computing the incremental uncached sequence length and comparing it against a dynamically tuned threshold $t$. Requests with $l>t$ are offloaded and their KVCache shipped; others are decoded locally (Qin et al., 16 Apr 2026).
• The split of prefill between local and offload clusters ($N_p:N_{prf}$) and the threshold $t$ are co-optimized to maximize $\Lambda_{\max}$, the sustained inbound request rate, under observed bandwidth and cluster sizes.

3. Deadline, Consistency, and Multipoint Replication Strategies

KVCache transfer is not limited to single-source, single-destination protocols. Update replication (e.g., invalidations or broadcast cache entries) over WANs is mapped to point-to-multipoint (P2MP) or multicast flows, motivating algorithms from the network replication literature (Noormohammadpour et al., 2017, Noormohammadpour et al., 2017):

P2MP Scheduling with Forwarding Trees:

• In DCCast and DDCCast, each replication request is assigned a Steiner tree $T_R$ rooted at the source data center and spanning all destinations. This approach minimizes aggregate bandwidth and reduces duplication on core WAN links compared to naive P2P transfer, especially as the number of destinations grows ($d \gtrsim 4$).
• Rate allocation across the tree is optimized by tracking per-link load $[N_{tok},L,H,D]$0, ensuring no individual link exceeds its scheduled capacity in any slot.

Deadline-Aware Admission and ALAP Scheduling:

• DDCCast extends DCCast by enforcing deadlines $[N_{tok},L,H,D]$1 for each KVCache replication. Admission control tests whether aggregate path capacity suffices to complete transfer before $[N_{tok},L,H,D]$2, rejecting requests otherwise.
• Admitted requests are scheduled using As-Late-As-Possible (ALAP) policy, allocating bandwidth near the deadline to maximize concurrent admissions and minimize interference, subject to network constraints.

Consistency Guarantees and TTL Enforcements:

• For cache invalidation/replication, end-to-end correctness is maintained by enforcing transfer completion prior to cache Time-To-Live (TTL) expiry. Per-update deadlines and batch-based policies can be layered onto forwarding-tree mechanisms to map object update semantics onto network flows.

4. Load Balancing, KVCache Placement, and Routing Under Network Variability

Cross-datacenter KVCache systems must handle fluctuating demand, uneven prefix distribution, and time-varying WAN characteristics. Key approaches include:

Cache-Aware Placement and Trie Summaries:

• GORGO maintains a radix-trie of cached prefixes per region; for each new request, the longest-prefix hit is computed per region, providing an exact metric for residual prefill (Toniolo et al., 12 Feb 2026).
• A global (centralized) or decentralized (peer-to-peer) load balancer then minimizes $[N_{tok},L,H,D]$3 over all serving regions $[N_{tok},L,H,D]$4, dynamically balancing cache reuse against network and queueing delays.
• Experiments reveal that centralized proxies can cut median TTFT by 2.5× and reduce P99 pathologies compared to locality-only or overlap-only routing.

Cross-Cluster Hybrid Block Pools:

• In PrfaaS, a hybrid global block allocator assigns prefix (reuse) or transfer (ephemeral) status to each block, irrespective of whether it is stored in local or offload clusters, supporting flexible rebalancing and cross-site migrations triggered by real-time link and cache conditions.

Adaptive Rate and Resource Allocation:

• Both PrfaaS and KVFetcher employ bandwidth and load-aware adaptive control: historical link bandwidths and pool utilization drive chunk sizing, resolution, and dynamic rejection or deferment of offloaded requests to avoid saturating links or inducing queue congestion (Mi et al., 10 Feb 2026, Qin et al., 16 Apr 2026).

5. Performance Metrics, Experimental Highlights, and Limitations

Experimental studies across the referenced systems enumerate the key tradeoffs and bottlenecks:

Metric	PrfaaS-PD	KVFetcher	GORGO-Proxy
Compression ratio	–	11.9× lossless	–
Mean TTFT	2.22 s	up to 3.51× faster	225 ms (median)
Cross-DC BW usage	13 Gbps (13%)	–	–
Throughput gain	1.54× homog. PD	–	Highest tokens/s

• PrfaaS validates that selective offloading, hybrid attention models, and adaptive scheduling strategies collectively yield a 54% throughput improvement and reduce mean and tail TTFT by 50–64% compared to homogeneous deployments, using only a fraction (13%) of commodity bandwidth (Qin et al., 16 Apr 2026).
• KVFetcher demonstrates 1.52–3.51× TTFT reduction, $[N_{tok},L,H,D]$512× improvement in compression ratio over generic schemes, and zero accuracy loss, due to codec-friendly tensor layouts and pipelined GPU-native decode on NVDEC ASICs (Mi et al., 10 Feb 2026).
• GORGO confirms that network latency must be included in the optimization objective: policies that chase cache-hit only can increase TTFT by incurring high inter-region RTTs. By integrating all three cost components, GORGO’s global proxy achieves substantial reductions in TTFT and avoids queueing storms (Toniolo et al., 12 Feb 2026).
• The DCCast/DDCCast line of work quantifies bandwidth reduction (up to 50%) and increased admission (up to 25% more sessions) for multicast replication, at the expense of higher per-object setup overhead and controller complexity (Noormohammadpour et al., 2017, Noormohammadpour et al., 2017).

Limitations and caveats: Micro-sized, highly latency-sensitive transfers may not benefit from forwarding-tree machinery, since tree-setup and coordination delays dominate. Large fan-out (hundreds of replicas) can stress group-table scalability of SDN implementations, motivating hierarchical or hierarchical aggregation schemes. KVCache state consistency may require additional version management not addressed solely through multicast scheduling.

6. Practical Deployment Guidelines and Systems Integration

Recent systems literature emphasizes best practices for operationalizing cross-datacenter KVCache transfer:

• Integrate hardware-codec acceleration (e.g., NVIDIA NVDEC/NVENC with H.265 lossless) for interference-free decompression, reserving all GPU streaming multiprocessors for inference (Mi et al., 10 Feb 2026).
• Pipeline transmission, decode, and restoration of each chunk, ensuring per-layer decode completes ahead of downstream compute, enforced via queue schedulers and on_frame_probe callbacks.
• Implement bandwidth-aware schedulers that maintain historical bandwidth records, pool load, and chunk resolution lookup tables, selecting chunk sizes to minimize the pipeline bubble.
• Employ hybrid prefix-cache pools and request placement algorithms that dynamically select the destination or offload cluster based on real-time cache content, link utilization, and bandwidth thresholds (Qin et al., 16 Apr 2026).
• For cache replication and broadcast, leverage SDN-programmed multicast trees (OpenFlow or vendor-specific) and ALAP rate allocation to batch small updates and meet strict deadline requirements (Noormohammadpour et al., 2017, Noormohammadpour et al., 2017).
• Run global or regional load balancers, periodically synchronizing compact queue, trie, and latency summaries to minimize coordination traffic and avoid legacy global lock contention (Toniolo et al., 12 Feb 2026).
• Monitor egress utilization and queue dynamics at two timescales: short-term (to reject or delay offloads during saturation) and long-term (to re-optimize split of compute resources and threshold parameters).

Protocols and policies must be periodically profiled and grid-searched for new models, hardware and workload patterns, as optimal decision points (e.g., offload threshold $[N_{tok},L,H,D]$6, cluster splits) shift with model size, sequence length distribution, and real-world traffic.

---

References

• "Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter" (Qin et al., 16 Apr 2026)
• "Efficient Remote Prefix Fetching with GPU-native Media ASICs" (Mi et al., 10 Feb 2026)
• "GORGO: Maximizing KV-Cache Reuse While Minimizing Network Latency in Cross-Region LLM Load Balancing" (Toniolo et al., 12 Feb 2026)
• "DDCCast: Meeting Point to Multipoint Transfer Deadlines Across Datacenters using ALAP Scheduling Policy" (Noormohammadpour et al., 2017)
• "DCCast: Efficient Point to Multipoint Transfers Across Datacenters" (Noormohammadpour et al., 2017)