---
title: 'GCache: Distributed KVCache for Hybrid SWA'
url: https://www.emergentmind.com/topics/gcache
type: topic
---

# GCache: Distributed KVCache for Hybrid SWA

GCache is a high-performance, general-purpose distributed cache infrastructure developed by the Xiaomi storage team and used as the L3 (third-level) KVCache backend within the MiMo-V2.5 inference pipeline. In the MiMo-V2.5 serving stack, it supports a model family that combines Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders, and it is positioned as a system component for realizing the theoretical KVCache and attention-efficiency gains of Hybrid SWA in production. The associated optimization stack combines layerwise prefetch, SWA-aware prefix cache trees, specialized placement strategies, RDMA-optimized networking, and a KVCache-affinity router, with the stated goals of strict $O(W)$ SWA storage and high cache hit rates [2607.13095].

## 1. Position within Hybrid SWA inference serving

Hybrid SWA can ideally reduce both attention compute and KVCache storage significantly compared to Full Attention, but realizing these gains in production requires substantial engineering effort. GCache addresses the distributed-systems side of that problem by providing scalable, robust distributed L3 KVCache for long-context and high-concurrency inference, where cache coherence, hit rate, and KVCache management dictate both latency and infrastructural costs [2607.13095].

Its role is explicitly tied to the multi-tier cache hierarchy used in MiMo-V2.5 inference. In conjunction with optimized device/host (L1/L2) caches, GCache enables strict $O(W)$ SWA storage at L3, ensures low-overhead, high-TTL cache storage, and maximizes reuse for both active and historical or killed sessions. The paper also places GCache inside a broader full-pipeline optimization that covers not only text generation but also sparse MoE execution and multimodal inputs, including GPU image preprocessing, parallel video decoding, and multimodal cache sharing [2607.13095].

This system-level placement matters because the benefits of Hybrid SWA are not limited to a local KVCache layout. The paper’s framing is that storage complexity, request routing, cross-tier data movement, and multimodal preprocessing must all be co-optimized; otherwise the theoretical reduction in KVCache footprint does not translate into end-to-end serving efficiency.

## 2. Distributed architecture and storage organization

GCache was originally designed to accelerate distributed file system training workloads, and was later adapted and extended into a standalone storage and caching solution for both model checkpoint distribution and as the distributed cache tier for inference. Its architecture is decentralized and scalable: consistent hashing shards keys and data homogeneously across GCache servers, while centralized state is minimized to heartbeats and service discovery through a Raft-based master that is not on the IO path [2607.13095].

The core deployment consists of distributed GCache-servers, SDK clients, a Raft-based Master, and an optional backend object store such as Ceph or HDFS. GCache-servers store KV entries and support both memory and disk as backends. SDK clients are language-agnostic and handle request slicing, asynchronous dispatch, and batched or compositional IO. Hot data, such as active session KVCache, resides in memory, while cold or inactive data persists on NVMe SSDs. The implementation emphasizes single-path, zero-copy data flows and shared-memory persistence to minimize overhead [2607.13095].

Fault tolerance is provided without multi-replica storage. The paper describes single-replica operation, fast failure detection, proactive migration, and logical session grouping to reduce the impact of node loss. Co-deployment on GPU inference nodes uses excess DRAM and NVMe SSDs on those servers, which the paper characterizes as “zero additional storage cost” [2607.13095].

## 3. SWA-aware KVCache management and cache algorithms

The defining algorithmic feature of GCache in this setting is that SWA KVCache management remains faithful to the attention mask at every tier. The SWA KVCache pool is physically distinct from the Full Attention pool, and only the sliding window of $W$ tokens per SWA layer is kept. All data movement across D2H, H2D, and L2 $\leftrightarrow$ L3 strictly tracks the SWA mask to avoid bandwidth and memory waste. Logical mapping ensures that upper layers see a unified sequence, with dual indices recording Full versus SWA state [2607.13095].

Because Hybrid SWA changes the semantics of prefix reuse, classical token-sequence-equality $\Rightarrow$ KVCache hit no longer suffices. GCache therefore uses an SWA-aware prefix cache tree with “window-safe length” matching: a prefix can be reused only up to the last $W$ tokens that still exist in the SWA pool. Prefix nodes maintain dual indices for Full and SWA segment indicators, enabling granular eviction and safe reuse. Out-of-window SWA segments are freed after chunk completion or every $N$ generated tokens [2607.13095].

Cross-tier consistency is handled explicitly. The system includes host/device repair to detect and copy lost SWA segments from device to host or vice versa, periodic L3 touch to prevent eviction of high-frequency, long-lived session prefixes, and retention for short or medium sequence requests so that certain slots are held longer in anticipation of multi-user prompts or common short requests. The paper reports that strict $O(W)$ complexity for SWA layers enables a system-level $\sim 7\times$ reduction in required cache or memory against Full Attention baselines, and it attributes higher concurrency and longer TTLs directly to these savings [2607.13095].

The theoretical justification is tied to MiMo-V2.5’s layer composition: since the majority of layers are SWA, for example 60/70 with window size 128, the theoretical reduction in both storage and computation is near $1/7$ of Full Attention. In the paper’s interpretation, smaller KVCache footprints widen the reuse window because the same storage budget can retain severalfold more concurrent cache keys [2607.13095].

## 4. RDMA data movement and KVCache-affinity routing

GCache relies on RDMA-optimized high-throughput networking, including GPU NICs such as $8\times$ 400G NICs per server, for direct, low-latency data movement between GPU servers. The reported network optimizations include NUMA binding, same-rail affinity, and IO request slicing. On the stated benchmarks, the paper reports 170 GB/s at 280$\mu$s latency for 1MB IO, and up to 350 GB/s under GPU Direct RDMA [2607.13095].

A separate but tightly integrated component is the KVCache-affinity router, described as a stateless, scalable scheduling service called LLM-Router. It tracks, per prefix, the distribution of KVCache across model instances or nodes and steers incoming requests toward the worker most likely to yield a cache hit while still load-balancing to avoid hot spots. The scoring function is

$$
\text{score}(\text{worker}) = \text{matchWeight} \times \text{prefixMatchPercentage} - \text{normalizedLoad}.
$$

The paper reports that this cache-aware scheduling boosts L2 KVCache hit rates by $\sim 25\%$ and improves input throughput by $\sim 30\%$ [2607.13095].

The significance of this router is not merely load balancing. In the system described by the paper, routing and storage are coupled: prefix locality determines whether expensive reconstruction or transfer can be avoided, so routing policy becomes part of cache design rather than a separate serving concern.

## 5. Multimodal execution and cache sharing

GCache is part of a serving pipeline that extends beyond text-only KVCache management. The paper states that the full-pipeline optimization covers multimodal inputs through GPU image preprocessing, parallel video decoding, and multimodal cache sharing, and it identifies this integration as part of the first large-scale LLM serving system in production that efficiently covers the Hybrid SWA + MoE + multimodal composite architecture [2607.13095].

Within the multimodal subsystem, embedding caches can be shared across multiple modalities such as image, audio, and text, and across GPUs on the same host through shared memory. Encoder routing uses consistent hashing so that requests with the same key are routed to the same encoder, thereby maximizing cache reuse. The paper also reports efficient cross-request batching for encoders to improve both cache affinity and resource utilization [2607.13095].

An empirical result attached to this design is that multimodal cache hit rates improved by 30% after introducing consistent hashing for encoder cache routing. In context, this indicates that the cache system is not restricted to decoder-side token KVCache; it also supports encoder-side reuse across multimodal workloads, which is essential when vision or audio preprocessing becomes a material fraction of end-to-end latency [2607.13095].

## 6. Reported performance and other uses of the name

The paper reports high cache reuse and low infrastructure overhead for the MiMo-V2.5 deployment. The following figures are explicitly stated for the production system and its benchmarked components [2607.13095].

| Metric | Reported value |
|---|---|
| Server-side average cache hit rate | 93% |
| Cache hit rate for power users or heavy sessions | exceeds 95% |
| RDMA read throughput and latency | 170 GB/s at 280$\mu$s latency (1MB IO) |
| GPU Direct RDMA throughput | up to 350 GB/s |
| L2 KVCache hit-rate gain from affinity scheduling | $\sim 25\%$ |
| Input throughput gain from affinity scheduling | $\sim 30\%$ |
| Multimodal cache hit-rate improvement | 30% |
| SWA-related system-level cache or memory reduction | $\sim 7\times$ |

The name **GCache** also appears in unrelated arXiv contexts. In "A Generative Caching System for Large Language Models" [2503.17603], GCache denotes a semantic and generative cache for LLM responses rather than a distributed KVCache backend. That system emphasizes generative synthesis over multiple cached responses, adaptive semantic thresholds, hierarchical L1/L2 caching, and a reported throughput of $\sim 45$ lookups/sec versus $\sim 5$ for GPTCache on the stated setup [2503.17603]. In a different hardware context, "Flexible Support for Fast Parallel Commutative Updates" describes CCache and states that it is also referred to as GCache in some contexts; there the focus is on on-demand privatization for commutative shared-memory updates, with reported speedups up to 3.2$\times$ [1709.09491].

Within the MiMo-V2.5 literature, however, GCache has a specific and narrower meaning: a distributed L3 KVCache infrastructure whose importance lies in making Hybrid SWA, sparse MoE, and multimodal encoders operational at production scale through coordinated storage design, prefix-safe reuse, RDMA transport, and cache-affinity scheduling [2607.13095].

Source: https://www.emergentmind.com/topics/gcache