---
title: 'CrossPool: Disaggregated GPU Memory for MoE'
url: https://www.emergentmind.com/topics/crosspool
type: topic
---

# CrossPool: Disaggregated GPU Memory for MoE

Searching arXiv for the specified CrossPool paper and closely related systems to ground the article.
CrossPool is a serving engine for cold mixture-of-experts (MoE) models that disaggregates GPU memory into a weights pool and a KV-cache pool in order to improve utilization under sparse, bursty, long-context traffic. In the source paper, CrossPool corresponds to the authors’ “HybridCP” system. Its central premise is that stable, model-determined FFN weights and transient, demand-determined KV-cache should not compete inside a monolithic GPU memory region, especially when colocated cold models rarely reach peak KV-cache demand simultaneously. The design combines KV-cache planning and virtualization, a layer-wise pipeline scheduler, and GPU-resident persistent kernels with control lowering; under the reported evaluation it reduces P99 TBT by up to $10.4\times$ relative to a kvcached-based multi-LLM serving baseline [2606.24506].

## 1. Problem setting and operating regime

CrossPool is formulated for **cold MoE models**, meaning sparse models that are hosted together but receive sparse requests and remain cold. The paper characterizes the resulting memory asymmetry in two parts: model weights are stable and model-determined, whereas KV-cache is transient and demand-determined. Because cold requests are sparse, their KV-cache demands fluctuate and can be multiplexed across models. This motivates shared provisioning of active KV demand rather than reserving each model’s worst-case KV capacity independently [2606.24506].

A core argument of the design is that **KV-cache sharing alone is not sufficient** when weights and KV-cache remain in a monolithic GPU memory pool. In that setting, static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity. The reported consequence is low GPU memory utilization and weak long-context support. CrossPool therefore targets a specific systems regime: low-concurrency, bursty, long-context inference for multiple sparse MoE models, rather than uniformly high-concurrency serving.

The paper’s quantitative framing also makes explicit why cold-serving differs from standard dense-model colocations. FFN layers dominate the parameter footprint, accounting for approximately $95\%$ of MoE model parameters in the authors’ table. This makes FFN-weight consolidation the dominant opportunity on the static side of memory, while dynamic sharing of KV-cache addresses the variable side.

## 2. Disaggregated memory organization

At a high level, CrossPool decomposes a single-GPU memory region into two independent pools. The **weights pool** consolidates the FFN weights of many cold MoE models and contains FFN weights only. The **KV-cache pool** holds all attention and other non-FFN operators together with the transient KV tensors. Hidden states move between the pools over NVLink, while attention remains local to the KV-cache pool and FFN execution remains local to the weights pool [2606.24506].

The paper formalizes the pooling benefit using the notation
$$
W_i = \text{footprint of model } i \text{’s weights}, \qquad
K_i(t) = \text{model } i \text{’s live KV-cache at time } t.
$$
It then defines the pooled KV-cache reservation as
$$
K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).
$$
For a cluster with $N$ GPUs of capacity $C$ each, a unified monolithic scheme must reserve
$$
\sum_i \left(W_i + \max_t K_i(t)\right)
$$
bytes in the worst case and wastes
$$
\sum_i \left(\max_t K_i(t) - E[K_i(t)]\right).
$$
Under CrossPool, the weights pool reserves $\sum_i W_i$ across GPUs, and the KV-cache pool reserves $K_{\text{total}}$ across GPUs. Overall utilization is defined as
$$
U = \frac{\sum_i W_i + K_{\text{total}}}{N\cdot C}.
$$
Since $K_{\text{total}} \le \sum_i \max_t K_i(t)$, the disaggregated design always satisfies
$$
U_{\text{disagg}} \ge U_{\text{mono}}.
$$

The reported empirical implication is that the design shows **10–30% higher utilization** on the authors’ MoE colocation traces. This is the architectural basis for the paper’s claim that CrossPool supports bursty long-context requests more effectively than monolithic weight-and-KV pooling.

## 3. KV-cache planning and virtual memory management

CrossPool’s KV-cache planner computes a **single shared KV-cache budget** that provisions all cold MoE models jointly at a chosen quantile, rather than allocating each model’s individual worst-case KV quota. The planner is trace-driven and uses a Monte Carlo quantile method over recorded or sampled requests. For model $M$, the arrival process is modeled as Poisson with rate $\lambda_M$. For request $i$, the prompt token count is $O_{p,i}$, the generated token count is $O_{d,i}$, the total decode time is $T_i$, and $\kappa(M)$ denotes bytes per KV-token for the model’s attention layout [2606.24506].

At request age $u$, the paper defines live KV tokens as
$$
Q_i(u) = \left(O_{p,i} + O_{d,i}\cdot u/T_i\right)\cdot \mathbf{1}_{0\le u<T_i}.
$$
Instantaneous model-level demand is
$$
K_M(t) = \sum_{i\,:\,A_i\le t} \kappa(M)\cdot Q_i(t-A_i),
$$
and aggregate pool demand is
$$
K_{\text{pool}}(t) = \sum_M K_M(t).
$$
Given a quantile $\alpha$, such as the 95th or 99th percentile, the planner sets
$$
P = \text{quantile}_\alpha \{K_{\text{pool}}(t)\}.
$$

Online execution uses CUDA Virtual Memory Management to reserve a virtual KV address range of size $P$ and back it with pages on demand. New requests are admitted only if free pages remain; otherwise they queue or reject. Attention kernels read and write a normal paged KV buffer, while the virtualizer maps and unmaps physical pages off the critical path. The paper’s emphasis is that this preserves attention locality while turning KV capacity into a shared, quantile-provisioned resource rather than a per-model static reservation.

## 4. Layer-wise pipeline scheduler and control lowering

CrossPool’s execution model is designed to avoid serializing the sequence **attention $\rightarrow$ hidden-state transfer $\rightarrow$ FFN $\rightarrow$ hidden-state transfer**. Instead, it overlaps attention on one batch with FFN on another batch by maintaining two in-flight batches, denoted $B_1$ and $B_2$, each tagged with $(\text{model}, \text{next\_layer})$. The GPU-resident scheduler dispatches attention graphs in the KV-cache pool and FFN graphs in the weights pool, then transfers hidden states between the pools with NVSHMEM at each boundary crossing [2606.24506].

Each persistent kernel polls its own pool’s request queue, launches the appropriate CUDA graph, and issues an NVSHMEM copy when the batch moves to the other pool. The paper highlights that exactly **two graph types** are captured—attention and FFN—which preserves CUDA Graph launch-overhead reduction while retaining a fine-grain cross-pool pipeline. Persistent kernels keep the control loop on the device rather than on the host, thereby cutting host-to-GPU latency at every layer crossing.

The scheduling complexity per decoded token is given as $O(L\times B)$, where $L$ is the number of layers and $B$ is the number of in-flight batches, with $B=2$ in the prototype. More generally, if $M$ models are colocated, worst-case host-driven admission and bookkeeping is $O(M + \#\text{active\_batches})$, but this is dominated by GPU graph launches only at model-load time.

A key systems property is **attention locality**. All KV tensors remain in the KV-cache pool and are never migrated. FFN layers execute entirely in the weights pool and exchange only $O(\text{batch\_size}\times \text{hidden\_dim})$ data per layer, independent of context length. This means that long-context requests do not incur any cost to fetch or push the growing KV tensors.

## 5. Evaluation, workloads, and empirical results

The evaluation uses **five A100 GPUs (40 GB each) connected by NVLink**. The colocated models are Qwen3-30B-A3B with 48 layers and 29 B FFN parameters, GLM-4.7-Flash with 47 layers and 28.3 B FFN parameters, and DeepSeek-V2-Lite with 27 layers and 14.9 B FFN parameters, for total weights of approximately 154 GB. Two baselines are reported: **Static Partition**, in which each model is pinned to MIG partitions, and **kvcached (Chimera)**, which elastically pools weights and KV-cache in one memory pool [2606.24506].

On **context-length scalability**, the paper reports that as LongAlign context grows from 4 K to 10 K tokens, both Static Partition and kvcached hit OOM cliffs at approximately 3 RPS because their per-model KV quotas exhaust. Under the same $5\times 40$ GB budget, CrossPool maintains approximately $1.5\times$ higher maximum RPS at 10 K tokens by sharing a larger KV-cache pool.

On **tail latency**, the balanced Vicuna ShareGPT decode trace at 0.8 RPS/model yields the following P99 TBT reductions relative to kvcached:

| Model | P99 TBT reduction |
|---|---:|
| Qwen3-30B-A3B | $7.6\times$ |
| DeepSeek-V2-Lite | $10.4\times$ |
| GLM-4.7-Flash | $7.3\times$ |

At 1.0 RPS/model, the reported speedups remain $2.1\times$, $5.0\times$, and $2.2\times$ respectively. Static Partition is reported as slightly faster in the low-latency tail because it has zero sharing overhead, but it cannot admit long-context bursts without OOM.

The ablation at 0.5 RPS/model isolates the contribution of the layer-wise pipeline and persistent kernels:

| Layer-wise pipeline / Persistent kernels | Throughput |
|---|---:|
| Off / Off | 55.4 token/s |
| Off / On | 77.9 token/s |
| On / Off | 60.8 token/s |
| On / On | 111.4 token/s |

The paper interprets these techniques as complementary: persistent kernels reduce host dispatch overhead, while the layer-wise scheduler keeps both pools busy. In the authors’ summary, the full system improves memory utilization by up to 30%, supports $2\times$–$10\times$ P99 TBT reductions, and serves $2\times$ more long-context RPS under the same hardware budget.

## 6. Limitations, future directions, and terminological scope

The paper identifies several explicit caveats. First, hidden-state transfers from attention to FFN and from FFN back to attention remain on the critical path per layer; although these transfers are small relative to KV movement, they are not fully overlapped in the current design. Second, pipeline imbalance can arise when colocated models have widely different per-layer compute costs, causing the faster model to idle while waiting for its partner’s stage. Third, CrossPool assumes a **low-concurrency cold regime**; at very high concurrency, the benefits of sharing diminish and one may revert to per-model or per-batch data parallelism [2606.24506].

The proposed future work follows directly from those limitations. The paper suggests a finer-grain **four-stage pipeline** that separates attention compute, attention-to-FFN communication, FFN compute, and FFN-to-attention communication, with the goal of overlapping communication with both adjacent compute stages. It also suggests cluster-wide model grouping by attention and FFN cost profile to reduce imbalance within each weights/KV pair of pools, and adaptive resizing of the KV pool quantile $P$ in response to diurnal or bursty variation rather than relying on a fixed offline trace.

A plausible broader implication is that CrossPool belongs to a wider movement toward **memory-centric GPU systems**. This suggests an affinity with work on CXL-backed shared memory pools for node-spanning GPU collectives, such as CCCL, although the mechanisms and target workloads differ [2602.22457]. The term **CrossPool** also has an unrelated use in the literature on proof-of-work mining, where it denotes a strategy for diversifying computational power across mining pools and cryptocurrencies under a mean–variance objective; that usage is separate from the multi-LLM serving system described here [1905.04624].

Source: https://www.emergentmind.com/topics/crosspool