CrossPool: Disaggregated GPU Memory for MoE

• CrossPool is a serving engine for cold mixture-of-experts models that separates stable FFN weights from dynamic KV-cache to optimize GPU memory utilization.
• It employs KV-cache planning using Monte Carlo quantiles and a layer-wise pipeline scheduler to reduce tail latencies by up to 10.4× relative to baselines.
• The design achieves 10–30% higher memory utilization and supports 2× more long-context requests under sparse, bursty traffic on multi-LLM serving systems.

Searching arXiv for the specified CrossPool paper and closely related systems to ground the article. CrossPool is a serving engine for cold mixture-of-experts (MoE) models that disaggregates GPU memory into a weights pool and a KV-cache pool in order to improve utilization under sparse, bursty, long-context traffic. In the source paper, CrossPool corresponds to the authors’ “HybridCP” system. Its central premise is that stable, model-determined FFN weights and transient, demand-determined KV-cache should not compete inside a monolithic GPU memory region, especially when colocated cold models rarely reach peak KV-cache demand simultaneously. The design combines KV-cache planning and virtualization, a layer-wise pipeline scheduler, and GPU-resident persistent kernels with control lowering; under the reported evaluation it reduces P99 TBT by up to $10.4\times$ relative to a kvcached-based multi-LLM serving baseline (Ye et al., 23 Jun 2026).

1. Problem setting and operating regime

CrossPool is formulated for cold MoE models, meaning sparse models that are hosted together but receive sparse requests and remain cold. The paper characterizes the resulting memory asymmetry in two parts: model weights are stable and model-determined, whereas KV-cache is transient and demand-determined. Because cold requests are sparse, their KV-cache demands fluctuate and can be multiplexed across models. This motivates shared provisioning of active KV demand rather than reserving each model’s worst-case KV capacity independently (Ye et al., 23 Jun 2026).

A core argument of the design is that KV-cache sharing alone is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. In that setting, static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity. The reported consequence is low GPU memory utilization and weak long-context support. CrossPool therefore targets a specific systems regime: low-concurrency, bursty, long-context inference for multiple sparse MoE models, rather than uniformly high-concurrency serving.

The paper’s quantitative framing also makes explicit why cold-serving differs from standard dense-model colocations. FFN layers dominate the parameter footprint, accounting for approximately $95\%$ of MoE model parameters in the authors’ table. This makes FFN-weight consolidation the dominant opportunity on the static side of memory, while dynamic sharing of KV-cache addresses the variable side.

2. Disaggregated memory organization

At a high level, CrossPool decomposes a single-GPU memory region into two independent pools. The weights pool consolidates the FFN weights of many cold MoE models and contains FFN weights only. The KV-cache pool holds all attention and other non-FFN operators together with the transient KV tensors. Hidden states move between the pools over NVLink, while attention remains local to the KV-cache pool and FFN execution remains local to the weights pool (Ye et al., 23 Jun 2026).

The paper formalizes the pooling benefit using the notation

$$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$

It then defines the pooled KV-cache reservation as

$$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$

For a cluster with $N$ GPUs of capacity $C$ each, a unified monolithic scheme must reserve

$\sum_i \left(W_i + \max_t K_i(t)\right)$

bytes in the worst case and wastes

$\sum_i \left(\max_t K_i(t) - E[K_i(t)]\right).$

Under CrossPool, the weights pool reserves $\sum_i W_i$ across GPUs, and the KV-cache pool reserves $K_{\text{total}}$ across GPUs. Overall utilization is defined as

$95\%$0

Since $95\%$1, the disaggregated design always satisfies

$95\%$2

The reported empirical implication is that the design shows 10–30% higher utilization on the authors’ MoE colocation traces. This is the architectural basis for the paper’s claim that CrossPool supports bursty long-context requests more effectively than monolithic weight-and-KV pooling.

3. KV-cache planning and virtual memory management

CrossPool’s KV-cache planner computes a single shared KV-cache budget that provisions all cold MoE models jointly at a chosen quantile, rather than allocating each model’s individual worst-case KV quota. The planner is trace-driven and uses a Monte Carlo quantile method over recorded or sampled requests. For model $95\%$3, the arrival process is modeled as Poisson with rate $95\%$4. For request $95\%$5, the prompt token count is $95\%$6, the generated token count is $95\%$7, the total decode time is $95\%$8, and $95\%$9 denotes bytes per KV-token for the model’s attention layout (Ye et al., 23 Jun 2026).

At request age $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$0, the paper defines live KV tokens as

$$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$1

Instantaneous model-level demand is

$$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$2

and aggregate pool demand is

$$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$3

Given a quantile $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$4, such as the 95th or 99th percentile, the planner sets

$$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$5

Online execution uses CUDA Virtual Memory Management to reserve a virtual KV address range of size $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$6 and back it with pages on demand. New requests are admitted only if free pages remain; otherwise they queue or reject. Attention kernels read and write a normal paged KV buffer, while the virtualizer maps and unmaps physical pages off the critical path. The paper’s emphasis is that this preserves attention locality while turning KV capacity into a shared, quantile-provisioned resource rather than a per-model static reservation.

4. Layer-wise pipeline scheduler and control lowering

CrossPool’s execution model is designed to avoid serializing the sequence attention $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$7 hidden-state transfer $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$8 FFN $$W_i = \text{footprint of model } i \text{’s weights}, \qquad K_i(t) = \text{model } i \text{’s live KV-cache at time } t.$$9 hidden-state transfer. Instead, it overlaps attention on one batch with FFN on another batch by maintaining two in-flight batches, denoted $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$0 and $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$1, each tagged with $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$2. The GPU-resident scheduler dispatches attention graphs in the KV-cache pool and FFN graphs in the weights pool, then transfers hidden states between the pools with NVSHMEM at each boundary crossing (Ye et al., 23 Jun 2026).

Each persistent kernel polls its own pool’s request queue, launches the appropriate CUDA graph, and issues an NVSHMEM copy when the batch moves to the other pool. The paper highlights that exactly two graph types are captured—attention and FFN—which preserves CUDA Graph launch-overhead reduction while retaining a fine-grain cross-pool pipeline. Persistent kernels keep the control loop on the device rather than on the host, thereby cutting host-to-GPU latency at every layer crossing.

The scheduling complexity per decoded token is given as $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$3, where $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$4 is the number of layers and $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$5 is the number of in-flight batches, with $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$6 in the prototype. More generally, if $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$7 models are colocated, worst-case host-driven admission and bookkeeping is $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$8, but this is dominated by GPU graph launches only at model-load time.

A key systems property is attention locality. All KV tensors remain in the KV-cache pool and are never migrated. FFN layers execute entirely in the weights pool and exchange only $$K_{\text{total}} = \max_t \sum_{i\in \text{Active}(t)} K_i(t).$$9 data per layer, independent of context length. This means that long-context requests do not incur any cost to fetch or push the growing KV tensors.

5. Evaluation, workloads, and empirical results

The evaluation uses five A100 GPUs (40 GB each) connected by NVLink. The colocated models are Qwen3-30B-A3B with 48 layers and 29 B FFN parameters, GLM-4.7-Flash with 47 layers and 28.3 B FFN parameters, and DeepSeek-V2-Lite with 27 layers and 14.9 B FFN parameters, for total weights of approximately 154 GB. Two baselines are reported: Static Partition, in which each model is pinned to MIG partitions, and kvcached (Chimera), which elastically pools weights and KV-cache in one memory pool (Ye et al., 23 Jun 2026).

On context-length scalability, the paper reports that as LongAlign context grows from 4 K to 10 K tokens, both Static Partition and kvcached hit OOM cliffs at approximately 3 RPS because their per-model KV quotas exhaust. Under the same $N$0 GB budget, CrossPool maintains approximately $N$1 higher maximum RPS at 10 K tokens by sharing a larger KV-cache pool.

On tail latency, the balanced Vicuna ShareGPT decode trace at 0.8 RPS/model yields the following P99 TBT reductions relative to kvcached:

Model	P99 TBT reduction
Qwen3-30B-A3B	$N$2
DeepSeek-V2-Lite	$N$3
GLM-4.7-Flash	$N$4

At 1.0 RPS/model, the reported speedups remain $N$5, $N$6, and $N$7 respectively. Static Partition is reported as slightly faster in the low-latency tail because it has zero sharing overhead, but it cannot admit long-context bursts without OOM.

The ablation at 0.5 RPS/model isolates the contribution of the layer-wise pipeline and persistent kernels:

Layer-wise pipeline / Persistent kernels	Throughput
Off / Off	55.4 token/s
Off / On	77.9 token/s
On / Off	60.8 token/s
On / On	111.4 token/s

The paper interprets these techniques as complementary: persistent kernels reduce host dispatch overhead, while the layer-wise scheduler keeps both pools busy. In the authors’ summary, the full system improves memory utilization by up to 30%, supports $N$8–$N$9 P99 TBT reductions, and serves $C$0 more long-context RPS under the same hardware budget.

6. Limitations, future directions, and terminological scope

The paper identifies several explicit caveats. First, hidden-state transfers from attention to FFN and from FFN back to attention remain on the critical path per layer; although these transfers are small relative to KV movement, they are not fully overlapped in the current design. Second, pipeline imbalance can arise when colocated models have widely different per-layer compute costs, causing the faster model to idle while waiting for its partner’s stage. Third, CrossPool assumes a low-concurrency cold regime; at very high concurrency, the benefits of sharing diminish and one may revert to per-model or per-batch data parallelism (Ye et al., 23 Jun 2026).

The proposed future work follows directly from those limitations. The paper suggests a finer-grain four-stage pipeline that separates attention compute, attention-to-FFN communication, FFN compute, and FFN-to-attention communication, with the goal of overlapping communication with both adjacent compute stages. It also suggests cluster-wide model grouping by attention and FFN cost profile to reduce imbalance within each weights/KV pair of pools, and adaptive resizing of the KV pool quantile $C$1 in response to diurnal or bursty variation rather than relying on a fixed offline trace.

A plausible broader implication is that CrossPool belongs to a wider movement toward memory-centric GPU systems. This suggests an affinity with work on CXL-backed shared memory pools for node-spanning GPU collectives, such as CCCL, although the mechanisms and target workloads differ (Xu et al., 25 Feb 2026). The term CrossPool also has an unrelated use in the literature on proof-of-work mining, where it denotes a strategy for diversifying computational power across mining pools and cryptocurrencies under a mean–variance objective; that usage is separate from the multi-LLM serving system described here (Chatzigiannis et al., 2019).