CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to $10.4\times$.

• The paper introduces a novel architecture that disaggregates static FFN weights from dynamic KV-cache to optimize GPU memory usage for cold MoE model serving.
• It employs trace-driven provisioning, virtualized paging, and layer-wise pipeline scheduling to significantly reduce decode latency and improve throughput.
• Experimental evaluations demonstrate sustained throughput and context-length scalability, reducing tail latency by up to 10.4× compared to baselines.

HybridCP: Efficient Multi-LLM Serving for Cold MoE Models via KV-Cache and Weight Disaggregation

Background: Memory Bottlenecks in Cold MoE Model Serving

The proliferation of LLM-based services has led to datacenter environments where a wide array of models, both hot (frequently requested) and cold (rarely requested), must be kept online concurrently. The gap between model weights (large, stable tensors) and KV-cache (dynamically allocated, request-dependent tensors) is particularly acute for sparse MoE architectures, which dominate frontier models due to their parameter efficiency. Empirical analyses reveal that over 90% of models are cold and poorly utilized, but their weights contribute to persistent GPU memory occupancy, constraining system scalability (Figure 1). Meanwhile, KV-cache exhibits highly bursty usage patterns, with low aggregate footprints most of the time (Figure 1, panel b).

Figure 1: Cold MoE models are underutilized, but their accumulated KV-cache usage varies dynamically under low request rate conditions.

A monolithic GPU memory pool is a systemic bottleneck: static model weights compete with ephemeral KV-cache, limiting the amount of memory that can be dynamically provisioned for active requests. The mismatch is especially severe for recent attention architectures (MQA, GQA, MLA), where KV-head counts can fall below the GPU replica count—resulting in DP attention configurations that handicraft the available KV-cache for low-concurrency workloads (Figure 2).

Figure 2: Aggregated memory pool demonstrates monolithic placement of weights and KV-cache for multi-LLM serving.

Disaggregating weights and KV-cache into separate pools promises improved KV-cache sharing across many cold models, but introduces design complexities: heterogeneous KV-cache planning, persistent communication overhead between pools, and advanced scheduling requirements.

HybridCP Architecture: KV-Cache and Weight Disaggregation

HybridCP concretizes this architectural separation by maintaining two distinct GPU memory pools: (1) a consolidated weights pool containing FFN weights from multiple cold MoE models; (2) a shared KV-cache pool that dynamically accommodates the KV demands of active requests (Figure 3).

Figure 3: HybridCP system architecture separates FFN weights in the weights pool and KV-cache with attention/local modules in the KV-cache pool.

During decoding, the inference path alternates between local computation in the KV-cache pool (attention and non-FFN modules), hidden-state transfer to the weights pool for FFN execution, and transfer back for the next layer. The communication boundary exchanges hidden states (bounded by batch and hidden dimension), sidestepping exponential growth with context length.

A trace-driven offline KV-cache planner computes the required page budget by sampling workload statistics. Virtualized KV-cache interfaces (via CUDA VMM APIs) decouple logical from physical page allocation, while pool-aware DP scheduling maximizes KV-cache exposure for each request. Admission control prevents over-subscription.

Key challenges addressed include:

• Heterogeneous KV-cache allocation: Pool sizing and parallelism are tailored to model-specific $\kappa(M)$, prompt/output/service time distributions, and desired provisioning quantile.
• Communication-efficient pipeline scheduling: Layer-wise pipelining overlaps computation and hidden-state transfers across batches, minimizing ping-pong overhead (Figure 4).
• GPU-resident control: Persistent kernels and control lowering keep high-frequency dispatch and communication control on-device, reducing CPU-GPU transitions (Figure 5).

  Figure 4: The layer-wise pipeline scheduler interleaves computation and communication across batches, enabling simultaneous attention and FFN execution.

  

Figure 5: Persistent kernels and control lowering shift frequent scheduling and inter-pool communication to GPUs.

Implementation Highlights

HybridCP extends SGLang runtime with disaggregated memory pools, proxy layers for seamless inter-pool hidden-state exchange (using NVSHMEM), KV-cache virtualization, and persistent graph-captured kernels for attention/FFN. Prefill and decode phases are separately handled, with decode-phase benchmarks focusing on scalability, TBT, and throughput.

Experimental Evaluation

Context-Length Scalability

Using LongAlign traces, HybridCP demonstrates sustained max aggregate request rate as context length increases—even when baselines suffer OOM or fatal failures due to limited per-model/reprlica KV-cache (Figure 6). Vertical drops in baseline curves mark capacity cliffs, while HybridCP maintains positive throughput owing to larger shared KV-cache pools.

Figure 6: Maximum aggregate request rate under increasing context length; HybridCP avoids early capacity drops by optimizing KV-cache pooling.

Decode Latency and Throughput

On balanced ShareGPT traces, HybridCP achieves competitive or substantially superior P95/P99 TBT compared to Chimera (kvcached) and Static Partition (Figure 7). Notably, HybridCP reduces P99 TBT by up to $10.4\times$, even as it delivers improved memory pooling for long-context requests.

Figure 7: Decode-side TBT on ShareGPT traces demonstrates HybridCP’s improved tail latency for multi-cold MoE serving.

Ablation studies further isolate throughput improvements from layer-wise pipeline scheduling and persistent GPU kernels, showing a $2.01\times$ aggregate throughput gain over unoptimized baselines.

Theoretical and Practical Implications

HybridCP systematically resolves the algorithm-system mismatch in multi-cold-model serving by decoupling static and transient memory objects at the hardware level. The architecture exposes larger KV-cache pools without incurring high tail latency or throughput penalties. It addresses DP attention’s intrinsic KV-cache visibility problem for head-limited models under cold concurrency, supporting more robust, flexible long-context inference.

Potential future directions include finer-grained pipelining (overlapping A-to-F and F-to-A communications with adjacent compute stages), compute profile-aware model grouping for pipeline balance, and integration with cluster-wide placement algorithms to optimize global throughput and latency.

The disaggregated memory pool approach is a foundational step toward scalable, cost-efficient, multi-LLM serving in heterogeneous datacenter deployments. It makes principled tradeoffs in communication and scheduling overhead to maximize memory utilization and context scalability, setting architectural precedents for future GPU serving frameworks.

Conclusion

HybridCP introduces a disaggregated memory pool architecture for cold MoE model serving, separating FFN weights and KV-cache into purpose-built pools. By leveraging trace-driven provisioning, virtualized paging, layer-wise pipeline scheduling, and persistent GPU kernels, HybridCP achieves strong context-length scalability and competitive decode latency, outperforming state-of-the-art multi-LLM systems on aggregate throughput and tail latency metrics. This work establishes a robust framework for addressing memory sharing and algorithm-system mismatches inherent in modern, sparse attention models, with clear implications for scalable LLM service infrastructure and algorithm co-design (2606.24506).