CoCoServe: Elastic LLM Serving

• CoCoServe is an elastic LLM serving system that leverages module-level replication and migration to adapt dynamically to fluctuating traffic and resource constraints.
• It employs fine-grained scaling of Transformer modules—such as decoder layers, attention, and KV-cache blocks—to rebalance compute and memory load efficiently.
• Empirical evaluations show up to 75.8% latency reduction, 4× throughput improvement, and 46% cost savings compared to traditional instance-level scaling.

Searching arXiv for the specified CoCoServe paper and closely related serving systems to ground the article. CoCoServe is an elastic LLM serving system that performs dynamic and fine-grained scaling at the module level, rather than treating a full model instance as the atomic scaling unit. In CoCoServe, decoder layers, attention and FFN blocks, Q/K/V/O projections, and KV-cache segments can be replicated or migrated across GPUs to adapt to fluctuating traffic, limited GPU capacity, and multi-instance deployments. The system is designed to exploit idle memory and compute fragments, rebalance compute-heavy and memory-heavy components, and provide faster elasticity than instance-level replication or migration, which the paper characterizes as slow and memory-expensive (Wu et al., 24 Jul 2025).

1. Problem setting and design objective

CoCoServe is motivated by a tension between dynamic serving demand and static, coarse-grained deployments. Production LLM workloads exhibit highly variable request rates, burstiness, prompt length distributions, prefill-heavy bursts, decode phases that are more memory/KV-bound, and strong SLA/SLO constraints under tight cost budgets. Existing stacks usually scale at instance granularity by adding or removing full replicas, often already parallelized internally with tensor or pipeline parallelism. The paper argues that this granularity wastes resources at low load, produces performance cliffs at high load, and makes fast elasticity impractical because adjusting a single 13B instance requires loading tens of GB of weights, takes 8–25 s, and consumes 20–30 GB extra GPU memory during the operation (Wu et al., 24 Jul 2025).

The empirical motivation is concrete. For a LLaMA-13B instance on a single A100, the default configuration reaches latency spikes up to 37 s once RPS exceeds 50, with OOMs causing retries and SLO violations. The paper reports that migrating one decoder layer to another GPU keeps latency around 11.2 s in the 50–55 RPS range, a roughly 70% reduction without lowering throughput. This result is used to support the central claim that small changes in model placement can materially change latency and SLO behavior.

The design objective is therefore not merely to add more replicas, but to use the compositional structure of Transformer inference. In the paper’s characterization of LLaMA-13B, a decoder layer is approximately 317M parameters, about 605 MB in memory, and about 127.5 GFLOPs; attention is about 200 MB and 55.02 GFLOPs; FFN gate/up/down projections are about 135 MB and 36.24 GFLOPs; and KV cache is highly memory-intensive with size scaling linearly with batch size and sequence length. This heterogeneity is what CoCoServe turns into a scaling opportunity.

2. Module-level elasticity model

The core idea of CoCoServe is dynamic module-level scaling. A “module” can be a full decoder layer, a self-attention block, an FFN block, finer-grained Q/K/V/O projection matrices, or KV-cache segments associated with layers. CoCoServe exposes two operations over these modules: replication, which creates extra replicas of selected modules on additional GPUs and routes part of the batch through them, and migration, which moves modules to other GPUs to rebalance compute and memory load (Wu et al., 24 Jul 2025).

Replication is localized data parallelism. If a model instance sits on one GPU and other GPUs have free memory, CoCoServe can replicate a consecutive layer range onto the other devices. During a forward pass, the batch is split across the replicas for that segment, and outputs are gathered back in the original batch order. The paper emphasizes that overhead appears only at the boundaries of replicated contiguous regions; internal layers within a replicated segment execute locally. This is why continuity of replicated segments is a recurring optimization principle.

Migration is a more general placement adjustment. CoCoServe can move an entire decoder layer, only attention or FFN components, or KV-cache blocks. The paper recommends migrating entire layers where possible to reduce boundary communication and complexity, migrating KV cache to GPUs that are memory-rich but compute-poor, and migrating compute-heavy modules to GPUs that are compute-rich but memory-limited. This separation is central to CoCoServe’s claim that full-instance mobility is unnecessarily coarse.

The execution semantics are intended to remain transparent at the instance level. The scheduler routes requests to instances, not to individual modules, while module hooks perform pre-forward scatter, remote execution, and post-forward gather. The paper states that this preserves sequential consistency from the model’s perspective, apart from tiny numerical differences.

3. Analytical cost model and placement strategy

CoCoServe analyzes replication with a generalized Amdahl-style model. For module $i$, let $p_i$ be its parallelism degree, and let $P = [p_1,\ldots,p_n]$ denote the replication strategy. With hidden dimension $d$, final sequence length $l$, per-replica sub-batch sizes $bs_{ij}$, compute capacities $C_{ij}$, interconnect bandwidths $B_{ij}$, and $\delta$ denoting the number of communication events at non-consecutive boundaries, the paper defines compute time as

$$W(P) = \sum_{i=1}^{n} \max_{j=1}^{p_i} \frac{d^2 \times bs_{ij} \times l}{C_{ij}}$$

and communication time as

$p_i$0

The resulting speedup is

$p_i$1

where $p_i$2 is the sequential baseline (Wu et al., 24 Jul 2025).

In a homogeneous cluster with evenly split mini-batches, this reduces to

$p_i$3

where $p_i$4. The simplified form makes two placement heuristics explicit. First, larger $p_i$5 improves speedup only until communication begins to dominate. Second, fragmented replication increases $p_i$6, which increases $p_i$7 and reduces speedup. CoCoServe therefore prioritizes contiguous layer ranges.

The paper’s empirical replication study matches this interpretation. On LLaMA-13B over four A100s, 30 replicated layers with dop=2 yield a throughput improvement of 329.42% (4.3×) at 50 RPS, with latency staying under 5 s across all load levels and a 75.8% average latency reduction. With 20-layer replication, throughput at 50 RPS is +90.8% over baseline, while 15 replicated layers still produce a 45.07% throughput increase and 30.37% latency reduction. With fixed 20 replicated layers, dop=4 gives about 95.26% throughput increase and about 48.76% latency reduction at RPS < 30, but at RPS=50 the paper reports 163.68% throughput increase for dop=4, compared with 268.48% for 25-layer replication with dop=2. The authors use this to argue that deeper replicated segments can outperform simply adding more replicas in parallel at high load.

4. System architecture, runtime path, and auto-scaling

CoCoServe has a conventional control-plane / data-plane split. The data plane comprises a backend LLM engine such as HFT, vLLM, or xFormers-based engines; worker GPUs hosting one or more model instances; and instrumented modules that support remote execution via hooks. The control plane comprises a Monitor, an Auto-scaling Controller, and a Scheduler (Wu et al., 24 Jul 2025).

The Monitor collects per-GPU metrics through NVML, notably GPU utilization and memory utilization, and also collects per-instance metrics such as tokens/s, end-to-end latency, and SLO violation rate. The Controller maintains a placement state vector $p_i$8, periodically polls metrics, decides whether to scale up or scale down, invokes replication and migration primitives through hooks, and updates the scheduler after a placement change. The Scheduler continues to use engine-provided scheduling capabilities such as continuous batching and streaming, but now does so with updated performance and placement information.

Runtime execution proceeds layer by layer. For replicated layers, a pre-hook splits the batch and transfers slices to each replica; replicas process sub-batches in parallel; and a post-hook gathers outputs. For migrated modules, the pre-hook transfers inputs to the target device, the module computes there, and the post-hook returns outputs. The paper states that CoCoServe does not replace core engine kernels; it wraps existing backends with register_forward_pre_hook, register_forward_hook, and related mechanisms.

The auto-scaling logic has two thresholded paths. If resource vacancy exceeds $p_i$9, CoCoServe attempts scale-up via layer replication. It evaluates candidate replications using the simplified speedup formula, filters destination GPUs with GetEligibleNodes(G), and prioritizes candidate layers with SortCandidatesByContinuity(P, g_{dst}, max_replicas). If SLO violation exceeds $P = [p_1,\ldots,p_n]$0, it triggers scale-down in three stages: module migration, then replica eviction, and finally performance reduction through batch-size reduction and offloading. The ordering is deliberate: migration preserves capacity best, eviction reduces parallelism, and batch/offload adjustments are treated as the last resort.

A notable implementation claim is that module-level adjustments are much cheaper than instance-level adjustments. The paper reports that replicating or migrating 1 layer consumes about 1107 MB and takes about 0.25–0.30 s; 10 layers consume 6579 MB and take about 0.32–0.36 s; 20 layers consume 12659 MB and take about 0.34–0.38 s; and 40 layers consume 24819 MB and take about 0.81–0.89 s. Additional coordination between replicas after scaling costs only 39.1 ms.

5. Empirical behavior and reported gains

The evaluation uses a single server with 4× NVIDIA A100 40 GB PCIe, LLaMA2-13B and LLaMA2-70B, backend baselines Hugging Face Transformers 4.51 and vLLM 0.8.5, Alpaca prompts, maximum generation length 256 tokens, and synthetic loads from 3–50 RPS. Each RPS point is run five times. These experiments are reported for both single-instance and multi-instance settings (Wu et al., 24 Jul 2025).

In single-instance experiments on LLaMA-13B, CoCoServe reports, at 3–30 RPS, an average latency reduction of 56.89% and average throughput of 2.13× versus HFT, and an average latency reduction of 26.87% with average throughput 1.37× versus vLLM. At 31–50 RPS, CoCoServe reports latency 14–32% lower and throughput 1.16×–1.48× higher than vLLM on average. For LLaMA-70B, the paper reports, at low load, 75.22% average latency reduction and 4× throughput versus HFT, and 13.65% average latency reduction with 15.56% throughput improvement versus vLLM; at high load, 74.85% latency reduction and 3.9× throughput versus HFT, and 14.67% lower latency with 17.14% higher throughput versus vLLM.

The paper also reports memory-efficiency improvements. CoCoServe wastes up to 5.3 GB less memory than HFT and about 3.2 GB less than vLLM, reduces memory fragmentation by 3.12× versus HFT and 2.28× versus vLLM, and raises effective usable memory for the model to about 37.5 GB, versus about 32–34 GB in the baselines.

In multi-instance experiments on 4×A100s, CoCoServe deploys 2 LLaMA-13B instances using module-level scaling across GPUs. At low load, it is 14.23% lower in latency and 16.92% higher in throughput than HFT with 2 instances. At high load, the gap widens to 27.48% lower latency and 38.78% higher throughput. Compared with HFT with 4 instances, CoCoServe with 2 instances is slightly worse in raw performance, but the paper stresses the resource ratio: HFT(4) uses 119,573.34 MiB, HFT(2) uses 58,786.68 MiB, and CoCoServe(2) uses 64,015.44 MiB. This is summarized as using about 53.54% of the GPU memory of HFT(4), corresponding to about 46% cost reduction, while delivering about 90% of its throughput.

Robustness results are central to the system claim. Under high load with RPS > 50, HFT shows about 34% OOM error rate, whereas CoCoServe shows about 2% OOM, or 17× fewer OOMs. For SLO attainment, HFT begins dropping around 25 RPS and effectively fails entirely beyond 30 RPS; vLLM degrades significantly as RPS grows; CoCoServe maintains near-100% SLO attainment up to about 50 RPS. The abstract-level summary is therefore consistent with the full evaluation: CoCoServe reports 14%–75% lower latency, 1.16×–4× throughput on average across models and workloads, and 46% lower cost while maintaining availability.

6. Relation to adjacent systems, naming disambiguation, and limitations

The name “CoCoServe” is close to other recent serving systems but refers to a different problem class. The distinction is operationally important.

System	Optimization unit	Primary focus
CoCoServe	Module	Dynamic module-level replication and migration for LLM serving
ConServe	Conversation	Conversation-level disaggregated scheduling for agentic serving
CoServe	Expert	Dependency-aware scheduling and expert management for Collaboration-of-Experts inference

ConServe studies agentic multi-turn workloads and argues that the correct scheduling unit is the conversation, not the turn; it performs one cross-tier transition per conversation and pins the conversation to a decoder for its memory-bound tail (Ding et al., 1 Jun 2026). CoServe addresses Collaboration-of-Experts inference on heterogeneous CPU and GPU under limited memory by exploiting expert dependency, dependency-aware request scheduling, and dependency-aware expert management (Suo et al., 4 Mar 2025). CoCoServe, by contrast, assumes a conventional LLM instance structure and focuses on intra-model module placement.

The paper also states several limitations of CoCoServe itself. It requires dynamic, eager execution and is incompatible with static graph compilation or fused static kernels that cannot tolerate runtime dataflow changes. It assumes Transformer-like architectures with clearly separated modules such as decoder layers, attention, FFN, and KV cache. Its evaluation is on a single-node multi-GPU configuration with 4×A100, and distributed multi-node latency and network-bandwidth effects are not deeply explored. Multi-tenant SLAs involving priorities and fairness are outside scope. These constraints bound the generality of the results.

A plausible implication is that CoCoServe is most attractive where burst dynamics occur on timescales shorter than full-instance elasticity can track. That interpretation follows from the measured contrast between 8–25 s instance-level adjustments and 0.25–0.89 s module-level adjustments. The paper’s broader claim is therefore not that full-instance scaling becomes obsolete, but that the module is a useful elasticity granularity between kernels and whole replicas, and that this granularity is orthogonal to tensor parallelism, pipeline parallelism, data parallelism, KV-cache management, quantization, and offloading.