---
title: 'CoCoServe: Elastic LLM Serving'
url: https://www.emergentmind.com/topics/cocoserve
type: topic
---

# CoCoServe: Elastic LLM Serving

Searching arXiv for the specified CoCoServe paper and closely related serving systems to ground the article.
CoCoServe is an elastic LLM serving system that performs dynamic and fine-grained scaling at the module level, rather than treating a full model instance as the atomic scaling unit. In CoCoServe, decoder layers, attention and FFN blocks, Q/K/V/O projections, and KV-cache segments can be replicated or migrated across GPUs to adapt to fluctuating traffic, limited GPU capacity, and multi-instance deployments. The system is designed to exploit idle memory and compute fragments, rebalance compute-heavy and memory-heavy components, and provide faster elasticity than instance-level replication or migration, which the paper characterizes as slow and memory-expensive [2507.18006].

## 1. Problem setting and design objective

CoCoServe is motivated by a tension between dynamic serving demand and static, coarse-grained deployments. Production LLM workloads exhibit highly variable request rates, burstiness, prompt length distributions, prefill-heavy bursts, decode phases that are more memory/KV-bound, and strong SLA/SLO constraints under tight cost budgets. Existing stacks usually scale at instance granularity by adding or removing full replicas, often already parallelized internally with tensor or pipeline parallelism. The paper argues that this granularity wastes resources at low load, produces performance cliffs at high load, and makes fast elasticity impractical because adjusting a single 13B instance requires loading tens of GB of weights, takes **8–25 s**, and consumes **20–30 GB extra GPU memory** during the operation [2507.18006].

The empirical motivation is concrete. For a LLaMA-13B instance on a single A100, the default configuration reaches **latency spikes up to 37 s** once RPS exceeds 50, with OOMs causing retries and SLO violations. The paper reports that migrating **one decoder layer** to another GPU keeps latency around **11.2 s** in the **50–55 RPS** range, a roughly 70% reduction without lowering throughput. This result is used to support the central claim that small changes in model placement can materially change latency and SLO behavior.

The design objective is therefore not merely to add more replicas, but to use the compositional structure of Transformer inference. In the paper’s characterization of LLaMA-13B, a decoder layer is approximately **317M parameters**, about **605 MB** in memory, and about **127.5 GFLOPs**; attention is about **200 MB** and **55.02 GFLOPs**; FFN gate/up/down projections are about **135 MB** and **36.24 GFLOPs**; and KV cache is highly memory-intensive with size scaling linearly with batch size and sequence length. This heterogeneity is what CoCoServe turns into a scaling opportunity.

## 2. Module-level elasticity model

The core idea of CoCoServe is dynamic module-level scaling. A “module” can be a full decoder layer, a self-attention block, an FFN block, finer-grained Q/K/V/O projection matrices, or KV-cache segments associated with layers. CoCoServe exposes two operations over these modules: **replication**, which creates extra replicas of selected modules on additional GPUs and routes part of the batch through them, and **migration**, which moves modules to other GPUs to rebalance compute and memory load [2507.18006].

Replication is localized data parallelism. If a model instance sits on one GPU and other GPUs have free memory, CoCoServe can replicate a consecutive layer range onto the other devices. During a forward pass, the batch is split across the replicas for that segment, and outputs are gathered back in the original batch order. The paper emphasizes that overhead appears only at the boundaries of replicated contiguous regions; internal layers within a replicated segment execute locally. This is why continuity of replicated segments is a recurring optimization principle.

Migration is a more general placement adjustment. CoCoServe can move an entire decoder layer, only attention or FFN components, or KV-cache blocks. The paper recommends migrating entire layers where possible to reduce boundary communication and complexity, migrating KV cache to GPUs that are memory-rich but compute-poor, and migrating compute-heavy modules to GPUs that are compute-rich but memory-limited. This separation is central to CoCoServe’s claim that full-instance mobility is unnecessarily coarse.

The execution semantics are intended to remain transparent at the instance level. The scheduler routes requests to instances, not to individual modules, while module hooks perform pre-forward scatter, remote execution, and post-forward gather. The paper states that this preserves **sequential consistency** from the model’s perspective, apart from tiny numerical differences.

## 3. Analytical cost model and placement strategy

CoCoServe analyzes replication with a generalized Amdahl-style model. For module \(i\), let \(p_i\) be its parallelism degree, and let \(P = [p_1,\ldots,p_n]\) denote the replication strategy. With hidden dimension \(d\), final sequence length \(l\), per-replica sub-batch sizes \(bs_{ij}\), compute capacities \(C_{ij}\), interconnect bandwidths \(B_{ij}\), and \(\delta\) denoting the number of communication events at non-consecutive boundaries, the paper defines compute time as

\[
W(P) = \sum_{i=1}^{n} \max_{j=1}^{p_i} \frac{d^2 \times bs_{ij} \times l}{C_{ij}}
\]

and communication time as

\[
T(P) = \delta \times \sum_{i=1}^{n} \sum_{j=1}^{p_i - 1} \frac{d \times bs_{ij} \times l}{B_{ij}}.
\]

The resulting speedup is

\[
S(P) = \frac{W(P_0)}{W(P) + T(P)},
\]

where \(P_0 = [1,\ldots,1]\) is the sequential baseline [2507.18006].

In a homogeneous cluster with evenly split mini-batches, this reduces to

\[
S(P) = \frac{1}{\gamma + (1-\gamma)\frac{1}{n} \left\|\mathbf{1} \oslash P\right\|_1},
\]

where \(\gamma = \frac{\delta \cdot C}{d \cdot B}\). The simplified form makes two placement heuristics explicit. First, larger \(p_i\) improves speedup only until communication begins to dominate. Second, fragmented replication increases \(\delta\), which increases \(\gamma\) and reduces speedup. CoCoServe therefore prioritizes contiguous layer ranges.

The paper’s empirical replication study matches this interpretation. On LLaMA-13B over four A100s, **30 replicated layers** with **dop=2** yield a throughput improvement of **329.42% (4.3×)** at **50 RPS**, with latency staying under **5 s** across all load levels and a **75.8% average latency reduction**. With **20-layer replication**, throughput at **50 RPS** is **+90.8%** over baseline, while **15 replicated layers** still produce a **45.07% throughput increase** and **30.37% latency reduction**. With fixed **20 replicated layers**, **dop=4** gives about **95.26% throughput increase** and about **48.76% latency reduction** at **RPS < 30**, but at **RPS=50** the paper reports **163.68%** throughput increase for **dop=4**, compared with **268.48%** for **25-layer replication with dop=2**. The authors use this to argue that deeper replicated segments can outperform simply adding more replicas in parallel at high load.

## 4. System architecture, runtime path, and auto-scaling

CoCoServe has a conventional control-plane / data-plane split. The data plane comprises a backend LLM engine such as HFT, vLLM, or xFormers-based engines; worker GPUs hosting one or more model instances; and instrumented modules that support remote execution via hooks. The control plane comprises a **Monitor**, an **Auto-scaling Controller**, and a **Scheduler** [2507.18006].

The Monitor collects per-GPU metrics through NVML, notably GPU utilization and memory utilization, and also collects per-instance metrics such as tokens/s, end-to-end latency, and SLO violation rate. The Controller maintains a placement state vector \(P\), periodically polls metrics, decides whether to scale up or scale down, invokes replication and migration primitives through hooks, and updates the scheduler after a placement change. The Scheduler continues to use engine-provided scheduling capabilities such as continuous batching and streaming, but now does so with updated performance and placement information.

Runtime execution proceeds layer by layer. For replicated layers, a pre-hook splits the batch and transfers slices to each replica; replicas process sub-batches in parallel; and a post-hook gathers outputs. For migrated modules, the pre-hook transfers inputs to the target device, the module computes there, and the post-hook returns outputs. The paper states that CoCoServe does not replace core engine kernels; it wraps existing backends with `register_forward_pre_hook`, `register_forward_hook`, and related mechanisms.

The auto-scaling logic has two thresholded paths. If resource vacancy exceeds **\(T_{up}\)**, CoCoServe attempts **scale-up** via layer replication. It evaluates candidate replications using the simplified speedup formula, filters destination GPUs with `GetEligibleNodes(G)`, and prioritizes candidate layers with `SortCandidatesByContinuity(P, g_{dst}, max_replicas)`. If SLO violation exceeds **\(T_{down}\)**, it triggers **scale-down** in three stages: **module migration**, then **replica eviction**, and finally **performance reduction** through batch-size reduction and offloading. The ordering is deliberate: migration preserves capacity best, eviction reduces parallelism, and batch/offload adjustments are treated as the last resort.

A notable implementation claim is that module-level adjustments are much cheaper than instance-level adjustments. The paper reports that replicating or migrating **1 layer** consumes about **1107 MB** and takes about **0.25–0.30 s**; **10 layers** consume **6579 MB** and take about **0.32–0.36 s**; **20 layers** consume **12659 MB** and take about **0.34–0.38 s**; and **40 layers** consume **24819 MB** and take about **0.81–0.89 s**. Additional coordination between replicas after scaling costs only **39.1 ms**.

## 5. Empirical behavior and reported gains

The evaluation uses a single server with **4× NVIDIA A100 40 GB PCIe**, **LLaMA2-13B** and **LLaMA2-70B**, backend baselines **Hugging Face Transformers 4.51** and **vLLM 0.8.5**, Alpaca prompts, maximum generation length **256 tokens**, and synthetic loads from **3–50 RPS**. Each RPS point is run five times. These experiments are reported for both single-instance and multi-instance settings [2507.18006].

In single-instance experiments on **LLaMA-13B**, CoCoServe reports, at **3–30 RPS**, an average latency reduction of **56.89%** and average throughput of **2.13×** versus HFT, and an average latency reduction of **26.87%** with average throughput **1.37×** versus vLLM. At **31–50 RPS**, CoCoServe reports latency **14–32% lower** and throughput **1.16×–1.48× higher** than vLLM on average. For **LLaMA-70B**, the paper reports, at low load, **75.22%** average latency reduction and **4×** throughput versus HFT, and **13.65%** average latency reduction with **15.56%** throughput improvement versus vLLM; at high load, **74.85%** latency reduction and **3.9×** throughput versus HFT, and **14.67%** lower latency with **17.14%** higher throughput versus vLLM.

The paper also reports memory-efficiency improvements. CoCoServe wastes up to **5.3 GB** less memory than HFT and about **3.2 GB** less than vLLM, reduces memory fragmentation by **3.12×** versus HFT and **2.28×** versus vLLM, and raises effective usable memory for the model to about **37.5 GB**, versus about **32–34 GB** in the baselines.

In multi-instance experiments on **4×A100s**, CoCoServe deploys **2 LLaMA-13B instances** using module-level scaling across GPUs. At low load, it is **14.23% lower** in latency and **16.92% higher** in throughput than **HFT with 2 instances**. At high load, the gap widens to **27.48% lower** latency and **38.78% higher** throughput. Compared with **HFT with 4 instances**, CoCoServe with 2 instances is slightly worse in raw performance, but the paper stresses the resource ratio: **HFT(4)** uses **119,573.34 MiB**, **HFT(2)** uses **58,786.68 MiB**, and **CoCoServe(2)** uses **64,015.44 MiB**. This is summarized as using about **53.54% of the GPU memory** of **HFT(4)**, corresponding to about **46% cost reduction**, while delivering about **90% of its throughput**.

Robustness results are central to the system claim. Under high load with **RPS > 50**, HFT shows about **34% OOM error rate**, whereas CoCoServe shows about **2% OOM**, or **17× fewer OOMs**. For SLO attainment, HFT begins dropping around **25 RPS** and effectively fails entirely beyond **30 RPS**; vLLM degrades significantly as RPS grows; CoCoServe maintains near-100% SLO attainment up to about **50 RPS**. The abstract-level summary is therefore consistent with the full evaluation: CoCoServe reports **14%–75%** lower latency, **1.16×–4×** throughput on average across models and workloads, and **46%** lower cost while maintaining availability.

## 6. Relation to adjacent systems, naming disambiguation, and limitations

The name “CoCoServe” is close to other recent serving systems but refers to a different problem class. The distinction is operationally important.

| System | Optimization unit | Primary focus |
|---|---|---|
| **CoCoServe** | Module | Dynamic module-level replication and migration for LLM serving |
| **ConServe** | Conversation | Conversation-level disaggregated scheduling for agentic serving |
| **CoServe** | Expert | Dependency-aware scheduling and expert management for Collaboration-of-Experts inference |

ConServe studies agentic multi-turn workloads and argues that the correct scheduling unit is the **conversation**, not the turn; it performs one cross-tier transition per conversation and pins the conversation to a decoder for its memory-bound tail [2606.01839]. CoServe addresses **Collaboration-of-Experts** inference on heterogeneous CPU and GPU under limited memory by exploiting expert dependency, dependency-aware request scheduling, and dependency-aware expert management [2503.02354]. CoCoServe, by contrast, assumes a conventional LLM instance structure and focuses on **intra-model module placement**.

The paper also states several limitations of CoCoServe itself. It requires dynamic, eager execution and is incompatible with static graph compilation or fused static kernels that cannot tolerate runtime dataflow changes. It assumes Transformer-like architectures with clearly separated modules such as decoder layers, attention, FFN, and KV cache. Its evaluation is on a **single-node multi-GPU** configuration with **4×A100**, and distributed multi-node latency and network-bandwidth effects are not deeply explored. Multi-tenant SLAs involving priorities and fairness are outside scope. These constraints bound the generality of the results.

A plausible implication is that CoCoServe is most attractive where burst dynamics occur on timescales shorter than full-instance elasticity can track. That interpretation follows from the measured contrast between **8–25 s** instance-level adjustments and **0.25–0.89 s** module-level adjustments. The paper’s broader claim is therefore not that full-instance scaling becomes obsolete, but that the **module** is a useful elasticity granularity between kernels and whole replicas, and that this granularity is orthogonal to tensor parallelism, pipeline parallelism, data parallelism, KV-cache management, quantization, and offloading.

Source: https://www.emergentmind.com/topics/cocoserve