CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

Abstract: Online LLM serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for fine-grained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%-99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7\%--28.1\% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.

• The paper introduces CoCoScale, a novel layer-wise scaling approach that decouples elasticity from full-instance replication to minimize cold-start latency and resource over-provisioning.
• It empirically achieves a 97.9%–99.3% reduction in scaling latency and a 20.7%–45.7% decrease in average latency in production traces on Alibaba and Azure, ensuring 100% SLO attainment.
• The methodology integrates layer-wise data parallelism with an optimized scatter-then-exchange communication paradigm within a unified closed-loop control system to dynamically adapt to workload fluctuations.

Layer-wise Scaling for Elastic LLM Serving: A Detailed Analysis of CoCoScale

Motivation and Scalability Challenges in Online LLM Serving

Online LLM serving faces operational difficulties primarily due to highly skewed and bursty workloads. Conventional scaling approaches rely on instance-level granularity, necessitating full-model replica deployment and triggering substantial cold-start latency—often resulting in temporary SLO violations and resource over-provisioning due to inability to match fine-grained workload fluctuations. The paper identifies that the prevalence of workload skewness cannot be effectively mitigated by standard request migration schemes due to entrenched user-model affinity, resulting in bottlenecked resource allocation and response lag.

Figure 1: Instance-wise scaling induces latency spikes during cold starts, while CoCoScale leverages layer-wise scaling for rapid SLO recovery by replicating hot layers onto idle resources.

Figure 2: Production traces illustrate temporal lag and staircase provisioning patterns in coarse-grained scaling relative to traffic spikes.

CoCoScale Architecture: Fine-Grained Layer-wise Scaling

CoCoScale addresses these limitations by decoupling elasticity from rigid instance boundaries, enabling selective layer replication onto underutilized devices. The architecture implements layer-wise data parallelism, forming replicas of targeted decoder layers to dynamically expand the throughput of prioritized model instances without necessitating full-container initialization. This paradigm exploits communication-computation overlap, minimizing synchronization overhead.

Figure 3: Layer-wise replication for decoder layers, with yellow blocks run on the main device and green blocks as replicas on cold devices.

Optimized Data Transfer and Replication Paradigm

The system employs a Scatter-then-Exchange ring topology to maximize bandwidth utilization and minimize scaling latency. This enables aggressive partitioning and transmission of weights and dynamic KV caches, avoiding bottlenecks prevalent in single-source multicast strategies and eliminating conflict-induced resends.

Figure 4: Comparison of multicast and ring-based scatter-exchange data transfer strategies for weight and KV cache migration.

Analytical Model of Layer-wise Scaling

CoCoScale derives a comprehensive latency and speedup model, decomposing inference cost into computation and communication components. Speedup ratio $S(N,P)$ reflects the layer count replicated ($N$), parallelism degree ($P$), and the fixed communication overhead ($\beta(P)$), demonstrating that gains are strictly bounded by diminishing returns as $N$ approaches total layer count. The system generalizes to heterogeneous environments using per-layer configuration vectors, capturing quantization artifacts and straggler effects under uneven batch splits and hardware variability.

Ablation studies verify that increasing $N$ at fixed $P$ yields higher throughput up to a point, beyond which residual sequential blocks limit further speedup. They also show that higher $P$ values amplify communication overhead, especially under high concurrency, warranting adaptable, workload-dependent configurations.

Figure 5: End-to-end latency under varying number of replicated layers ($N$), showing optimal throughput improvements.

Figure 6: Latency surface in low-concurrency regime, with modest speedup achievable by moderate values of $N$ and $N$0.

Unified Scaling Algorithm and System Design

CoCoScale's control plane consists of Monitor, Controller, and Scheduler, operating in a closed-loop for continuous, feedback-driven elasticity. The Controller executes a unified scaling logic, seeking the nearest feasible configuration upon SLO deviation, ranking candidates by L1-norm transition distance and minimizing operational perturbation. Resource harvesting and instance prioritization is formalized as a Multi-Knapsack scheduling problem, efficiently approximated by a greedy heuristic.

Figure 7: Overview of CoCoScale closed-loop system: Monitor, Controller, and Scheduler enable fine-grained resource scavenging.

Empirical Evaluation and Numerical Results

CoCoScale demonstrates marked latency improvements and SLO compliance across production traces from Alibaba and Azure, with varying traffic intensities and burstiness. Empirically, CoCoScale achieves a 97.9%–99.3% reduction in scaling latency compared to instance-level scale-up, with sub-second scale-up times invariant to model size (P50 latency 1.04–1.52 s). Under moderate-to-intensive loads, average end-to-end latency is reduced by 20.7%–28.1% (Alibaba trace) and 43.3%–45.7% (Azure trace) relative to the strongest baseline (Alibaba Autoscaler). Crucially, SLO attainment reaches 100% for all model variants and workloads—a decisive contrast to static and threshold-based autoscaling (SLO attainment drops to 0.7%–21.1%).

Figure 8: Temporal latency comparison; CoCoScale maintains stable latency during demand surges, while baselines spike due to resource contention and slow reconfiguration.

Figure 9: CDF of CoCoScale scale-up latency; P50 remains stable across models, evidencing consistently low tail latency.

Practical and Theoretical Implications

CoCoScale establishes non-integer, fractional elasticity aligned to highly dynamic LLM serving environments, dramatically reducing cold-start latency, resource over-provisioning, and SLO violations. The model exposes layer-wise scaling as an operational axis orthogonal to tensor or pipeline parallelism, composable with existing static parallelism strategies. The fine-grained control allows systems to absorb bursts with minimal memory overhead and predictable speedup, bypassing architectural rigidity and reducing the need for complex instance redeployment. Compatibility with CUDA Graph execution further ensures maximal inference efficiency during dynamic scaling.

The system's formal model accurately characterizes marginal speedup saturation and communication penalties, providing a tractable decision framework for real-time configuration. The layer-wise strategy is extendable to disaggregated and multi-tenant environments, potentially integrating with advanced KV-cache management and distributed inference architectures for further scalability.

Conclusion

CoCoScale introduces a rigorous approach to online LLM serving by leveraging layer-wise scaling for fine-grained, elastic resource adaptation. The system significantly outperforms instance-level autoscaling baselines, attaining strict SLO compliance, reducing cold-start delays by up to 99.3%, and minimizing average latency under volatile and bursty workloads. Through analytical modeling and empirical validation, CoCoScale demonstrates that decoupling scaling logic from instance boundaries unlocks efficient, precise service alignment to dynamic demand, providing a foundation for broader advances in AI system-level scalability and resource optimization.

(2607.04181)