OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

Abstract: Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving LLMs. Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

• The paper introduces operator-level provisioning and autoscaling that models heterogeneous LLM operators with queueing analysis, resource profiling, and interference-aware placement to optimize GPU allocation under TTFT and TBT SLOs.
• OpScale reduces average GPU use by 20.1% for dense models and 35.7% for MoE models, while achieving about 98% SLO attainment and scaling operators in as little as 0.03 seconds on average.
• The system delivers its strongest benefits for large or MoE models, moderate sequence lengths, and prefill-heavy workloads, but gains diminish at low request rates, very long contexts, and highly fused runtimes.

Motivation and problem statement

Serving LLMs in cloud GPU clusters requires meeting strict latency SLOs—time-to-first-token (TTFT) and time-between-tokens (TBT)—while minimizing GPU footprint and power. The dominant mechanism for this is SLO-aware autoscaling, and existing state-of-the-art systems (DynamoLLM, AIBrix, the vLLM Production Stack) all share a common design assumption: the unit of scaling is the entire model replica. OpScale challenges this assumption, arguing that monolithic model-level scaling is both too slow and too coarse to match the dynamics of production inference traffic (2608.13499).

The paper grounds this claim in two mismatches. First, a speed mismatch: loading a full 70B model onto new GPUs takes on average more than ten seconds even with optimized startup methods, while production traffic exhibits bursts where peak demand exceeds minimum demand by 2× (Chat) to 5× (Code) within a ten-second window. Slower autoscalers therefore miss progressively more of these fluctuations, forcing either SLO violations or standing over-provisioning—the paper cites utilization figures of only 50% and 39% for text and multimodal workloads provisioned at P95 demand. Second, a granularity mismatch: an LLM is a dataflow graph of heterogeneous operators (attention, linear projections, normalization, MoE experts), and uniform scaling of all operators wastes capacity on non-bottlenecks.

Operator characterization

The paper's empirical foundation is a systematic profiling study across dense models (Qwen2-7B, Llama3-8B, Qwen2.5-VL-32B) and MoE models (Qwen2-57B-A14B, Mixtral-8x7B), using CUDA event profiling and instrumented vLLM layerwise profiling on A100s. Three findings drive the design:

• Compute sensitivity varies widely across operators and across models. Heavy per-token arithmetic operators (MLPs, fused MoE kernels) scale near-linearly or super-linearly with batch size; lightweight operators (layer norms, element-wise activations) scale sub-linearly; attention grows quadratically in sequence length ($\mathcal{O}(L^2 b)$). Consequently, linear operators dominate compute at short sequences while attention dominates at long contexts, and the bottleneck shifts dynamically with workload.
• Memory sensitivity varies less, bounded by approximately linear growth under FlashAttention, with attention still the dominant contributor.
• Sensitivity to SM allocation is phase-dependent: for long prefill sequences, reducing SM allocation sharply increases latency of compute-intensive operators, whereas decode-phase curves are nearly flat because SM utilization is too low to saturate resources—making spatial sharing via CUDA Green Contexts attractive for short sequences.

A cross-dimensional analysis further shows that operators can be compute-heavy but memory-light (attention), memory-heavy but compute-light (RMSNorm), or heavy/light in both, so provisioning must jointly consider both resource dimensions.

Analytical formulation and benefit analysis

OpScale formalizes operator-level autoscaling as a multi-objective optimization over an operator DAG: given arrival rate $\lambda$ and sequence-length distribution, choose per-operator batch size $B_v$, replica count $R_v$, tensor-parallel degree $P_v$, device assignment $A_v$, and SM allocation $S_v$ to minimize total shard-replicas subject to TTFT/TBT SLOs. Per-operator waiting time is modeled as an M/M/$R_v$ queue with Erlang-C delays, and placements must respect per-device memory and SM capacity constraints.

Exhaustive search over this formulation serves as an offline oracle quantifying the opportunity. The results delineate where operator-level scaling pays off:

Condition	Peak savings vs. model-level
Sequence length ~4K	~30% GPUs (dense), ~40% (MoE); up to 25% energy
Long sequences (32K)	>60% memory savings
QPS ≈ 40	~30% GPU savings; >50% memory at 100 QPS
Model size scaling (Qwen2 family)	Up to ~50% energy/memory at 72B

Two boundary conditions are stated plainly: savings are negligible at low QPS (<20) where no scaling is needed, and they drop beyond 4K–8K sequences as compute saturates SM cores and limits sharing. Larger models benefit more because their operators exhibit greater heterogeneity; small models show modest gains.

System design

Realizing the paradigm requires taming two space explosions. Naive exhaustive profiling spans roughly $10^7$ configurations per operator (batch size × sequence length × SM fraction), which would take weeks even for a 0.5B model; and optimal provisioning is NP-hard integer non-linear programming, with brute-force solvers taking minutes where decisions are needed within seconds. OpScale addresses these with three planes:

• Operator Profiler uses sparse random sampling with piecewise interpolation over $(B, L)$, exploiting monotonic sensitivity trends and structural equivalence of operators across layers and model families. Multi-billion-parameter models profile in under an hour, and profiles largely transfer across model generations that share kernel implementations.
• Operator Provisioning runs a millisecond-scale greedy heuristic: initialize from inherited parallelism, pick locally latency-optimal $\lambda$0 pairs, then iteratively upscale the critical-path bottleneck or downscale top non-critical operators. Its resource cost stays within 8% of the brute-force oracle.
• Operator Placement treats packing as multi-dimensional bin-packing with a locality-aware Best-Fit Decreasing heuristic. Colocation interference is captured by an empirical factor $\lambda$1—the ratio of co-execution to isolated latency—profiled offline into an O(1)-lookup table at 5% SM granularity, with three-way colocation approximated by multiplying pairwise factors. Candidate placements are validated against the end-to-end SLO before acceptance, and tie-breaking follows the communication hierarchy (intra-device, NVLink/NVL domain, InfiniBand).
• Execution Plane extends nano-vLLM (~17K lines of Python) with dynamic operator replication via forward pre-hooks for late binding, an ElasticBlockManager (built on kvcached) for runtime memory allocation, a GlobalStreamPool bounding stream overhead to $\lambda$2 regardless of depth, cross-device pipelining of transfers, and weighted shortest-queue request dispatch.

Evaluation results

Evaluated on clusters of up to 40 A100s and 24 GB200s against DynamoLLM, AIBrix, and Production Stack—all ported onto the same nano-vLLM backend so that differences isolate scaling policy rather than runtime—over production traces totaling 929K requests and 1.5B tokens:

• Dynamic autoscaling: OpScale averages 7.1 GPUs for Qwen2-7B versus 11.2–14.3 for baselines (a 35–50% reduction) while achieving 98.4% SLO attainment versus 88–95%. For the MoE model it averages 10.8 GPUs versus 17.3–23.1, with 98.1% attainment versus 84.2–97%.
• Scale-up latency: sub-second elasticity is the headline result. Model-level scaling averages 10.68 s; OpScale scales one operator in 0.03 s on average (P99 0.10 s) and all operators simultaneously in 0.33 s (P99 0.45 s)—up to two orders of magnitude faster. The paper notes fairly that warm standby replicas could match this latency but merely convert it into persistent over-provisioning.
• Cost at fixed SLO: 20.1% fewer GPUs on average for dense and 35.7% for MoE at 1K sequences; peak savings of 36.3% at 4K dropping to 22.1% at 8K, consistent with the analytical model. Cluster power drops 14–28% at high load despite higher per-GPU power draw (282 W vs. 245 W), and OpScale outperforms power-aware baselines (DynamoLLM+DVFS, $\lambda$3-Serve++) by 16–20% at P90 power.
• Static throughput: at fixed budgets up to 40 GPUs, peak token throughput improves 3–38% for dense and up to 44% for MoE. Against Attn-FFN disaggregation, OpScale reduces GPU demand by up to 33% on A100s (1.7× throughput gain), widening to a 52% GPU reduction on GB200's NVL domain versus 38% for the baseline—indicating that finer-grained elasticity exploits faster interconnects better than block-level decomposition.
• Model fidelity: operator sensitivity prediction averages 7% error (P90 15%), SM contention slowdown prediction 5% (P90 9.4%), and queueing-model latency prediction 0.8% (P90 1.9%). Control-plane planning takes 2.6 ms median for Qwen2-7B (4.4 ms for the 57B MoE), leaving ~370× headroom in a one-second interval; execution-plane overhead is 0.3%.

Limitations and open questions

The paper is explicit about several boundaries. First, op-level autoscaling yields negligible benefit at low QPS and diminishing returns at very long sequences or relaxed SLOs, where its advantages shrink. Second, the approach presumes operator-level decomposition is available: systems pursuing ultra-low latency through megakernels that fuse all operators leave little room for this technique. Third, op-level resharding (changing tensor parallelism online) incurs 11× higher overhead than replica scaling (P99 1.15 s), so horizontal scaling remains the preferred action. Fourth, the evaluation targets single-model serving; extension to multi-tenant, multi-model settings requires tenant fairness policies and cross-model interference modeling that remain future work. Fifth, the interference model approximates three-way colocation by multiplying pairwise factors and operates at SM-allocation granularity—kernel-level spatial-temporal multiplexing could improve accuracy but is unexplored here. Finally, prefill stages capture most of the benefit (up to 2–3× greater savings than decode), raising the open question of how op-level scaling composes with prefill-decode disaggregation architectures.

Conclusion

OpScale demonstrates that the operator, not the model replica, is a viable and practical unit of autoscaling for LLM serving. Grounded in a characterization of operator heterogeneity and an Erlang-C queueing formulation, its greedy provisioning and interference-aware placement achieve near-oracle resource efficiency at millisecond decision cost and sub-second actuation. On production traces, the system meets strict TTFT/TBT SLOs with up to 36.3% fewer GPUs and 28% less power, or delivers up to 44% higher throughput at fixed cost—with gains concentrated in MoE models, larger models, moderate sequence lengths, and prefill-dominated workloads.