ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters

Abstract: As LLM services become widely adopted, the cost of GPU resources for serving these models in cloud environments has emerged as a critical concern. Spot instances offer up to 90% cost savings over on-demand instances, but their frequent interruptions and limited availability pose significant challenges for continuous LLM serving. GPU spot instances, in particular, exhibit lower and more volatile availability than CPU-based instances, making homogeneous clusters that depend on a single GPU type vulnerable to correlated failures. Heterogeneous clusters spanning multiple GPU types can address this by leveraging complementary availability patterns across diverse spot pools, yet existing LLM serving systems are designed for homogeneous environments and suffer from load imbalance when deployed on heterogeneous GPUs. This paper presents ShuntServe, a cost-efficient LLM serving system for heterogeneous spot GPU clusters. ShuntServe employs a roofline model-based analytical serving performance estimator and a dynamic programming-based model placement optimizer that jointly determines node configuration, parallelization strategy, and layer assignment to maximize throughput across heterogeneous GPUs. To enhance fault tolerance when using spot instances, ShuntServe combines output-preserving request migration with concurrent initialization via a shared tensor store, minimizing migration downtime by overlapping replacement node preparation with ongoing serving. Evaluation on Llama-3.1-70B and Qwen3-32B with a heterogeneous AWS cluster of L4, A10G, and L40S GPUs shows that ShuntServe achieves 1.42x and 1.35x higher throughput than state-of-the-art baselines and attains 31.9% and 31.2% cost efficiency improvements over on-demand instances for offline and online serving, respectively.

• The paper introduces a roofline-model-based performance estimator and DP+beam search optimizer to effectively partition LLMs across heterogeneous GPUs.
• It demonstrates fault-tolerant techniques such as output-preserving migration and concurrent initialization to minimize downtime during spot instance preemptions.
• Empirical results show up to 31% cost savings and improved throughput over conventional homogeneous and naive heterogeneous serving systems.

ShuntServe: Architecting Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters

Motivation and Challenges in Spot-Based Heterogeneous LLM Serving

LLM serving is characterized by substantial GPU demand and volatile load profiles, making cost-efficient and reliable hosting in the public cloud non-trivial. Spot GPU instances can reduce costs by up to 90% relative to on-demand pricing, but their unreliability—epitomized by abrupt preemptions and low/volatile availability, particularly for high-end devices—compromises service continuity. Homogeneous clusters are fragile under these conditions due to correlated failures across instance pools of the same type.

Figure 1: Time-series availability of different GPU instance types in an AWS region, illustrating volatility and scarcity, especially for high-end accelerators.

Consequently, exploiting heterogeneity—spanning diverse GPU types with complementary spot availability—offers resilience and economic efficiency at the cost of complexity in model partitioning, load balancing, and parallelism orchestration. However, existing LLM serving systems are not designed to manage the load, memory, and performance disparities across heterogeneous devices, resulting in severe performance pathologies when naively deployed.

The ShuntServe System Architecture

ShuntServe addresses these challenges via a systems approach to cost-effective and reliable LLM serving in heterogeneous spot GPU clusters. Its key architectural innovations are:

• A predictive, roofline-model-based serving performance estimator with one-time (device-specific) calibration, not requiring exhaustive per-configuration profiling.
• A DP-based model placement optimizer leveraging beam search to efficiently explore the exponential space of model partitionings, stages, and parallelization parameters.
• Integrated fault tolerance mechanisms for spot preemptions, including output-preserving request migration and concurrent initialization with shared tensor storage, minimizing observed downtime.

  Figure 2: ShuntServe system overview, highlighting analytical throughput estimation, optimized placement, and instance-agnostic spot management.

The architectural design ensures that each GPU instance is confined to a single pipeline path to contain the blast radius of preemptions and leverages both pipeline and tensor parallelism within and across devices to maximize the effective use of memory and compute.

Analytical Model-Based Throughput and Placement Optimization

The central technical contribution is the analytical performance estimator for heterogeneous LLM inference, rooted in the roofline model. Stage-wise latency is computed as the maximum of compute- and bandwidth-bound kernel execution times, calibrated by device-specific microbenchmarks to determine effective (vs. theoretical) FLOPS and memory bandwidth.

ShuntServe’s DP+beam search optimizer computes, for each placement candidate, the precise maximal feasible batch size satisfying all available device memory constraints, then evaluates throughput per dollar as the primary utility objective, soft-penalized for excessive latency. This permits dynamically balancing compute, memory, and cost across the heterogenous pool in a configuration-agnostic manner.

Figure 3: Schematic of asymmetric model partitioning (e.g., unequal layer assignments, per-stage TP) to mitigate pipeline bottlenecks under GPU heterogeneity.

A notable result is that for resource-constrained placements, ShuntServe adopts increasingly asymmetric model partitions, using per-stage TP to aggregate weaker GPUs, aligning stage durations and maximizing utilization. Bottlenecks in prefill (compute-bound) and decode (memory-bound) phases become decoupled in heterogenous topologies, underscoring the need for phase-aware placement optimization.

Figure 4: Decoupled critical paths for compute-bound and memory-bound phases, demonstrating shifting bottlenecks in heterogenous pipelines.

Fault Tolerance: Output-Preserving Migration and Zero-Downtime Reconfiguration

Spot preemptions can instantly disrupt long-running, stateful, pipelined LLM inference. ShuntServe integrates two main mechanisms for preemption resilience:

• Output-preserving request migration: Rather than transferring large and potentially inconsistent KV caches, ShuntServe triggers recomputation-based recovery on replacement pipelines. Empirical data demonstrates that up to moderate context lengths, recomputation is lower-latency than KV cache transfer, especially under spot constraints.

Figure 5: Empirical comparison of KV cache transfer vs recomputation latency across Llama 3 model scales; recomputation is often lower-latency in practical spot scenarios.

Figure 6: Output-preserving request migration schematic, showing the preservation and restoration of in-flight micro-batches after preemption.

• Concurrent initialization with shared tensor store: Decoupling model weights from engine lifecycle, a shared tensor store process enables overlapping pipeline (re-)initialization with ongoing inference. This mitigates the downtime associated with exclusive GPU memory and OOM errors typical of standard frameworks upon process duplication.

  Figure 7: Pipeline reconfiguration using concurrent initialization with shared tensor store to minimize downtime.

Empirical Evaluation: Throughput, Latency, and Cost-Efficiency

Extensive experimental results using Llama-3.1-70B and Qwen3-32B on diversified AWS GPU hardware validate the efficacy of ShuntServe. The analytical estimator achieves a 6.63% overall MAPE between predicted and measured inference latency for a wide matrix of parallelism and batch configurations, showing that per-device calibration suffices for robust placement decisions.

Figure 8: Analytical estimation vs. empirical measurement of inference latency across hardware, batch, and parallelism settings; high fidelity is observed.

ShuntServe demonstrates:

• Up to 1.42× throughput improvement over state-of-the-art homogeneous baseline and up to 1.35× improvement over heterogeneous baselines for saturated inference.
• For cost efficiency, 31.9% (offline) and 31.2% (online) savings over on-demand only configurations, with latency penalties contained to 2.6%–7.7% under even adverse interruption regimes.

Figure 9: Throughput comparison under optimal placements for Llama-3.1-70B and Qwen3-32B; ShuntServe shows superior throughput in both scenarios.

Figure 10: Online serving latency distributions (TTFT, TPOT) show ShuntServe retaining low-latency serving even as placement configurations scale with model size.

Figure 11: Throughput under spot interruptions, highlighting the gap in recovery efficiency among fault-tolerance strategies.

Figure 12: Trailing average and P90 tail latency over time during repeated spot interruptions, showing ShuntServe sharply curtails latency spikes.

Figure 13: Normalized cost efficiency (lower is better); ShuntServe consistently outperforms all baselines.

Constraints, Limitations, and Outlook

While ShuntServe’s analytical estimator avoids exhaustive profiling and generalizes to arbitrary hardware compositions, it inherits first-order limitations: nonlinearities due to scheduling, cache, and kernel overheads are not perfectly captured, and context-length variance is not explicitly modeled. For extreme context lengths, recomputation-based migration is suboptimal; hybrid recovery strategies may be needed.

Integration of ML-based nonlinearity correction on top of the roofline estimator, online adaptation to region/time-varying spot availability, and the adoption of phase-aware model placement strategies represent natural directions for further research.

Scalability to multi-region and cross-cloud deployments—addressing globally synchronized spot events—also remains an open challenge. Combining ShuntServe’s instance-level optimizer with multi-region placement and failover (cf. [skyserve-multi-region-inference]) is likely to further enhance both cost and reliability.

Conclusion

ShuntServe demonstrates that analytical modeling, algorithmic placement optimization, and tailored fault tolerance can unlock substantial cost efficiency and resilience for LLM inference over heterogeneous spot GPU clusters. Its design sets a new baseline for production-quality, cost-sensitive LLM serving frameworks in the face of the public cloud’s volatility and diversity, with significant implications for scalable, affordable AI deployment at both academic and industrial scale (2606.18600).