Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

Abstract: The usage of LLMs has grown increasingly fragmented, with no single model dominating. Meanwhile, cloud providers offer a wide range of mid-tier and older-generation GPUs that enjoy better availability and deliver comparable performance per dollar to top-tier hardware. To efficiently harness these heterogeneous resources for serving multiple LLMs concurrently, we introduce Coral, an adaptive heterogeneity-aware multi-LLM serving system. The key idea behind Coral is to jointly optimize resource allocation and the serving strategy of each model replica across all models. To keep pace with shifting throughput demand and resource availability, Coral applies a lossless two-stage decomposition that preserves joint optimality while cutting online solve time from hours to tens of seconds. Our evaluation across 6 models and 20 GPU configurations shows that Coral reduces serving cost by up to 2.79$\times$ over the best baseline, and delivers up to 2.39$\times$ higher goodput under scarce resource availability.

• The paper introduces a joint resource allocation and model placement strategy using a lossless two-stage decomposition that enables cost-efficient multi-LLM serving on heterogeneous GPUs.
• It leverages precomputed Serving Templates and an optimized online ILP to dynamically adapt to workload shifts while meeting per-model latency SLOs.
• Empirical evaluation demonstrates up to 2.79× cost reduction and 2.39× higher goodput compared to baselines, validating the hybrid pipeline/data-parallel approach.

Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs with Coral

Problem Formulation and Motivation

Coral addresses the challenge of cost-efficient, adaptive serving of multiple LLMs over heterogeneous cloud GPU resources. The contemporary LLM ecosystem is highly fragmented; deployments must support numerous models spanning diverse architectures and sizes, each with specific throughput demands and latency SLOs. At the same time, public clouds offer a range of GPU generations where mid-tier and older cards exhibit superior cost efficiency per computation or memory unit compared to cutting-edge hardware. The central technical problem is to optimally allocate a heterogeneous set of GPU nodes and decide the model placement strategy for each LLM instance, such that all models’ demands and per-model latency SLOs are met at minimum total cost, under rapidly shifting workload intensities, pricing, and resource availabilities.

This constitutes a joint optimization over two tightly coupled, exponentially large spaces: resource allocation (which nodes to provision) and model placement (how to distribute LLM computation over them). Existing systems solve only parts of this problem—optimizing resource allocation or placement in isolation, or lacking support for multi-model serving under heterogeneity—and are thus suboptimal for the general case.

System Design: Two-Stage Lossless Decomposition

Coral introduces a theoretically justified, lossless two-stage decomposition that decouples model placement from resource allocation by leveraging a key observation: for a fixed model and latency SLO, the throughput-optimal placement on any node set depends solely on the node combination, irrespective of allocations made for other models. Thus, model placement optimization can be moved offline, producing reusable “Serving Templates.” The online problem then reduces to selecting an optimal mixture of templates to instantiate, tailored to real-time workload and resource conditions.

Figure 2: Two-stage Coral workflow: offline Serving Template generation, and periodic online allocation/placement selection given live workload and resource state.

The offline phase constructs, for each (model, SLO) pair and a pruned, cost-relevant subset of node combinations, a throughput-optimal hybrid pipeline/data-parallel device assignment (the Serving Template). The per-template optimization is solved as an ILP considering pipeline stage partitioning and heterogeneous nodes, subject to per-stage latency budgets.

Figure 4: Structure of a Serving Template for a model across heterogeneous nodes and pipeline stages, combining pipeline and data-parallel replication per stage.

Online, the Coral allocation ILP picks, for each model, which templates to instantiate, and distributes instances across regions, such that all throughput/SLO/per-model requirements are met, cost is minimized, and real-time availability constraints are obeyed. The online ILP is large but highly sparse and solved efficiently (sub-minute wall time), enabling rapid elastic reconfiguration.

Runtime Architecture

The runtime comprises a central coordinator (resource allocator and request router), and pools of prefill and decode Serving Instances. Each Serving Instance follows the precomputed pipeline and device allocations, leveraging a custom pipeline-parallel communication system atop ZeroMQ and NCCL. Request routing is done proportional to measured template throughput, enabling high resource utilization. The system supports both prefill-decode disaggregation and classic inference.

Figure 1: Coral runtime system: coordinator, Serving Instances, per-request routing, and pipeline execution with direct KV cache handoff.

A high-fidelity, event-driven simulator is developed to enable large-scale experiments, fitted to profiling traces for all supported GPU types, achieving average latency deviations under 7% relative to hardware deployments.

Figure 3: Simulated prefill/decode latency distributions closely match real-system CDFs, validating evaluation methodology.

Empirical Evaluation

Cost Efficiency: Across experiments involving up to 6 models and 20 GPU types, Coral achieves substantial serving cost reductions compared to previous approaches—up to $2.79\times$ over the best baseline—while always meeting per-model latency SLOs. This is attributed primarily to intra-replica heterogeneity (mixing different GPU types inside a single pipeline/data parallel group) rather than inter-replica heterogeneity.

Figure 5: Cost comparison for Qwen-3 235B prefill: mixed L40S/H100 pipeline outperforms any pure H100 setup under SLO constraint.

Goodput Under Scarce Resources: When GPU availability is highly constrained, Coral maintains up to $2.39\times$ higher goodput compared to the best prior system. Baselines that optimize per-model in isolation or restrict themselves to homogeneous resource groupings both fail to satisfy aggregate SLO-constrained throughput.

Figure 6: Per-model decode goodput across epochs under resource scarcity, highlighting Coral’s ability to maintain SLO compliance and maximize aggregate throughput.

Robustness to Demand Skew: Coral’s cost advantage increases as more request load targets large models, since larger models expose more opportunities for resource-efficient intra-replica heterogeneity. When large models dominate, Coral’s cost reduction exceeds $3\times$ in extended setups.

Figure 9: Core setup cluster composition used for scalability and heterogeneity studies.

Comparison with Helix: On the previously state-of-the-art single-model heterogeneous placement ILP system, Coral delivers improved throughput (by 26%) and reduces cost (by 21%) via its multi-instance resource decomposition and hybrid pipeline/data parallel templates, even when constrained to the same node pool.

Serving Template Generation Sensitivity: Coral’s offline search rapidly saturates cost efficiency as node count and aggregate memory caps are increased beyond moderate values (e.g., 6 nodes, $12\times$ model size), while the computational burden grows exponentially. Pruning the templates to this regime preserves nearly all cost efficiency gains.

Figure 8: Cost efficiency plateaus with relatively small Serving Template enumeration caps, justifying tractable subset selection.

Key Contributions

• First joint resource allocation and placement system for multi-LLM, heterogeneous SLO-constrained serving: Previous works (e.g., SkyServe, Helix) tackle only one dimension. Coral’s joint formulation unlocks new feasible/efficient deployment regimes.
• Lossless two-stage decomposition via Serving Templates: This enables scalable, real-time re-optimization for large clusters.
• Hybrid intra-replica heterogeneity with hybrid DP/PP: Empirically validated as essential for optimally filling the cost-throughput spectrum and contending with scarce resources.
• Full runtime implementation and system-calibrated simulator: Demonstrated scalability, fidelity, and significant cost/goodput benefits across realistic scaling regimes.

Implications and Future Directions

Practically, Coral allows LLM providers to maximize utilization of fragmented, dynamic, heterogeneous cloud GPU pools without sacrificing SLO guarantees or incurring combinatorially explosive runtime costs. Theoretically, it validates the efficacy of hybrid pipeline/data-parallel decomposition for resource-efficient inference at the multi-model scale under latency constraints.

Future research directions suggested by the work include extending the joint optimization to include energy-aware or carbon-constrained objectives (important for sustainable large-scale AI serving), further integration with runtime sharing/scheduling frameworks (e.g., Prism, Aegaeon, FlexPipe), and exploring the incorporation of online model selection or mixture-of-experts routing into the same serving optimization loop. Moreover, advances in heterogeneity-tolerant communication primitives or elastic inference engine design may yield even greater gains when integrated with Coral’s framework.

Conclusion

Coral advances the systems state-of-the-art in multi-LLM serving on heterogeneous cloud resources by introducing a theoretically lossless, scalable, and empirically robust solution to the joint resource allocation and model placement problem under latency SLOs. Its hybrid offline/online optimization, judicious use of intra-replica heterogeneity, and demonstrated cost/goodput advantages position it as the reference architecture for future heterogeneous LLM serving systems.