OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters

Abstract: Serving LLMs on a shared, heterogeneous GPU cluster requires users and operators to select the GPU type, tensor-parallel degree, and precision before committing valuable node-hours. Making these choices is challenging because effective throughput, launch-success rates, and cluster demand and utilization continuously fluctuate. Furthermore, static configuration recipes miss critical interactions: quantization effects depend heavily on the model family, key-value cache pressure creates size-by-precision trade-offs, and failure rates vary by more than twofold across different tensor-parallel degrees. Additionally, cluster resources are frequently constrained by unpredictable hardware failures. To address these challenges, we present \textbf{OmniPilot}, a launch advisor that predicts serving costs for feasible configurations and abstains when requests fall outside its measured support envelope. OmniPilot pairs a conformally calibrated quantile cost model (spanning eight serving targets) with an out-of-distribution (OOD) abstention layer. It ranks configurations using an economic utility metric calibrated to an operator's revealed preferences. In evaluations across 460 benchmark runs on A100, H100, and H200 hardware across four precisions, OmniPilot predicts aggregate throughput with a 6.2\% mean absolute percentage error (MAPE) and a log-space $R<sup>2=0.92$. The advisor achieves 95\% top-1 accuracy with a mean utility regret of just 0.003. When tested on an OOD holdout of unsupported cells, prediction error climbs to 24-46\% and conformal intervals cover 0 of 5 points; however, the abstention layer successfully flags all five as low-confidence. Over time, these OOD scenarios will be integrated into the training dataset to continuously expand the advisor's support envelope.

• The paper introduces OmniPilot, a calibrated launch advisor that predicts serving costs with explicit uncertainty estimates to guide optimal GPU configuration.
• It employs gradient-boosted quantile regressors and conformal calibration to model non-monotonic interactions among GPU types, TP degrees, and precision formats.
• Empirical results show 6.2% MAPE in throughput predictions and a mean utility regret of 0.003, validating its risk-aware abstention strategy.

OmniPilot: An Uncertainty-Aware Inference Advisor for Heterogeneous GPU Clusters

Problem Motivation and System Overview

Efficiently deploying LLM inference on heterogeneous, multi-tenant GPU clusters involves complex, high-stakes configuration choices—GPU type, tensor-parallel (TP) degree, and numerical precision—where suboptimal decisions cause significant performance and reliability penalties. Traditional schedulers and serving engines address only a subset of these challenges, focusing on either static resource matching (e.g., Slurm) or execution-level kernel optimizations (e.g., vLLM, TensorRT), lacking cluster-wide, macro-level job advising with quantitative uncertainty.

OmniPilot addresses this critical cluster orchestration gap by providing a calibrated launch advisor that predicts inference serving costs, abstains in unsupported regions of the configuration space, and exposes explicit uncertainty estimates. The system combines gradient-boosted quantile regressors with conformal calibration, economic utility modeling, and an out-of-distribution abstention mechanism to rank and recommend configuration arms in a utility-optimal, risk-aware manner.

Figure 1: OmniPilot pipeline, describing request intake, cost/uncertainty modeling, policy recommendation, telemetry-based retraining, and error calibration.

Decision Space Complexity

The inference launch decision space is inherently combinatorial, spanning GPU models (A100, H100, H200), TP degrees (1, 2, 4, 8), precision formats (bf16, FP8, AWQ, GPTQ), context window size, and concurrency. Empirical observations underscore that interactions across these axes are highly non-monotonic: e.g., model quantization does not universally yield faster inference, and memory or bandwidth contention can invert expected configuration orderings. The configuration feasibility, cost, and failure probability are highly context-dependent.

Figure 2: Inference launch decision space, with interacting axes leading to complex, non-monotonic configuration performance regions.

Architecture and Modeling Methodology

Cost Model and Feature Engineering

OmniPilot employs an ensemble of gradient-boosted decision trees trained with quantile regression loss across eight key serving targets (aggregate tokens/s, request throughput, TTFT, cold-start, KV-cache usage, average power, SM and tensor activity). The model is feature-rich, incorporating both continuous (log-transformed and raw numerics), categorical (model family, GPU kind, quantization method), and workload-supplied features (concurrency, context length), explicitly engineered for axis-by-axis generalization.

Notably, precision and quantization effects are handled using a hybrid scheme: effective bit-width encodes monotonic compression trends, while one-hot features capture off-trend anomalies such as the unexpectedly slow performance of AWQ in certain settings.

Figure 4: Quantization features capture non-monotonic performance; omitting them severely degrades predictive accuracy, especially for FP8.

Uncertainty Estimation and Abstention

Predictions yield conformalized quantile intervals, calibrated through split-conformal cross-validation grouped by workload "cell"—joint configurations of model, GPU, context, and concurrency. Out-of-distribution detection is realized via a support envelope check: any request falling outside the training regime receives a downgrading of its confidence, triggering abstention rather than relying on potentially misleading interval width.

Figure 5: Model decision flow integrates envelope checks, utility scoring, and selective benchmark probing, culminating in a ranked launch plan with confidence labeling.

Telemetry, Retraining, and Measurement Freshness

OmniPilot is tightly coupled with job- and cluster-level telemetry; all predictions, configuration recommendations, and observed launches feed into an automated retraining loop. Explicit metric freshness tracking ensures stale hardware/telemetry data is down-weighted or omitted, mitigating artifacts from job startup transients and ensuring consistent instrumentation.

Figure 3: Per-metric telemetry freshness state machine for prioritizing recent and relevant signals in the decision pipeline.

Numerical Evaluation and Ablative Analysis

Empirical evaluation on a fixed grid of 460 controlled inference jobs across A100, H100, H200, four precision types, and varying TP degrees demonstrates strong numerical results: aggregate throughput is predicted with 6.2% MAPE and log-$R^2=0.92$, while main latency and throughput targets maintain 6–12% MAPE. OmniPilot achieves 95% top-1 placement accuracy and mean utility regret of 0.003, vastly outperforming model-free and simple recipe baselines (size-threshold, throughput-max, nearest-neighbor policies).

Figure 6: Fixed-task learning curve shows rapid error reduction and $R^2$ gains saturate after ~150 labeled rows, indicating high data efficiency.

Ablation studies confirm the necessity of several key design elements:

• Quantization feature encoding: Omitting quantization nearly doubles throughput error in FP8 (3.88% → 7.36%).
• KV-cache utility penalty: Absence increases placement regret by 0.28 in affected cells.
• Launch-success priors: Removal leads to inappropriate over-recommendation of multi-GPU configurations, whose real-world cluster success rates are lower than the default flat prior.

Figure 7: Data efficiency is axis-driven—incremental coverage of new quantization or MoE axes yields substantial $R^2$ gains, independent of raw row count.

On held-out out-of-distribution cells (unseen model families/context/concurrency settings), prediction error grows substantially to 24–46% and conformal intervals fail to cover any of these points, but the abstention mechanism successfully flags 100% of such scenarios, validating the conservative, guardrail function of the envelope mechanism.

Figure 8: Cross-validated errors for all eight cost-model targets demonstrate high accuracy on decision-driving metrics.

Cluster-Scale Signaling and Practical Implications

Analysis of the 475,398-job Slurm cluster log provides two operationally relevant priors:

1. Single-GPU demand dominates: 81% of inference demand is on single-GPU configurations, with serving-specific workloads yielding 83% median GPU utilization.
2. Launch success is configuration-dependent: Success rate declines from 72% (H100×1) to 38% (H100×4), highlighting the importance of incorporating per-config priors into placement utility.

Scheduler-level metadata alone is insufficient for a cluster-wide throughput advisor, as it lacks direct insight into workload description, precision, or concurrency—necessitating serving-layer instrumentation and telemetry.

Figure 9: Cluster-wide analysis of job success rates, demand, and utilization stratified by configuration; advises operational regime for advisor design.

Theoretical and Practical Implications

OmniPilot establishes that macro-level LLM inference launch advising should be driven by calibrated, axis-generalized empirical modeling, not static recipes or rule-based heuristics. This paradigm enables efficient, low-regret resource allocation and enhances operational reliability via uncertainty abstention. The design emphasizes feature engineering discipline ("feature-before-data"), measurement-driven utility design (e.g., accounting for non-monotonic quantization and KV cache dynamics), and safety-first abstention in OOD regions.

Pragmatically, OmniPilot can be extended with minimal instrumentation cost to support broader classes of ML workloads (e.g., training, recovery) and richer multi-node and multi-cluster environments. The required adaptation is primarily in utility specification and collection of new feat/label targets.

This approach also surfaces a key theoretical result for uncertainty quantification in systems: simple interval width is a non-robust OOD signal, necessitating support-envelope reasoning and explicit abstention logic for safety-critical advising. Future utility gains will require posterior mean-aware benchmarking gates and deeper integration with engine-level runtime parameters.

Conclusion

OmniPilot provides a robust, uncertainty-aware macro-scheduler for LLM inference on heterogeneous GPU clusters, synergistically complementing execution-level serving systems. By framing hardware/precision choices as an economic optimization under calibrated uncertainty, the advisor achieves near-optimal placement accuracy, maintains safety through abstention, and yields operational insights for cluster-scale MLOps. Methodologically, the work demonstrates the primacy of axis-driven featurization, telemetry-integration, and uncertainty calibration for practical deployment of cluster resource advisors.

Future research should focus on generalizing the model to further workload paradigms, integrating posterior-updating gates, formalizing long-term drift/stability evaluation, and extending the system to multi-node orchestration scenarios.