WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

Abstract: LLM inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce \textit{WattGPU}, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of $\leq3.4\%$ for offline and $\leq13.5\%$ for server scenarios on unseen GPUs, while the latency model achieves $\leq8.5\%$ in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall $τ\geq0.76$). Compared to standard physically grounded baselines -- Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency -- our models reduce median absolute percentage error by approximately 4$\times$ on unseen LLM-GPU combinations for server scenarios or approximately 2$\times$ for completely unseen GPUs. WattGPU's data and code are publicly available at https://github.com/maufadel/wattgpu.

• The paper introduces learned XGBoost models to predict mean GPU power draw and inter-token latency, significantly outperforming traditional heuristics.
• It leverages structured, publicly available GPU and LLM specifications to generalize predictions across unseen hardware and model configurations.
• The approach facilitates energy-optimal GPU selection and capacity planning, reducing operational costs and enabling sustainable AI deployments.

Authoritative Summary of "WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs" (2607.02391)

Motivation and Problem Statement

The rapid proliferation of LLM inference workloads has triggered unprecedented GPU energy consumption in both hyperscale and independent cloud deployments. Current practice for energy- and latency-aware deployment of LLM-GPU configurations requires exhaustive empirical profiling, which is resource-intensive and often infeasible for typical operators. Analytical heuristics (e.g., TDP scaling, simple roofline models) provide only coarse estimates and fail to generalize to new hardware architectures or unseen LLM configurations. "WattGPU" directly addresses this limitation by proposing learned predictive models for mean GPU power draw and inter-token latency (ITL) that generalize to GPUs and LLMs not present at training time, using only structured, publicly available specifications as features.

Predictive Modeling Approach

WattGPU introduces two regression models, each implemented with optimized XGBoost ensembles:

• The mean power draw model ($\bar{P}$) predicts average instantaneous GPU power draw in both offline and server scenarios, normalized by the device's TDP to support generalization across different thermal design regimes.
• The ITL model predicts per-token inference latency, targeting both high-throughput and low-load interactive workloads.

Both models rely exclusively on structured metadata: GPU hardware details (e.g., memory bandwidth, transistor count, FP16 TFLOPS, boosting ratio, release year), LLM architecture attributes (parameter count, number of layers/heads, hidden size), and scenario context. Feature engineering includes derived metrics such as bandwidth latency (model memory footprint divided by bandwidth) and compute latency (theoretical lower bound per token given FP16 throughput), which are critical for capturing hardware-model interactions. Notably, all input features are computable from public sources, requiring no empirical measurement or hardware access.

Dataset and Experimental Setup

Evaluation is performed on the Watt Counts dataset (Argerich et al., 10 Apr 2026), which comprehensively benchmarks 42 dense LLMs (125M–27B params) and 8 NVIDIA server-grade GPUs (Volta, Turing, Ampere, Ada, Hopper architectures) across offline, low-, and high-load server scenarios. Evaluation rigor is ensured via three strategies: (a) 5-fold GroupKFold CV to assess generalization to unseen (GPU, LLM) pairs, (b) LOGO (Leave-One-GPU-Out) for unseen GPU generalization, and (c) LOLO (Leave-One-LLM-Out) for unseen model generalization.

Main Empirical Results

Absolute accuracy: On unseen GPUs (LOGO), the mean power model achieves median absolute percentage error (MdAPE) $\leq3.4\%$ in offline and $\leq13.5\%$ in server settings; the ITL model achieves MdAPE $\leq8.5\%$ in server mode.

Relative (ranking) accuracy: Strong Kendall $\tau$ for GPU ranking is maintained (e.g., server scenario $\tau \geq 0.76$), facilitating reliable selection of energy-optimal or SLA-compliant GPU for a given deployment.

Comparison to Analytical Baselines: Both the TDP-based and roofline baselines exhibit substantially higher errors (e.g., TDP MdAPE $>26\%$ in server scenarios; roofline ITL MdAPE at $80\%$ offline, $29.6\%$ server), and their ranking ability degrades for non-saturated load conditions. Learned models outperform analytical counterparts by approximately 2–4$\times$ on unseen hardware-model configurations.

Figure 1: GPU selection can yield up to a 43% reduction in power draw (A30 vs. H100) for Llama 3.1 8B at a fixed SLA—analytical heuristics would mispredict the optimal choice.

Analysis of Model Behavior

The feature importance analysis reveals domain-conformant findings:

Figure 2: XGBoost feature importances highlight scenario and memory bandwidth as dominant factors for power; bandwidth latency is paramount for ITL predictions.

• Power model: Dominated by scenario type and GPU bandwidth features; model size and architectural features are secondary, except on large-core, memory-saturated GPUs.
• ITL model: Bandwidth latency is the most significant feature, validating that inference is primarily memory-bound across modern LLMs, with compute latency and FP16 throughput contributing in large-scale or compute-saturated conditions.

  Figure 3: Distribution of mean power draw across 42 LLMs on 8 GPUs and scenario types; GPU architecture and scenario are the primary axes of variation.

Interestingly, while TDP and throughput per watt remain useful as scaling heuristics for saturated inference, they actively degrade GPU selection quality in latency-constrained scenarios or when model size is near the hardware's capacity limit.

Implications and Applications

Practically, WattGPU enables pre-deployment energy and SLA estimation at scale, including:

• Energy-optimal GPU selection for a target LLM under latency constraints—critical when maximizing fleet-wide efficiency or minimizing operational costs under modern regulatory pressures (e.g., EU AI Act reporting).
• Capacity planning in heterogeneous GPU clusters: accurate ex ante predictions facilitate instance selection and workload allocation without on-premises profiling infrastructure.
• Model selection for a given GPU: Valid ranking of LLMs by energy and latency properties directly facilitates cost/Eco-aware backend selection, model distillation evaluation, or quantization ablations.

Theoretically, the work establishes that structured, context-enriched machine learning can bridge the generalization gap left by first-principles heuristics. It demonstrates that complex, non-linear interactions among hardware scaling, architectural choices, and runtime heterogeneity can be effectively learned even on modest cross-product datasets.

Limitations and Future Directions

The largest generalization gap is observed for offline ITL ($\leq3.4\%$0 MdAPE on unseen GPUs) versus server scenarios, which is attributed to missing workload-aware features (e.g., continuous batching, prompt length effects, KV-cache artifacts). Extending to mixture-of-experts, quantized, and multi-GPU models will require explicit modeling of routing, gating, or communication effects not captured in current feature sets. Incorporating workload statistics or developing hybrid ensemble models that leverage both analytical and learned structure are proposed as fruitful next steps, in line with directions from Imai et al. [2024].

Conclusion

This work advances energy- and latency-aware AI system design by introducing the first predictive models for LLM inference power and latency that generalize to previously unseen GPUs and LLMs with high accuracy, using only static metadata. The models operationally outperform conventional heuristics, reducing both energy and cost while alleviating the need for expensive profiling. These contributions will be increasingly relevant as scalable, sustainable, and regulation-compliant AI deployment becomes an industry imperative. The WattGPU framework and its accompanying dataset set a rigorous foundation for continued research into sustainable and resource-efficient AI at inference scale.