WattGPU: Energy-Aware GPU Inference Estimation

• WattGPU is a framework that predicts deployment-critical GPU power draw and inter-token latency from public LLM metadata and GPU specifications.
• It employs gradient-boosted regression trees (XGBoost) to model different operating scenarios, distinguishing between offline and server inference settings.
• Empirical evaluations on 42 open-source LLMs and 8 NVIDIA GPUs demonstrate significant error reduction and improved GPU ranking for energy-aware scheduling.

Searching arXiv for papers on WattGPU and closely related energy-aware GPU inference benchmarking/modeling. WattGPU is a predictive framework for LLM inference that estimates two deployment-critical quantities—mean GPU power draw and inter-token latency (ITL)—from publicly available LLM metadata and GPU specifications, without requiring profiling of the target LLM–GPU pair or hardware access. It is designed for pre-deployment GPU selection, capacity planning, and energy-aware scheduling, and it is evaluated on 42 open-source LLMs spanning 0.1B–27B parameters and 8 NVIDIA server-grade GPUs under offline and server scenarios (Argerich et al., 2 Jul 2026).

1. Problem setting and scope

WattGPU addresses a practical systems problem: selecting the most energy-efficient GPU for a given LLM and serving scenario without exhaustively benchmarking every model–hardware combination. The motivating use case is heterogeneous inference deployment, where both electrical load and service quality depend on the interaction among model architecture, GPU characteristics, and operating scenario. A concrete example given by the authors is that, for serving Llama 3.1 8B under a latency target of mean ITL $< 25$ ms, selecting an NVIDIA A30 instead of an H100 can reduce power draw by 43%, provided the model fits in memory (Argerich et al., 2 Jul 2026).

The framework is explicitly organized around generalization to unseen GPUs and unseen LLMs. In this context, unseen GPUs are hardware models absent from training data, and unseen LLMs are model architectures or scales not observed during training. This emphasis distinguishes WattGPU from profiling-dependent approaches that require runtime measurements on the target platform.

WattGPU is also scenario-aware. It separates offline execution, where many concurrent requests are processed for maximum throughput, from server execution, where requests arrive according to a load pattern. That distinction is central because the paper treats power draw and ITL not as invariant hardware properties but as deployment-conditioned quantities.

2. Predictive targets and feature space

WattGPU uses two separate predictive targets: mean GPU power draw $\bar{P}$ in Watts and ITL in seconds. The paper defines ITL differently for the two main operating regimes:

$ITL=\frac{L-TTFT}{R_{tok}-1}$

for the server scenario, where $L$ is end-to-end latency from arrival to last token, $TTFT$ is time to first token, and $R_{tok}$ is the number of generated response tokens; and

$ITL=\frac{W}{N_{tok}}$

for the offline scenario, where $W$ is wall-clock time and $N_{tok}$ is total generated tokens (Argerich et al., 2 Jul 2026).

The feature set combines LLM-side, GPU-side, and scenario metadata. The LLM features are Model Type, Parameters (log), Number of Layers, KV Heads, and, for ITL only, Attention Heads and Hidden Size. The GPU features are Base Clock Frequency, Boosting Ratio, Transistor Count, Memory Bandwidth, Release Year, and, for ITL only, Memory Size, FP16 (T)FLOPS, and Memory Type. Both models also include Operational Scenario.

Three engineered features are important. Boosting Ratio captures GPU frequency scaling. Bandwidth latency represents the time required to read model weights from memory once. Compute latency is defined as

$$C_{\mathrm{lat}}=\frac{2P_{\mathrm{model}}\cdot 10^{9}}{FLOPS_{GPU}}$$

where $\bar{P}$0 is model parameters in billions and $\bar{P}$1 is maximum FP16 FLOPS. The corresponding roofline-style ITL baseline is

$\bar{P}$2

with $\bar{P}$3 the GPU memory bandwidth and $\bar{P}$4 the expected batch size, set to $\bar{P}$5 for server and $\bar{P}$6 for offline (Argerich et al., 2 Jul 2026).

The paper is explicit about what the current feature space does not include: prompt lengths, output lengths, detailed concurrency, and KV-cache usage are excluded. The authors identify this as a limitation, especially for offline ITL.

3. Model architecture and baselines

WattGPU uses two separate supervised regression models, both implemented as gradient-boosted regression trees with XGBoost. The authors evaluate linear regression, elastic net, multi-layer perceptrons, and gradient-boosted regression, and select XGBoost because it performs best in Median Absolute Percentage Error (MdAPE) and Pearson correlation $\bar{P}$7, especially in leave-one-GPU-out validation (Argerich et al., 2 Jul 2026).

A notable design choice concerns the power target. Rather than predicting raw power directly, the power model normalizes the target by GPU TDP or configured power limit. The paper explains that this encourages the model to learn scale-invariant relative power behavior, improving extrapolation to unseen hardware. The intended form is

$\bar{P}$8

or by configured power limit when applicable.

The selected hyperparameters are fixed and modest. For the power model, the paper reports $\bar{P}$9, $ITL=\frac{L-TTFT}{R_{tok}-1}$0, and $ITL=\frac{L-TTFT}{R_{tok}-1}$1. For the ITL model, it reports $ITL=\frac{L-TTFT}{R_{tok}-1}$2, $ITL=\frac{L-TTFT}{R_{tok}-1}$3, and $ITL=\frac{L-TTFT}{R_{tok}-1}$4. Hyperparameters were chosen by grid search over $ITL=\frac{L-TTFT}{R_{tok}-1}$5, $ITL=\frac{L-TTFT}{R_{tok}-1}$6, and $ITL=\frac{L-TTFT}{R_{tok}-1}$7, with all other settings left at default values.

The baseline comparators are intentionally simple and physically grounded. For power, the paper uses TDP and Load-Scaled TDP:

$ITL=\frac{L-TTFT}{R_{tok}-1}$8

with $ITL=\frac{L-TTFT}{R_{tok}-1}$9 for offline and $L$0 for low- and high-load server scenarios. For latency, it uses the roofline-derived memory-streaming estimate above. The paper’s central claim is not that these baselines are useless, but that they are too coarse for accurate pre-deployment selection, particularly in server scenarios.

Feature-importance analysis supports that interpretation. For power prediction, the most important features are Scenario, Memory bandwidth, and Bandwidth latency. For ITL prediction, the most important are Bandwidth latency, Scenario, Compute latency, and GPU FP16 TFLOPS. The authors interpret ITL as largely memory-bound.

4. Dataset, hardware coverage, and evaluation protocol

WattGPU is trained and evaluated on the Watt Counts benchmark and dataset, which provides the underlying energy-aware inference measurements across heterogeneous NVIDIA GPUs (Argerich et al., 10 Apr 2026). For WattGPU itself, the filtered dataset contains 42 open-source dense LLMs and 8 server-grade NVIDIA GPUs. Two consumer GPUs, RTX 4090 and RTX 3090, and 8 Mixture-of-Experts models are excluded from the main WattGPU study (Argerich et al., 2 Jul 2026).

The 8 GPUs are Tesla V100 SXM2, Tesla T4, A100 SXM4, A30 PCIe, L40S, L4, H100 NVL, and H200 NVL. The model set spans multiple families, including GPT-2, Qwen2, Qwen3, Llama, Llama 2, Llama 3.x, Gemma 2, Gemma 3, Phi, Phi-3, Phi-4, Mistral, Granite, OLMo2, Falcon, GPT-J, GPT-NeoX, InternLM, Solar, and Nemotron.

The scenarios are Offline, Server low-load, and Server high-load. In the server setting, requests arrive according to a Poisson process with

$L$1

for low-load and high-load conditions.

Evaluation uses three protocols of increasing difficulty. 5-fold CV with GroupKFold on $L$2 tests unseen combinations of known GPUs and known LLMs. Leave-One-GPU-Out (LOGO) tests generalization to completely unseen GPUs. Leave-One-LLM-Out (LOLO) tests generalization to completely unseen LLMs. The main metrics are MdAPE,

$L$3

Pearson correlation $L$4, and Kendall $L$5 for ranking quality (Argerich et al., 2 Jul 2026).

5. Empirical accuracy and ranking behavior

The strongest results are in power prediction and server-side ITL. On unseen GPUs, the mean power model achieves MdAPE $L$6 for offline and $L$7 for server scenarios. For server ITL on unseen GPUs, the latency model achieves MdAPE $L$8. The paper also reports strong GPU ranking correlations in server scenarios, with Kendall $L$9 (Argerich et al., 2 Jul 2026).

Task	Setting	WattGPU result	Strongest baseline
Power	Offline, LOGO	3.4% MdAPE	4.4%
Power	Server, LOGO	13.5% MdAPE	26.0%
ITL	Server, LOGO	8.5% MdAPE	29.6%
ITL	Offline, LOGO	24.9% MdAPE	80.2%

These results are most striking in server power prediction. In that regime, plain TDP reaches 190.1% MdAPE, while Load-Scaled TDP reaches 26.0% MdAPE; WattGPU reduces the error to 13.5% MdAPE under LOGO. For server ITL, the roofline baseline retains strong ranking behavior but poor absolute accuracy, while WattGPU lowers error from 29.6% to 8.5% on unseen GPUs.

Offline ITL is clearly the weakest case. The paper reports 24.9% MdAPE under LOGO for offline ITL and attributes this to static metadata being insufficient to capture the dynamics of vLLM continuous batching in throughput-oriented execution. This is an important corrective to a possible misconception: WattGPU is not presented as uniformly strong across all deployment modes.

The practical significance of ranking quality is emphasized throughout. The framework is intended not merely to approximate absolute values, but to preserve the relative ordering of GPUs for deployment selection. That objective is operationally important because hardware decisions often depend more on identifying the best feasible GPU than on minimizing pointwise regression error.

6. Operational significance, relation to benchmarking, and limitations

WattGPU is designed as a zero-profiling, pre-deployment estimator. The intended uses include energy-aware scheduling, GPU instance selection, advance cost and energy estimation, capacity planning, procurement, and sustainability reporting. Its most direct operational role is pre-screening candidate GPUs before costly benchmarking (Argerich et al., 2 Jul 2026).

Its empirical foundation is the Watt Counts benchmark, which provides measured GPU power draw, energy per token, throughput, TTFT, queueing time, end-to-end latency, and telemetry across more than 5,000 experiments for 50 LLMs and 10 NVIDIA GPUs (Argerich et al., 10 Apr 2026). A plausible implication is that WattGPU should be understood not as a replacement for benchmarking, but as a predictor that narrows the benchmarking search space.

Several limitations define the present scope. Hardware coverage is restricted to NVIDIA server-grade GPUs; consumer GPUs are excluded from the main evaluation, and no AMD or Intel accelerators are studied. The model scope is limited to dense LLMs; Mixture-of-Experts models, quantized models, and multi-GPU deployments are left for future work. The serving stack is tied to vLLM, and the paper explicitly links offline ITL error to vLLM continuous batching behavior. Current features are static and do not incorporate prompt lengths, output lengths, detailed concurrency, or KV-cache usage.

These boundaries also clarify a common misunderstanding. WattGPU is not an energy-measurement framework in the strict sense of direct wattmeter-style hardware instrumentation; it predicts mean GPU power draw and ITL from metadata and specifications. Its contribution lies in pre-deployment inference, not in replacing runtime telemetry. The authors therefore present it as a practical step toward energy-aware GPU selection rather than as a complete model of all serving-system dynamics.

The framework’s reproducibility posture is unusually strong for this class of work. The paper states that data and code are publicly available at https://github.com/maufadel/wattgpu, and the benchmark lineage is likewise open through Watt Counts. That openness is consequential because the central claim of WattGPU—that useful pre-deployment prediction of LLM inference power and latency is possible without profiling—depends on public access to both training data and evaluation methodology (Argerich et al., 2 Jul 2026).