---
title: 'WattGPU: Energy-Aware GPU Inference Estimation'
url: https://www.emergentmind.com/topics/wattgpu
type: topic
---

# WattGPU: Energy-Aware GPU Inference Estimation

Searching arXiv for recent papers on WattGPU and closely related energy-aware GPU inference benchmarking/modeling.
WattGPU is a predictive framework for large language model (LLM) inference that estimates two deployment-critical quantities—mean GPU power draw and inter-token latency (ITL)—from publicly available LLM metadata and GPU specifications, without requiring profiling of the target LLM–GPU pair or hardware access. It is designed for pre-deployment GPU selection, capacity planning, and energy-aware scheduling, and it is evaluated on 42 open-source LLMs spanning 0.1B–27B parameters and 8 NVIDIA server-grade GPUs under offline and server scenarios [2607.02391].

## 1. Problem setting and scope

WattGPU addresses a practical systems problem: selecting the most energy-efficient GPU for a given LLM and serving scenario without exhaustively benchmarking every model–hardware combination. The motivating use case is heterogeneous inference deployment, where both electrical load and service quality depend on the interaction among model architecture, GPU characteristics, and operating scenario. A concrete example given by the authors is that, for serving Llama 3.1 8B under a latency target of mean ITL \(< 25\) ms, selecting an NVIDIA A30 instead of an H100 can reduce power draw by 43%, provided the model fits in memory [2607.02391].

The framework is explicitly organized around generalization to **unseen GPUs** and **unseen LLMs**. In this context, unseen GPUs are hardware models absent from training data, and unseen LLMs are model architectures or scales not observed during training. This emphasis distinguishes WattGPU from profiling-dependent approaches that require runtime measurements on the target platform.

WattGPU is also scenario-aware. It separates **offline** execution, where many concurrent requests are processed for maximum throughput, from **server** execution, where requests arrive according to a load pattern. That distinction is central because the paper treats power draw and ITL not as invariant hardware properties but as deployment-conditioned quantities.

## 2. Predictive targets and feature space

WattGPU uses two separate predictive targets: mean GPU power draw \( \bar{P} \) in Watts and ITL in seconds. The paper defines ITL differently for the two main operating regimes:

\[
ITL=\frac{L-TTFT}{R_{tok}-1}
\]

for the server scenario, where \(L\) is end-to-end latency from arrival to last token, \(TTFT\) is time to first token, and \(R_{tok}\) is the number of generated response tokens; and

\[
ITL=\frac{W}{N_{tok}}
\]

for the offline scenario, where \(W\) is wall-clock time and \(N_{tok}\) is total generated tokens [2607.02391].

The feature set combines LLM-side, GPU-side, and scenario metadata. The LLM features are **Model Type**, **Parameters (log)**, **Number of Layers**, **KV Heads**, and, for ITL only, **Attention Heads** and **Hidden Size**. The GPU features are **Base Clock Frequency**, **Boosting Ratio**, **Transistor Count**, **Memory Bandwidth**, **Release Year**, and, for ITL only, **Memory Size**, **FP16 (T)FLOPS**, and **Memory Type**. Both models also include **Operational Scenario**.

Three engineered features are important. **Boosting Ratio** captures GPU frequency scaling. **Bandwidth latency** represents the time required to read model weights from memory once. **Compute latency** is defined as

\[
C_{\mathrm{lat}}=\frac{2P_{\mathrm{model}}\cdot 10^{9}}{FLOPS_{GPU}}
\]

where \(P_{\mathrm{model}}\) is model parameters in billions and \(FLOPS_{GPU}\) is maximum FP16 FLOPS. The corresponding roofline-style ITL baseline is

\[
T_{ITL}=\frac{2P_{\mathrm{model}}\cdot 10^{9}}{(M_{bw} \cdot B)}
\]

with \(M_{bw}\) the GPU memory bandwidth and \(B\) the expected batch size, set to \(B=1\) for server and \(B=256\) for offline [2607.02391].

The paper is explicit about what the current feature space does **not** include: prompt lengths, output lengths, detailed concurrency, and KV-cache usage are excluded. The authors identify this as a limitation, especially for offline ITL.

## 3. Model architecture and baselines

WattGPU uses two separate supervised regression models, both implemented as gradient-boosted regression trees with XGBoost. The authors evaluate linear regression, elastic net, multi-layer perceptrons, and gradient-boosted regression, and select XGBoost because it performs best in Median Absolute Percentage Error (MdAPE) and Pearson correlation \(r\), especially in leave-one-GPU-out validation [2607.02391].

A notable design choice concerns the power target. Rather than predicting raw power directly, the power model normalizes the target by GPU TDP or configured power limit. The paper explains that this encourages the model to learn scale-invariant relative power behavior, improving extrapolation to unseen hardware. The intended form is

\[
y_{\text{power}}=\frac{\bar{P}}{\mathrm{TDP}}
\]

or by configured power limit when applicable.

The selected hyperparameters are fixed and modest. For the power model, the paper reports \(\texttt{reg\_lambda}=100\), \(\texttt{max\_depth}=6\), and \(\texttt{n\_estimators}=100\). For the ITL model, it reports \(\texttt{reg\_lambda}=100\), \(\texttt{max\_depth}=5\), and \(\texttt{n\_estimators}=200\). Hyperparameters were chosen by grid search over \(\texttt{max\_depth}\), \(\texttt{reg\_lambda}\), and \(\texttt{n\_estimators}\), with all other settings left at default values.

The baseline comparators are intentionally simple and physically grounded. For power, the paper uses **TDP** and **Load-Scaled TDP**:

\[
\bar{P}_{\mathrm{baseline}}=\rho \cdot TDP
\]

with \(\rho = 1.0\) for offline and \(\rho \in \{0.2, 0.6\}\) for low- and high-load server scenarios. For latency, it uses the roofline-derived memory-streaming estimate above. The paper’s central claim is not that these baselines are useless, but that they are too coarse for accurate pre-deployment selection, particularly in server scenarios.

Feature-importance analysis supports that interpretation. For power prediction, the most important features are **Scenario**, **Memory bandwidth**, and **Bandwidth latency**. For ITL prediction, the most important are **Bandwidth latency**, **Scenario**, **Compute latency**, and **GPU FP16 TFLOPS**. The authors interpret ITL as largely memory-bound.

## 4. Dataset, hardware coverage, and evaluation protocol

WattGPU is trained and evaluated on the **Watt Counts** benchmark and dataset, which provides the underlying energy-aware inference measurements across heterogeneous NVIDIA GPUs [2604.09048]. For WattGPU itself, the filtered dataset contains **42 open-source dense LLMs** and **8 server-grade NVIDIA GPUs**. Two consumer GPUs, **RTX 4090** and **RTX 3090**, and 8 Mixture-of-Experts models are excluded from the main WattGPU study [2607.02391].

The 8 GPUs are **Tesla V100 SXM2**, **Tesla T4**, **A100 SXM4**, **A30 PCIe**, **L40S**, **L4**, **H100 NVL**, and **H200 NVL**. The model set spans multiple families, including GPT-2, Qwen2, Qwen3, Llama, Llama 2, Llama 3.x, Gemma 2, Gemma 3, Phi, Phi-3, Phi-4, Mistral, Granite, OLMo2, Falcon, GPT-J, GPT-NeoX, InternLM, Solar, and Nemotron.

The scenarios are **Offline**, **Server low-load**, and **Server high-load**. In the server setting, requests arrive according to a Poisson process with

\[
\lambda \in \{0.017, 0.33\}
\]

for low-load and high-load conditions.

Evaluation uses three protocols of increasing difficulty. **5-fold CV** with GroupKFold on \((GPU, LLM)\) tests unseen combinations of known GPUs and known LLMs. **Leave-One-GPU-Out (LOGO)** tests generalization to completely unseen GPUs. **Leave-One-LLM-Out (LOLO)** tests generalization to completely unseen LLMs. The main metrics are MdAPE,

\[
\mathrm{MdAPE} = \mathrm{median}\left( \left| \frac{y_i - \hat{y}_i}{y_i} \right| \times 100 \right),
\]

Pearson correlation \(r\), and Kendall \(\tau\) for ranking quality [2607.02391].

## 5. Empirical accuracy and ranking behavior

The strongest results are in power prediction and server-side ITL. On unseen GPUs, the mean power model achieves MdAPE \(\leq 3.4\%\) for offline and \(\leq 13.5\%\) for server scenarios. For server ITL on unseen GPUs, the latency model achieves MdAPE \(\leq 8.5\%\). The paper also reports strong GPU ranking correlations in server scenarios, with Kendall \(\tau \geq 0.76\) [2607.02391].

| Task | Setting | WattGPU result | Strongest baseline |
|---|---|---:|---:|
| Power | Offline, LOGO | 3.4% MdAPE | 4.4% |
| Power | Server, LOGO | 13.5% MdAPE | 26.0% |
| ITL | Server, LOGO | 8.5% MdAPE | 29.6% |
| ITL | Offline, LOGO | 24.9% MdAPE | 80.2% |

These results are most striking in server power prediction. In that regime, plain TDP reaches **190.1% MdAPE**, while Load-Scaled TDP reaches **26.0% MdAPE**; WattGPU reduces the error to **13.5% MdAPE** under LOGO. For server ITL, the roofline baseline retains strong ranking behavior but poor absolute accuracy, while WattGPU lowers error from **29.6%** to **8.5%** on unseen GPUs.

Offline ITL is clearly the weakest case. The paper reports **24.9% MdAPE** under LOGO for offline ITL and attributes this to static metadata being insufficient to capture the dynamics of **vLLM continuous batching** in throughput-oriented execution. This is an important corrective to a possible misconception: WattGPU is not presented as uniformly strong across all deployment modes.

The practical significance of ranking quality is emphasized throughout. The framework is intended not merely to approximate absolute values, but to preserve the relative ordering of GPUs for deployment selection. That objective is operationally important because hardware decisions often depend more on identifying the best feasible GPU than on minimizing pointwise regression error.

## 6. Operational significance, relation to benchmarking, and limitations

WattGPU is designed as a **zero-profiling, pre-deployment estimator**. The intended uses include energy-aware scheduling, GPU instance selection, advance cost and energy estimation, capacity planning, procurement, and sustainability reporting. Its most direct operational role is pre-screening candidate GPUs before costly benchmarking [2607.02391].

Its empirical foundation is the Watt Counts benchmark, which provides measured GPU power draw, energy per token, throughput, TTFT, queueing time, end-to-end latency, and telemetry across more than 5,000 experiments for 50 LLMs and 10 NVIDIA GPUs [2604.09048]. A plausible implication is that WattGPU should be understood not as a replacement for benchmarking, but as a predictor that narrows the benchmarking search space.

Several limitations define the present scope. Hardware coverage is restricted to **NVIDIA server-grade GPUs**; consumer GPUs are excluded from the main evaluation, and no AMD or Intel accelerators are studied. The model scope is limited to **dense LLMs**; Mixture-of-Experts models, quantized models, and multi-GPU deployments are left for future work. The serving stack is tied to **vLLM**, and the paper explicitly links offline ITL error to vLLM continuous batching behavior. Current features are static and do not incorporate prompt lengths, output lengths, detailed concurrency, or KV-cache usage.

These boundaries also clarify a common misunderstanding. WattGPU is not an energy-measurement framework in the strict sense of direct wattmeter-style hardware instrumentation; it predicts mean GPU power draw and ITL from metadata and specifications. Its contribution lies in pre-deployment inference, not in replacing runtime telemetry. The authors therefore present it as a practical step toward energy-aware GPU selection rather than as a complete model of all serving-system dynamics.

The framework’s reproducibility posture is unusually strong for this class of work. The paper states that data and code are publicly available at `https://github.com/maufadel/wattgpu`, and the benchmark lineage is likewise open through Watt Counts. That openness is consequential because the central claim of WattGPU—that useful pre-deployment prediction of LLM inference power and latency is possible without profiling—depends on public access to both training data and evaluation methodology [2607.02391].

Source: https://www.emergentmind.com/topics/wattgpu