Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

Abstract: While the large energy consumption of LLMs is recognized by the community, system operators lack guidance for energy-efficient LLM inference deployments that leverage energy trade-offs of heterogeneous hardware due to a lack of energy-aware benchmarks and data. In this work we address this gap with Watt Counts: the largest open-access dataset of energy consumption of LLMs, with over 5,000 experiments for 50 LLMs across 10 NVIDIA Graphics Processing Units (GPUs) in batch and server scenarios along with a reproducible, open-source benchmark that enables community submissions to expand this dataset. Leveraging this dataset, we conduct a system-level study of LLM inference across heterogeneous GPU architectures and show that GPU selection is crucial for energy efficiency outcomes and that optimal hardware choices vary significantly across models and deployment scenarios, demonstrating the critical importance of hardware-aware deployment in heterogeneous LLM systems. Guided by our data and insights, we show that practitioners can reduce energy consumption by up to 70% in server scenarios with negligible impact on user experience, and by up to 20% in batch scenarios.

• The paper presents the largest open-access dataset benchmarking energy and performance trade-offs of 50 LLMs across 10 NVIDIA GPU types.
• The paper details a modular benchmarking framework using EnergyMeter and PSUtil to measure GPU energy consumption in both batch and server inference scenarios.
• The paper demonstrates that optimal energy efficiency depends on the interplay of model architecture, workload, and GPU selection, challenging reliance on standard vendor metrics.

Energy-Aware Benchmarking for Sustainable LLM Inference Across Heterogeneous GPUs

Introduction and Motivation

The rapid adoption of LLMs has amplified concerns about their aggregate energy footprint, particularly for inference deployments that must sustain high availability and low-latency service. While training energy costs are widely acknowledged, inference—due to the operational scale and continuous demand—can ultimately constitute a greater share of real-world energy consumption. However, practitioners lack practical guidance for energy-efficient deployments, particularly in heterogeneous GPU environments with diverse performance and power characteristics. “Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures” (2604.09048) addresses this gap. The work presents the largest open-access, systematically collected dataset measuring the energy and performance trade-offs of 50 LLMs across 10 NVIDIA GPU types, and introduces an extensible, reproducible benchmarking framework for rigorous hardware-aware evaluation.

Benchmark Design and Methodology

The authors implement a modular benchmarking suite designed for both batch (offline) and server (online) inference scenarios. The orchestrator parses user-supplied configurations, launches the model via vLLM (synchronous for batch, asynchronous for server), and collects metrics using EnergyMeter (GPU/CPU/DRAM power) and PSUtil (system stats) for reproducibility. Measurements include GPU-level energy, power draw, latency (e.g., time-to-first-token, TTFT), throughput, and memory/CPU utilization.

Figure 1: Architecture of the benchmarking system, orchestrating workload configuration, measurement, and aggregation for both batch and server scenarios.

The benchmark focuses on GPU energy consumption, justifying this by validating that GPUs consistently comprise 79–92% of total node energy under both high and low utilization. To ensure measurement fidelity and comparability, all experiments fix deterministic inference parameters and control for thermal/power state between runs.

Energy Metrics:

• Batch scenarios: Energy per generated token enables comparison across variable execution times and workloads.
• Server scenarios: Mean power draw is preferred due to the mixture of active and idle periods, reflecting real-world usage more accurately.

Dataset Construction and Scope

The Watt Counts dataset comprises over 5,000 experiments (370+ unique LLM-GPU combinations) across 50 open LLMs (≤30B parameters) and 10 NVIDIA GPUs (spanning Volta, Turing, Ampere, Ada Lovelace, and Hopper architectures, 16–141GB VRAM, TDPs from 70–700W). Both enterprise and consumer-class GPUs are included, enabling cross-architecture and cross-market analyses. Workloads are limited to single-GPU settings to isolate device-level characteristics.

Systematic Analysis of LLM Inference Energy Efficiency

1. GPU Architecture Efficiency Across LLMs

The principal finding is that no single GPU achieves universal energy-optimality across the full range of LLMs and workloads. The H100 NVL demonstrates the best energy-per-token in ~90% of batch scenarios, but certain workloads favor the H200 (very large models) or L4/A30 (very small models). Memory constraints prevent some GPUs from running the largest models, further influencing optimal hardware selection.

Figure 2: Mean GPU energy per token by model size and GPU class; H100 excels except for the smallest/very largest models.

Throughput trends non-monotonically with FLOPS and memory bandwidth; for instance, the H200 (higher bandwidth, lower TFLOPS/W) often exceeds the H100 in raw throughput but not in energy/token due to higher idle and max power.

Figure 3: Mean throughput distributions for selected GPUs and model size classes, emphasizing the memory bandwidth bottleneck typical of LLM inference workloads.

Crucially, system-level energy efficiency cannot be predicted directly from vendor-supplied FLOPS, TDP, or TFLOPS/W metrics. Memory bandwidth emerges as a primary determinant of LLM throughput, but actual energy efficiency is a compound function of workload, model architecture, and deployment mode.

2. Model Architectures and Their Influence

Analysis confirms a sublinear scaling law for energy per token with increasing model parameter count (exponent ≈0.24–0.28). However, model architecture has significant residual effects: increasing key-value head count and attention head count increases energy per token even after controlling for parameter count; wider hidden sizes can be marginally more efficient for a fixed parameter total.

Figure 4: Sublinear log-log relation between model active parameter count and GPU energy per token, with significant outliers due to architectural and kernel support differences.

Models with similar size thus may have substantially divergent energy footprints if they differ in architectural specifics (e.g., MoE vs. dense, differing head counts). Consequently, both systems and model designers must jointly consider these factors for optimal deployments.

3. Workload Scenario Sensitivity: Batch vs. Server

Deployment scenario exerts a dominant leverage on energy efficiency. In batch (offline) mode, mean power tracks close to TDP, with throughput and TDP jointly defining energy efficiency. In server scenarios—often idle or lightly loaded—idle and max power draw, rather than TDP, determine aggregate efficiency.

Figure 5: GPU power draw statistics stratified by model size and workload scenario, demonstrating the divergence between TDP, idle, and active power in real-world server deployments.

Different GPUs become optimal in different scenarios. Server tasks favor low idle power GPUs (T4, L4) for small/medium models, but these often incur unacceptable latency (TTFT), limiting practical applicability. Throughput remains essential to meeting latency SLAs, especially under high request loads, but the energy-optimal choice is frequently not the GPU with the highest theoretical throughput.

Figure 6: Trade-off between mean GPU power draw and 95th percentile TTFT, illustrating the inherent latency–energy efficiency dilemma in choosing hardware for deployed LLM services.

Implications, Limitations, and Future Directions

Implications:

• Optimal hardware selection is deployment-specific: Model size, architecture, and workload conditions must be considered together.
• Simple proxies (e.g., FLOPS/W, TDP) fail to capture system-level reality.
• Real-world deployments seeking Pareto-efficient trade-offs between energy and latency must leverage empirical, workload-driven measurements, not datasheet specifications.

Practical guidance enabled by this work includes actionable analyses for hardware-aware job placement, automated scheduling, and multi-objective inference optimization targeting both sustainability and service-level metrics.

Limitations:

• All experimentation is single-GPU, 16-bit precision. Extensions to quantized and multi-GPU/multisocket deployments are left for future work, as are benchmarks incorporating alternate inference engines and emerging model types.
• Host and cooling energy are not fully accounted for, though GPU energy tracks >80% of system consumption.

Future Directions:

• Systematic inclusion of quantized models;
• Automated, hardware-aware scheduling and placement;
• Integration with multi-GPU and distributed settings;
• Broader metrics encompassing carbon footprint and cooling.

Conclusion

Watt Counts delivers the largest, most systematic open-access resource for evaluating the energy efficiency of LLM inference across heterogeneous GPUs and deployment scenarios (2604.09048). The findings challenge common assumptions about GPU energy efficiency and demonstrate that optimizing LLM inference requires nuanced, empirically grounded, workload- and hardware-aware approaches. The benchmark and dataset catalyze further research into sustainable AI and provide a practical foundation for energy-conscious deployment at scale.