The Energy Cost of Execution-Idle in GPU Clusters

Abstract: GPUs are becoming a major contributor to data center power, yet unlike CPUs, they can remain at high power even when visible activity is near zero. We call this state execution-idle. Using per-second telemetry from a large academic AI cluster, we characterize execution-idle as a recurring low-activity yet high-power state in real deployments. Across diverse workloads and multiple GPU generations, it accounts for 19.7% of in-execution time and 10.7% of energy. This suggests a need to both reduce the cost of execution-idle and reduce exposure to it. We therefore build two prototypes: one uses automatic downscaling during execution-idle, and the other uses load imbalance to reduce exposure, both with performance trade-offs. These findings suggest that future energy-efficient GPU systems should treat execution-idle as a first-class operating state.

• The paper introduces a telemetry framework to precisely measure per-second GPU power during sustained execution-idle states.
• It quantifies that execution-idle can account for up to 65% energy in serving workloads, revealing a disparity between active and idle power consumption.
• The study evaluates mitigation strategies like frequency downscaling and workload consolidation, highlighting critical trade-offs between energy savings and increased latency.

Characterizing the Energy Cost of Execution-Idle States in GPU Clusters

Introduction and Problem Statement

The paper "The Energy Cost of Execution-Idle in GPU Clusters" (2604.04745) introduces and rigorously analyzes the execution-idle state in GPU clusters—a regime wherein a GPU, allocated and with a program resident, exhibits minimal activity but remains at an elevated power state. Unlike CPUs, which drop to near-baseline power during inactivity, GPUs persistently draw significant power even when visible computation, memory, and communication metrics are near zero.

Figure 1: CPU power falls with idle time, but GPU power remains elevated even when a loaded program is fully idle.

This persistent power draw is particularly notable in AI-serving contexts and large-scale multi-GPU deployments. Previous analyses often rely on aggregate power metrics (e.g., GPU-hours, TDP), failing to capture the nuanced runtime phases where energy may be spent unproductively. The authors fill this gap with a fine-grained telemetry-driven study, measuring per-second power and utilization signals across six GPU generations in a 31-day cluster-scale deployment, substantiated with replayed industry workload traces.

Methodology: Telemetry-Driven Execution-Idle Characterization

The paper implements a telemetry framework leveraging NVML, DCGM, and OS-level counters to collect metrics spanning GPU power, activity (SM, tensor, DRAM, FP16/32/64), communication (PCIe, NVLink), clocks, and host activity. Each GPU-second is attributed to a job using Slurm metadata, discarding samples lacking attribution or malformed records. An interval is classified as execution-idle if all compute/memory signals remain below 5% and communication signals under 1 GB/s for at least 5s continuously—a conservative threshold, aiming to capture sustained, not transient, low-activity gaps.

Prevalence, Magnitude, and Distribution of Execution-Idle States

Execution-idle states manifest across all six GPU generations studied, spanning RTX A6000, RTX 6000 Ada, L40(S), A100, H100, and B200. In the cluster, execution-idle accounts for 19.7% of in-execution time and 10.7% of energy, with deep idle constituting 24% of job-attributed GPU time yet only 7% of energy, underscoring the disproportionate energy impact of loaded-but-inactive regimes.

Figure 2: Time-aligned power, SM and DRAM utilization as well as normalized frequency on an L40S GPU, illustrating execution-idle intervals.

Cluster-wide energy utilization is substantially below the aggregate TDP upper bound, but this does not translate to power proportionality—execution-idle draws considerably more power than deep idle, as demonstrated in multiple generations (Figure 3).

Figure 3: Power in the execution-idle state remains substantially above deep idle across all GPU models in the study.

Workload-Level Analysis: Training, Inference, and Serving

Execution-idle fractions vary significantly across workload types:

• Serving workloads: Online serving jobs spent 61% of in-execution time and 48% of energy in execution-idle.
• Batch inference and training: 12–13% of time and 6–7% of energy are execution-idle.
• Industry traces (OpenAI, Qwen, Azure): Range from 14–76% of time and 7–65% of energy, with bursty arrival patterns exacerbating idle intervals.

  Figure 4: Execution-idle time and energy fractions across academic and replayed industry workloads.

CDF analyses show a heavy right-skew: a substantial tail of jobs (15.4%) spends over half their time in execution-idle, indicating significant exposure even outside pure serving contexts.

Figure 5: CDF of per-job execution-idle time and energy fractions.

Inter-request intervals further reveal that burstiness is inherent in realistic demand patterns, not just an artifact of academic workloads.

Figure 6: CDF of per-GPU inter-request intervals for industry serving traces.

Temporal Characteristics and Causes

Execution-idle intervals persist for substantial durations. Median duration is 9s; the 90th percentile is 44s, and the 99th percentile extends to 836s. The bulk of execution-idle onsets are correlated with periods of PCIe transfers (48%), followed by compute-to-idle transitions (33%), network-backed I/O (17%), and NVLink activity (2%). This clustering is robust across platforms, highlighting the impact of I/O bottlenecks and workload design.

Figure 7: CDF of execution-idle interval durations.

Figure 8: Clustering of execution-idle events by preceding signals, emphasizing PCIe and NIC activity.

Energy Management and Trade-offs

Two prototype interventions are evaluated:

• Frequency downscaling during execution-idle: Manual reduction of compute and memory clocks lowers execution-idle power significantly—from 105W to 61W (SM-only), 35W (SM+memory), yielding a 22–34% average power reduction but incurring a 29–160% increase in p95 latency.
• Load imbalance/scheduling: Consolidating work onto fewer GPUs to maximize deep idle periods halves total GPU energy consumption with negligible change in utilization but increases serving latency (80–93%).

  Figure 9: Energy, latency, and utilization trade-offs under load imbalance.

  

  Figure 10: Power profile under SM-only and SM+memory execution-idle-aware frequency control.

Power–latency tradeoff curves further quantify the explicit energy–performance tension involved.

Figure 11: Power CDF under frequency downscaling, revealing lower tail under aggressive clock reductions.

Implications and Future Research Directions

Execution-idle demands explicit management in GPU system design. Aggregate utilization metrics are poor proxies for cluster energy efficiency; practical interventions must navigate latency and responsiveness constraints. Workload-power interfaces are needed for context-aware downscaling. SLO-driven resource management can incorporate execution-idle detection to optimize the balance between responsiveness and energy efficiency. Component-level (SM, memory) power proportionality offers a promising direction for future system architectures.

These findings suggest that execution-idle is a pervasive, structurally induced energy inefficiency that transcends hardware and software boundaries, especially in serving workloads where responsiveness is prioritized and GPUs remain allocated during idle intervals. Explicit detection, management, and co-design between hardware and software are necessary to mitigate this cost.

Conclusion

This paper delineates execution-idle as a first-class operating state in GPU clusters, quantifies its prevalence and energy impact, and rigorously analyzes the efficacy and trade-offs of mitigation strategies. Execution-idle is substantial (as high as 65% of serving energy in some traces), not limited to a single hardware generation or workload, and critically intertwined with the energy–performance dynamics of AI infrastructure. Future systems must treat this state explicitly, leveraging telemetry-driven insights to inform workload management, hardware controls, and power-proportional design.