Input-Dependent Power Usage in GPUs (2409.18324v1)

Abstract: GPUs are known to be power-hungry, and due to the boom in artificial intelligence, they are currently the major contributors to the high power demands of upcoming datacenters. Most GPU usage in these popular workloads consist of large general matrix-matrix multiplications (GEMMs), which have therefore been optimized to achieve high utilization of hardware resources. In this work, we show that modifying the input data to GEMMs, while maintaining the matrix shapes and sizes can notably change the power consumption of these kernels. We experiment with four kinds of input variations: value distribution, bit similarity, placement, and sparsity, across different data types. Our findings indicate that these variations can change the GPU power usage during GEMM by almost 40%. We hypothesize that input-dependent power usage variations occur due to changes in the number of bit flips in the GPUs. We propose leveraging this property through compiler and scheduler optimizations to manage power and reduce energy consumption.

• The paper demonstrates that variations in GEMM input data can alter GPU power consumption by up to 40%, emphasizing the impact of input characteristics.
• The study validates its findings through rigorous experiments on NVIDIA GPUs like the A100, H100, V100, and RTX 6000 across multiple data types.
• The work suggests opportunities for developing power-aware algorithms and compiler optimizations to enhance GPU efficiency in AI and HPC environments.

Input-Dependent Power Usage in GPUs: An Analytical Exploration

The paper "Input-Dependent Power Usage in GPUs" by Gregersen, Patel, and Choukse, provides a detailed exploration of how input variations affect the power consumption of GPUs, specifically through general matrix-matrix multiplications (GEMMs). Given the significant role GPUs play in current AI workloads and data centers, understanding power efficiency at the granular level is of paramount importance for both economic and environmental sustainability.

Key Findings

The authors present experimental evidence showing that variations in GEMM input data, such as value distribution, bit similarity, placement, and sparsity, can influence power usage by up to 40%. These discoveries are particularly salient for workloads dominated by GEMMs, a common operation in machine learning and HPC environments. The focus on input-dependent power usage deviates from traditional strategies focused solely on hardware or systemic designs.

• Value Distribution: The paper shows that the power consumption doesn't significantly vary with standard deviation changes in input values. However, a higher mean in floating-point data types tends to reduce power consumption. Moreover, inputs drawn from a constrained set of unique values result in lower power usage.
• Bit Similarity: Results indicate a correlation between lower power usage and high bit similarity across input data. This is attributed to fewer bit flips during computation, aligning with prior research on minimizing power through reduced data flipping at the hardware level.
• Placement Patterns: Newly identified patterns suggest that the strategic sorting and alignment of input values can lead to decreased power usage. This behavior is significantly evident when values are sorted into columns or rows, with alignment providing additional benefits.
• Sparsity: Introducing sparsity within the input data results in significantly lower power consumption, but excessive sorting paired with sparsity may counteract this benefit.

Methodology

The authors utilized an extensive set of experiments on the NVIDIA A100 GPU, with further validation across other GPUs such as the NVIDIA H100, V100, and RTX 6000. Utilizing standard GEMM operations, they observed results consistent across various data types, including FP32, FP16, FP16-T, and INT8.

The team's approach to managing external variability, such as running all tests on the same VM instance, strengthens the reliability of their findings. Moreover, the exploration extends to analyzing runtime and energy consumption, reaffirming power as the more critical constraint in large-scale deployments.

Implications and Future Directions

The findings have profound implications, both theoretically and practically. They extend the understanding of how input characteristics influence energy consumption, opening avenues for substantial optimizations in compilers and system schedulers. These findings could lead to the development of power-aware algorithms and optimizations, such as:

• Power Model Development: To better capture input-dependent variations in power, future models could provide powerful tools for optimizing GPU workloads dynamically, based on the input data characteristics.
• Power-Efficient Machine Learning: Given the trend towards increasingly large and complex models, incorporating these findings into the development of machine learning models might yield substantial gains in efficiency. Techniques such as weight rearrangement and alignment could be explored.
• Hardware Design and Compiler Optimizations: Leveraging input-based power management can motivate new hardware design principles and compiler strategies, aiming to maximize sustainable computational performance.

The paper establishes a crucial foundation for future research aimed at reducing the power footprint of GPUs. The consistent experimental validation provides a robust framework for examining how subtle input variations can drive significant shifts in consumption metrics. By highlighting strategies for minimizing power usage without compromising computational efficacy, this paper offers strategic insights that hold the potential to influence the next generation of GPU-centric technologies.