The case for 4-bit precision: k-bit Inference Scaling Laws (2212.09720v2)

Abstract: Quantization methods reduce the number of bits required to represent each parameter in a model, trading accuracy for smaller memory footprints and inference latencies. However, the final model size depends on both the number of parameters of the original model and the rate of compression. For example, a 30B 8-bit model and a 60B 4-bit model have the same number of bits but may have very different zero-shot accuracies. In this work, we study this trade-off by developing inference scaling laws of zero-shot performance in LLMs to determine the bit-precision and model size that maximizes zero-shot performance. We run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size -- splitting the parameters into small independently quantized blocks -- and the quantization data type being used (e.g., Int vs Float). Overall, our findings show that {4-bit} precision is almost universally optimal for total model bits and zero-shot accuracy.

Overview of "The Case for 4-bit Precision: k-bit Inference Scaling Laws"

This paper addresses a pressing issue in the deployment of LLMs, which is the trade-off between model performance and resource efficiency, specifically concerning quantization. Dettmers and Zettlemoyer explore the limits of precision reduction in LLM inference by quantifying the benefits of lowering precision, with a focus on whether a 4-bit representation strikes an optimal balance.

The central contribution is a comprehensive empirical analysis involving over 35,000 experiments across various LLM architectures, such as BLOOM, OPT, NeoX/Pythia, and GPT-2. These models, ranging from 19 million to 176 billion parameters, were subjected to different quantization levels from 3-bit to 16-bit precision to understand their impact on zero-shot inference capabilities.

Key Findings

1. Optimal Precision: The primary finding indicates that 4-bit quantization consistently delivers the best trade-off between zero-shot accuracy and model size across all tested models and scales. Remarkably, this paper reveals that further reducing the precision to 3-bits results in a noticeable degradation in performance, emphasizing 4-bits as the lowest viable precision for most scenarios.
2. Scaling Laws and Zero-Shot Performance: The investigation establishes scaling laws related to bit-precision and model sizes, showing that zero-shot accuracy increases predictably with a reduction in precision down to 4-bits, with little benefit below this threshold.
3. Block Size and Data Types: Enhancements in scaling trends were observed by using smaller block sizes (e.g., 64 parameters per block) and specific data types such as floating point and quantile quantization, which fostered improved utilization of the available precision.
4. Insufficient Improvement in Higher Precisions: Neither data types nor other advanced quantization techniques demonstrated significant scaling improvements for higher (6 to 8-bit) precisions, likely because these precisions already inherently maintain sufficient fidelity for the model weights.

Implications and Future Directions

The implications of these findings are substantial both practically and theoretically. The recommendation to employ 4-bit quantization can lead to substantial savings in memory and computational requirements, thus making LLMs more accessible for deployment on smaller hardware configurations. This is particularly relevant when considering the constraints of GPU memory, as the compression achieved self-evidently allows larger models to be utilized without prohibitive increases in memory footprint.

Beyond the straightforward application of 4-bit precision, the paper highlights two promising avenues for future advances:

• ***Development of Novel Quantization Methods*: The potential for new quantization techniques and data types that can address the challenges posed by outliers and achieve low-bit precision efficiency is emphasized.
• One-Shot Quantization Techniques: While the paper largely focuses on zero-shot methods, the authors note the promise in one-shot methods with appropriate data blocking, which could theoretically push below 4-bit precisions without sacrificing performance.

Conclusion

This paper underscores the efficacy of 4-bit precision as a robust choice for quantizing LLMs, maximizing zero-shot performance while reducing resource demands. The comprehensive empirical approach sets a benchmark for future quantization strategies while providing a clear direction for further research into more efficient data representations and quantization algorithms. As such, it serves as a foundational reference point for researchers and practitioners seeking to optimize the deployment of large-scale AI models.