A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2402.16775v2)

Abstract: Increasing the number of parameters in LLMs usually improves performance in downstream tasks but raises compute and memory costs, making deployment difficult in resource-limited settings. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular due to the rise of LLMs. However, most quantization studies use pre-trained LLMs, and the impact of quantization on instruction-tuned LLMs and the relationship between perplexity and benchmark performance of quantized LLMs are not well understood. Evaluation of quantized LLMs is often limited to LLMing and a few classification tasks, leaving their performance on other benchmarks unclear. To address these gaps, we propose a structured evaluation framework consisting of three critical dimensions: (1) knowledge & capacity, (2) alignment, and (3) efficiency, and conduct extensive experiments across ten diverse benchmarks. Our experimental results indicate that LLMs with 4-bit quantization can retain performance comparable to their non-quantized counterparts, and perplexity can serve as a proxy metric for quantized LLMs on most benchmarks. Furthermore, quantized LLMs with larger parameter scales can outperform smaller LLMs. Despite the memory savings achieved through quantization, it can also slow down the inference speed of LLMs. Consequently, substantial engineering efforts and hardware support are imperative to achieve a balanced optimization of decoding speed and memory consumption in the context of quantized LLMs.

A Comprehensive Evaluation of Quantization Strategies for LLMs

The paper "A Comprehensive Evaluation of Quantization Strategies for LLMs" presents a thorough examination of various quantization methods applied to LLMs. The primary motivation behind this investigation is the increasing computational and memory burden associated with deploying LLMs, especially in resource-constrained environments. Quantization is proposed as a plausible solution to mitigate these limitations by reducing the precision of model parameters, thereby lowering resource demands while maintaining a tolerable performance trade-off.

Key Contributions and Framework

The authors introduce a structured evaluation framework that assesses quantized LLMs across three critical dimensions:

1. Knowledge Content Capacity: This dimension is evaluated through benchmarks such as MMLU and C-EVAL, which measure the model's comprehension across various knowledge domains.
2. Alignment: The adherence of models to human values and preferences is gauged using benchmarks like FollowBench, TruthfulQA, and BBQ.
3. Efficiency: This is measured in terms of computational aspects such as memory consumption and inference speed.

The framework is tested using ten diverse benchmarks, highlighting the models' performance in both knowledge understanding and alignment, alongside their computational efficiency.

Experimental Findings

The paper reveals several noteworthy outcomes:

• 4-bit Quantization Retains Performance: Models quantized to 4 bits demonstrate performance comparable to their full-precision counterparts across most benchmarks. This suggests a viable path for deploying memory-efficient models without significantly sacrificing accuracy.
• Perplexity as a Proxy: The perplexity of quantized models was found to correlate well with performance on various tasks, validating its utility as an indirect measure of model efficacy in the quantized setting.
• Outlier Weight Isolation: The paper highlights the significance of isolating outlier weights for extreme quantization levels (e.g., 2 bits). Methods like SpQR, which effectively manage such weights, perform better at lower precisions compared to alternatives like GPTQ.
• Graphical Hardness and Quantization: The quantized models' efficiency, especially concerning parallel computation, is hampered by current hardware limitations, stressing the need for tailored hardware optimizations for low-precision arithmetic.

Implications and Future Directions

This paper underscores the practicality of deploying quantized LLMs under constrained resources, suggesting that 4-bit quantization offers a commendable balance between efficiency and performance. It also points out the potential of quantized models with larger parameter counts outperforming smaller, non-quantized models given equivalent resource usage. This observation could drive a shift towards optimizing larger models for edge deployments in the future.

Furthermore, the paper hints at unresolved challenges, particularly in efficiently scaling current quantization techniques with existing hardware, suggesting avenues for future research in hardware-aligned algorithmic development.

Conclusion

In conclusion, the paper presents a compelling case for the utility of quantization in efficiently deploying LLMs. It offers robust evidence supporting the viability of lower-bit quantization without substantial performance loss, backed by a comprehensive evaluation framework. The insights drawn from this paper not only inform current practices but also pave the way for continued innovation in AI model compression techniques, potentially influencing future developments in scalable AI technology.