VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2409.17066v2)

Abstract: Scaling model size significantly challenges the deployment and inference of LLMs. Due to the redundancy in LLM weights, recent research has focused on pushing weight-only quantization to extremely low-bit (even down to 2 bits). It reduces memory requirements, optimizes storage costs, and decreases memory bandwidth needs during inference. However, due to numerical representation limitations, traditional scalar-based weight quantization struggles to achieve such extreme low-bit. Recent research on Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely low-bit model quantization by compressing vectors into indices using lookup tables. In this paper, we introduce Vector Post-Training Quantization (VPTQ) for extremely low-bit quantization of LLMs. We use Second-Order Optimization to formulate the LLM VQ problem and guide our quantization algorithm design by solving the optimization. We further refine the weights using Channel-Independent Second-Order Optimization for a granular VQ. In addition, by decomposing the optimization problem, we propose a brief and effective codebook initialization algorithm. We also extend VPTQ to support residual and outlier quantization, which enhances model accuracy and further compresses the model. Our experimental results show that VPTQ reduces model quantization perplexity by $0.01$-$0.34$ on LLaMA-2, $0.38$-$0.68$ on Mistral-7B, $4.41$-$7.34$ on LLaMA-3 over SOTA at 2-bit, with an average accuracy improvement of $0.79$-$1.5\%$ on LLaMA-2, $1\%$ on Mistral-7B, $11$-$22\%$ on LLaMA-3 on QA tasks on average. We only utilize $10.4$-$18.6\%$ of the quantization algorithm execution time, resulting in a $1.6$-$1.8\times$ increase in inference throughput compared to SOTA.

Summary of "VPTQ: Extreme Low-bit Vector Post-Training Quantization for LLMs"

The paper "VPTQ: Extreme Low-bit Vector Post-Training Quantization for LLMs" introduces a novel method for efficient model compression in LLMs. The approach focuses on Vector Post-Training Quantization (VPTQ), aiming for extremely low-bit quantization using vector quantization (VQ) techniques.

Key Contributions

1. Second-Order Optimization: The authors apply second-order optimization to guide the quantization process, enabling high-compression ratios while retaining model performance. The optimization problem is formulated to minimize quantization errors by focusing on channel-independent quantization.
2. Granular Quantization: VPTQ extends traditional VQ to support residual and outlier quantization, contributing to a fine-grained weight adjustment. The approach provides a significant reduction in perplexity over state-of-the-art (SOTA) methods at extreme low-bit levels, such as 2-bit quantization.
3. Algorithm Efficiency: The design minimizes computational complexity. The proposed codebook initialization algorithm expedites the process, and the residual vector quantization method controls error propagation, enhancing both model accuracy and compression efficacy.

Experimental Results

• VPTQ demonstrated a reduction in model perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B, and 4.41-7.34 on LLaMA-3, in comparison to SOTA at 2-bit quantization.
• The method improves average accuracy on QA tasks by 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, and a notably higher 11-22% on LLaMA-3.
• VPTQ achieves these results with 10.4-18.6% of the execution time required by existing methods, offering a throughput improvement of 1.6-1.8x during inference.

Methodology

• Channel-Independent Second-Order Optimization: By quantizing each column of matrices independently, the method reduces error accumulation, which is a limitation in traditional VQ.
• Residual and Outlier Quantization: These strategies allow a more efficient representation and compression of weights by accommodating deviations introduced by quantization, reducing the impact of outlier values, and addressing high-magnitude weight components separately.
• Hessian-Weighted Centroid Initialization: Optimizes the initialization of centroids to minimize errors by considering the Hessian's diagonal elements during quantization.

Implications and Future Directions

The VPTQ method introduces a promising pathway for deploying LLMs in resource-constrained environments by significantly reducing memory and computation overhead without major sacrifices in performance. This opens avenues for real-time applications across various domains, including mobile and edge computing, where computational resources are limited.

The strategy of leveraging second-order optimization and exploiting vector quantization can be further explored and potentially enhanced with sophisticated algorithms for reducing weight redundancy. Future work might explore integrating these quantization techniques with emerging hardware accelerations to further enhance speed and efficiency. Moreover, extending such quantization strategies to other modalities, such as vision and multimodal models, could expand the applicability and influence of these methodologies.

In conclusion, the VPTQ framework sets a new benchmark in the domain of model quantization, offering a balance between compression and accuracy—a critical requirement in the continued journey towards scalable and efficient artificial intelligence deployment.