ARB-LLM: Alternating Refined Binarizations for Large Language Models (2410.03129v2)

Abstract: LLMs have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$\text{X}$ and ARB-LLM$\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.

Overview of ARB-LLM: Alternating Refined Binarizations for LLMs

The paper "ARB-LLM" presents a novel approach to enhancing binarization in LLMs to address their high computational and memory demands. Binarization compresses model weights to one bit, offering a promising solution for deploying LLMs in resource-constrained environments. Despite its potential, traditional binarization methods often encounter significant challenges, particularly in aligning the distribution of binarized weights with that of their full-precision counterparts. This misalignment, along with column deviation in weight distribution, poses obstacles to achieving efficient model performance.

Proposed Method: ARB-LLM

The authors introduce ARB-LLM, an innovative 1-bit post-training quantization (PTQ) technique specifically designed for LLMs. The core component is the Alternating Refined Binarization (ARB) algorithm, which iteratively updates binarization parameters to minimize quantization errors. Notably, the algorithm ensures a more accurate representation of the original full-precision weights by addressing distribution shifts.

To further enhance the performance, the paper extends ARB into two variants: ARB-X and ARB-RC. These models incorporate calibration data and account for row and column deviations in the weight matrix. The introduction of a column-group bitmap (CGB) for refining weight partitioning elevates binarization outcomes, leading to ARB-LLM and ARB-XRCRLLM models.

Numerical Results and Contributions

Experimentation demonstrates that ARB-LLM outperforms state-of-the-art (SOTA) methods in binarization, even surpassing FP16 models of equivalent size. The paper provides rigorous theoretical analysis supporting the reduction in quantization error through iterative updates. Furthermore, ARB-LLM emerges as the first binary PTQ method to exceed FP16 models on zero-shot question answering datasets in terms of accuracy.

Key contributions of this work include:

• Algorithmic Innovation: ARB, ARB-X, and ARB-RC significantly enhance binarization precision by progressively aligning with full-precision weight distributions.
• Efficiency and Scalability: The implementation effectively lowers computational costs and memory usage, essential for deploying LLMs in mobile and edge devices.
• Advanced Extensions: Tailored solutions for leveraging calibration data and accommodating weight distribution peculiarities lead to substantial performance gains.

Implications and Future Directions

The implications of ARB-LLM on the practical deployment of LLMs are profound. By advancing weight binarization techniques, this method opens up pathways for broadening the reach of complex models into less powerful hardware environments, facilitating wider application in real-time scenarios.

Theoretically, this paper sets a precedent for addressing weight distribution alignment, suggesting potential future explorations into refining quantization approaches further. Upcoming research might focus on extending these techniques to other neural architectures or adapting them to integrate seamlessly with dynamic learning tasks.

Conclusively, ARB-LLM represents a significant methodological advancement in LLM quantization, laying groundwork for both theoretical innovations and practical applications in artificial intelligence.