PB-LLM: Partially Binarized Large Language Models (2310.00034v2)

Abstract: This paper explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for LLMs compression. Due to previous binarization methods collapsing LLMs, we propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs. Specifically, our exploration first uncovers the ineffectiveness of naive applications of existing binarization algorithms and highlights the imperative role of salient weights in achieving low-bit quantization. Thus, PB-LLM filters a small ratio of salient weights during binarization, allocating them to higher-bit storage, i.e., partially-binarization. PB-LLM is extended to recover the capacities of quantized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-aware training (QAT). Under PTQ, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. Under QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose a scaling mechanism based on this derived scaling strategy for residual binarized weights. Those explorations and the developed methodologies significantly contribute to rejuvenating the performance of low-bit quantized LLMs and present substantial advancements in the field of network binarization for LLMs.The code is available at https://github.com/hahnyuan/BinaryLLM.

Partially Binarized LLMs: A Comprehensive Examination

The proliferation of LLMs like GPT, BERT, and their variants have propelled advancements across various domains in artificial intelligence. These models, empowered by tens or hundreds of billions of parameters, often herald impressive performance but are constrained by their significant memory and computational demands. This has ignited interest in various model compression techniques, among which weight quantization holds particular significance. The paper “PB-LLM: Partially Binarized LLMs” proposes a novel compression methodology employing a partially binarized approach to LLMs, potentially bridging the gap between extreme quantization and performance retention.

Methodology Synopsis

The paper challenges the conventional quantization paradigm, which traditionally reduces weights to a uniformly low bit-width. It introduces the concept of Partially-Binarized LLM (PB-LLM), a two-phase strategy where the majority of model weights are binarized to one bit while maintaining a small fraction of salient weights in higher precision. The identification of salient weights, determined primarily by magnitude, plays a crucial role in this selective storage strategy. This is grounded in recognizing that certain weights disproportionately contribute to the model's overall reasoning capabilities.

The binarization framework is investigated through two primary avenues: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). These avenues aid in delineating strategies to recover and maintain the model's linguistic reasoning ability post-binarization.

1. PTQ with PB-GPTQ: This method extends the GPTQ framework to a partially binarized setting. The technique quantizes weights iteratively, appending a compensation mechanism to the non-salient weights after binarization. The exploration of PB-GPTQ unveils its efficacy in quantifying models like OPT-1.3B and LLaMA-7B with preserved reasoning capability, conditional on a judicious choice of salient weight ratios.
2. QAT Strategies: Implementing Quantization-Aware Training involves freezing salient weights and employing optimally derived scaling factors from binarized weights. These strategies not only facilitate efficient post-binarization training by conserving computational overhead but also underscore the rapid convergence capabilities intrinsic to PB-LLM.

Empirical Evaluation and Results

The paper substantiates its claims by subjecting PB-LLM to a series of rigorous evaluations on renowned tasks, including BoolQ, PIQA, and more, assessing both its reasoning prowess and generalization abilities. Notably, the results indicate that by retaining even a modest percentage of salient weights, the PB-LLM can achieve performances akin to higher-bit settings, with the appropriate scaling and frozen weight strategies employed adeptly.

Key empirical benchmarks on common sense reasoning tasks reveal that PB-LLM compares favorably against other recent methodologies in the 4-bit quantization domain. The PB-GPTQ framework, despite extreme low-bit conditions, offers a promising balance between reduced bit-widths and retained factual reasoning capability, especially with quantization-aware training further enhancing outcomes.

Implications and Future Trajectories

The implications continue to unfold, both in the capacity of LLMs to operate efficiently in resource-constrained environments and in expediting deployment in real-world applications. Practically, the methodologies foster promising avenues for reducing the operational costs of deploying advanced AI systems on edge devices. Theoretically, this work may inaugurate further research into granular quantization strategies and latent-sparse learning paradigms that bolster efficient AI.

In conclusion, the PB-LLM framework carves a significant niche in the evolving landscape of machine learning model optimization. Through astute implementation and insightful innovations, this research lays the groundwork for subsequent explorations in network binarization, potentially catalyzing a more efficient integration of LLMs into myriad applications globally. This work, while ground-breaking in its present form, also gestures toward uncharted explorations in leveraging binarization for deep language understanding in the realms of artificial intelligence.