BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration (2411.11745v2)

Abstract: LLMs have demonstrated remarkable performance across various machine learning tasks. Yet the substantial memory footprint of LLMs significantly hinders their deployment. In this paper, we improve the accessibility of LLMs through BitMoD, an algorithm-hardware co-design solution that enables efficient LLM acceleration at low weight precision. On the algorithm side, BitMoD introduces fine-grained data type adaptation that uses a different numerical data type to quantize a group of (e.g., 128) weights. Through the careful design of these new data types, BitMoD is able to quantize LLM weights to very low precision (e.g., 4 bits and 3 bits) while maintaining high accuracy. On the hardware side, BitMoD employs a bit-serial processing element to easily support multiple numerical precisions and data types; our hardware design includes two key innovations: First, it employs a unified representation to process different weight data types, thus reducing the hardware cost. Second, it adopts a bit-serial dequantization unit to rescale the per-group partial sum with minimal hardware overhead. Our evaluation on six representative LLMs demonstrates that BitMoD significantly outperforms state-of-the-art LLM quantization and acceleration methods. For discriminative tasks, BitMoD can quantize LLM weights to 4-bit with $<!0.5\%$ accuracy loss on average. For generative tasks, BitMoD is able to quantize LLM weights to 3-bit while achieving better perplexity than prior LLM quantization scheme. Combining the superior model performance with an efficient accelerator design, BitMoD achieves an average of $1.69\times$ and $1.48\times$ speedups compared to prior LLM accelerators ANT and OliVe, respectively.

• The paper proposes BitMoD, an algorithm-hardware co-design that reduces quantization error using fine-grained data type adaptation.
• The paper achieves sub-0.5% accuracy loss on 4-bit weights and improves generative task perplexity at 3-bit precision compared to state-of-the-art methods.
• The paper demonstrates 1.69x and 1.48x speed-ups over ANT and OliVe accelerators, highlighting enhanced energy efficiency and compact area-power trade-offs.

BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

The paper presents BitMoD, an innovative algorithm-hardware co-design approach addressing the challenges of LLMs deployment by enhancing both memory efficiency and computational acceleration. BitMoD focuses on optimizing the weight quantization process to achieve low weight precision without sacrificing accuracy, which is increasingly important given the substantial memory and computation demands of modern LLMs.

The crux of the BitMoD approach lies in its novel use of "fine-grained data type adaptation" and its synergy with hardware design, employing a bit-serial processing element to support various precisions effectively. The quantization strategy offers innovative methods for encoding LLM weights using low-precision formats by repurposing the redundant zero values in traditional floating-point encodings into special values, reducing the quantization error significantly. This flexibility allows BitMoD to operate on 4-bit and 3-bit weight precisions while maintaining competitive accuracy — an approach that proves beneficial across both discriminative and generative tasks.

The authors demonstrate that BitMoD can achieve a sub-0.5% accuracy loss for typical discriminative tasks when quantizing weights to 4-bit values. Notably, BitMoD quantization achieves better perplexity for generative tasks compared to previous state-of-the-art quantization methods while operating at 3-bit precision. These impressive results highlight BitMoD's robustness in quantizing variable weight distribution by effectively integrating extra resolution and asymmetry into floating-point data types.

In terms of hardware realization, the BitMoD accelerator leverages bit-serial computation to efficiently articulate between computational precision and hardware efficiency. This is achieved through a flexible processing element design that performs mixed-precision arithmetic. The proposed architecture displays a prominent throughput advantage and energy efficiency, achieving 1.69x and 1.48x speed-ups in comparison to the state-of-the-art LLM accelerators, ANT and OliVe. Furthermore, BitMoD facilitates a compelling area-power trade-off via efficient packaging of unified bit-serial representations, outperforming alternative decomposable bit-parallel configurations.

The implication of BitMoD extends further when considering its compatibility with additional quantization optimization techniques like AWQ and OmniQuant, suggesting a potential for compound improvements in model performance. The ability to adapt weight precision further offers utility in resource-constrained environments, enhancing the deployability of LLMs on devices with limited compute and memory resources.

For future exploration, this paper prompts further investigation into extending the adaptability of the customized data types to new model architectures and exploring dynamic quantization mechanisms that continually optimize precision during runtime. Additionally, implementing such a design in edge applications with varying input characteristics could test BitMoD's adaptability and uncover more diverse applications.

In summary, the paper provides a compelling blueprint for low-precision LLM acceleration, emphasizing an insightful combination of algorithmic novelty and hardware efficiency. By breaking down LLM barriers through rigorous quantization strategy and innovative hardware design, BitMoD represents a significant advancement in scaling the reach of sophisticated linguistic models to broader and more constrained computational platforms.