An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Abstract: The LLaMA family, a collection of foundation LLMs ranging from 7B to 65B parameters, has become one of the most powerful open-source LLMs and the popular LLM backbone of multi-modal LLMs (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality. Our project is released on https://github.com/Macaronlin/LLaMA3-Quantization , and quantized models are released at https://huggingface.co/Efficient-ML .

• The paper demonstrates that PTQ methods like GPTQ and AWQ enable LLaMA3 to retain reasonable performance at 3-bit configurations.
• The study finds that LoRA-Finetuning quantization yields limited improvements and sometimes underperforms compared to non-fine-tuned approaches.
• Implications highlight the need for advanced quantization strategies to address performance degradation in resource-limited deployments of LLaMA3.

Performance Evaluation of Low-Bit Quantization on Meta's LLaMA3 Models

Introduction

Meta's LLaMA3 models, introduced in April 2024, represent a significant advancement in the field of LLMs, boasting configurations of up to 70 billion parameters and extensive pre-training on over 15 trillion tokens. Despite their superior performance across various benchmarks, the real-world application of LLaMA3 models is often restricted by resource limitations, prompting a closer examination of low-bit quantization methods as a viable solution for compression. This study evaluates the effectiveness of different low-bit quantization techniques, both post-training and during fine-tuning, to maintain the operational integrity of LLaMA3 models under resource constraints.

Quantization Techniques Evaluated

The study categorizes the quantization methods into two main tracks:

1. Post-Training Quantization (PTQ)
   • Techniques such as RTN, GPTQ, AWQ, and SmoothQuant were tested across a bit-width spectrum from 1 to 8 bits.
   • Notable methods like PB-LLM and DB-LLM employ strategies for effective compression at ultra-low bit-widths, revealing promising capabilities in maintaining performance integrity.
2. LoRA-Finetuning (LoRA-FT) Quantization
   • Focused on the newer methods such as QLoRA and IR-QLoRA, the study explores adaptations of the model parameters during fine-tuning to achieve better quantization outcomes.
   • These methods were primarily evaluated on the MMLU benchmark and additional CommonSenseQA tasks to assess their capacity to handle lower-bit operational demands.

Experimental Results

• PTQ Evaluation: A comprehensive assessment using various benchmarks showed that while some methods like GPTQ and AWQ can maintain reasonable model performance down to 3-bit quantization, nearly all techniques faced severe degradation at ultra-low bit-widths (1-2 bits). However, specialized methods like PB-LLM introduced mixed-precision strategies that somewhat mitigated performance drops.
• LoRA-FT Evaluation: The findings indicate that LoRA-FT methods did not substantially improve the performance outcomes for the LLaMA3 model, especially when compared against their non-fine-tuned counterparts. In some cases, these methods performed worse, underscoring the challenges of applying low-rank adjustments to highly optimized models.

Implications and Future Directions

The observed performance degradation in low-bit scenarios highlights a critical challenge for deploying LLaMA3 in resource-limited environments. This issue prompts further research into developing more robust quantization techniques that can effectively bridge the performance gap identified in this study. Future advancements might focus on:

• Enhancing PTQ methods to support lower-bit operations without a substantial loss in accuracy.
• Innovating LoRA-FT approaches that can leverage the intrinsic capacity of LLaMA3 models more effectively, perhaps through more sophisticated parameter optimization or adjustment techniques.

Ultimately, by improving the efficacy of these quantization methods, LLMs like LLaMA3 could be deployed more widely, extending their utility to a variety of applications where computational resources are a limiting factor.