Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression (2403.15447v3)

Abstract: Compressing high-capability LLMs has emerged as a favored strategy for resource-efficient inferences. While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected. This study conducts the first, thorough evaluation of three (3) leading LLMs using five (5) SoTA compression techniques across eight (8) trustworthiness dimensions. Our experiments highlight the intricate interplay between compression and trustworthiness, revealing some interesting patterns. We find that quantization is currently a more effective approach than pruning in achieving efficiency and trustworthiness simultaneously. For instance, a 4-bit quantized model retains the trustworthiness of its original counterpart, but model pruning significantly degrades trustworthiness, even at 50% sparsity. Moreover, employing quantization within a moderate bit range could unexpectedly improve certain trustworthiness dimensions such as ethics and fairness. Conversely, extreme quantization to very low bit levels (3 bits) tends to reduce trustworthiness significantly. This increased risk cannot be uncovered by looking at benign performance alone, in turn, mandating comprehensive trustworthiness evaluation in practice. These findings culminate in practical recommendations for simultaneously achieving high utility, efficiency, and trustworthiness in LLMs. Code and models are available at https://decoding-comp-trust.github.io.

Decoding Compressed Trust: Insights into the Trustworthiness of Compressed LLMs

The paper "Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression" presents a comprehensive evaluation of the effects of model compression on the trustworthiness of LLMs. While compression seeks to manage the size and efficiency of LLMs—enabling their broader deployment—the impact on model safety and trustworthiness has been underexplored. The authors delve into this gap by evaluating three leading LLMs using five state-of-the-art compression techniques across eight critical trust dimensions.

Experimental Evaluation

The evaluation sets out to understand the implications of compressing LLMs into smaller, more efficient models, particularly focusing on the balance between enhancing utility and preserving model trustworthiness. The diversity in model architectures and compression algorithms provides a detailed landscape for examining these trade-offs.

1. Quantization vs. Pruning: The paper finds that quantization is more effective than pruning in maintaining the trustworthiness of LLMs. A specific finding is that a 4-bit quantized model closely retains the trustworthiness of its original counterpart, whereas pruning at even 50% sparsity results in significant trust deterioration. Such findings underscore the necessity to prioritize quantization techniques when aiming for efficient and reliable LLMs.
2. Compression Impact on Trust Dimensions: The authors discover that compressing LLMs does not uniformly impact all aspects of trustworthiness. Moderate quantization can improve certain dimensions, notably ethics and fairness, whereas extreme quantization to very low bit levels jeopardizes these trust areas. The paper thus suggests a complex interaction where compression influences different trust dimensions in varying ways.
3. Refinement and Calibration: The significant variance in model trustworthiness based on the calibration set further energizes the discussion around compression practices. The authors highlight the unpredictable nature of trust attributes post-compression, suggesting comprehensive evaluation protocols prior to deploying such models.

Insights and Implications

The implications of these findings span theoretical, practical, and future AI development domains:

• Theoretical: The intricate interplay between compression techniques and trust attributes suggests underlying model characteristics and behaviors that warrant further scrutiny. As models contract in size, their expressive capabilities in different trust dimensions exhibit non-linear scaling which requires deeper understanding.
• Practical: Practitioners are advised to adopt moderate quantization methods to strike a balance between efficiency and trust reliability—facilitating safer deployment in consumer-grade devices.
• Future Developments: The paper sets the stage for future investigations into scalable models that do not compromise on trustworthiness. Researchers are encouraged to explore new compression algorithms that prioritize safety and ethical fidelity, especially as LLMs are integrated into high-stakes applications.

Conclusion

This research uncovers the dual-edged sword of compression in LLM applications, illuminating the complex interdependencies affecting model trustworthiness. By identifying optimal compression pathways and providing insights into potential risks, this paper profoundly contributes to the ongoing discourse on how to responsibly harness the power of AI while safeguarding its deployment. The authors propose practical recommendations for synthesizing efficiency with trustworthiness, serving as a guide for future efforts in creating truly trustworthy AI systems.