Ranking LLMs by compression (2406.14171v1)

Abstract: We conceptualize the process of understanding as information compression, and propose a method for ranking LLMs based on lossless data compression. We demonstrate the equivalence of compression length under arithmetic coding with cumulative negative log probabilities when using a LLM as a prior, that is, the pre-training phase of the model is essentially the process of learning the optimal coding length. At the same time, the evaluation metric compression ratio can be obtained without actual compression, which greatly saves overhead. In this paper, we use five LLMs as priors for compression, then compare their performance on challenging natural language processing tasks, including sentence completion, question answering, and coreference resolution. Experimental results show that compression ratio and model performance are positively correlated, so it can be used as a general metric to evaluate LLMs.

An Examination of LLM Evaluation Through Compression Metrics

The paper entitled "Ranking LLMs by compression" introduces a novel approach to evaluating LLMs based on their capacity for lossless data compression. By positing the conceptual equivalence between understanding and information compression, it provides a framework to assess LLMs' performance more uniformly across various natural language processing tasks. The proposed methodology considers the compression ratio as a metric, hypothesizing that model performance is positively correlated with the compression efficiency.

Key Contributions

1. Equivalence of Compression and Model Training: A major theoretical contribution of this paper is the demonstration of the equivalence between the compression length under arithmetic coding and the model's pre-training goals. The authors argue that the negative log probabilities used in model training essentially mirror the arithmetic coding process, officially linking LLM understanding to efficient data representation.
2. A Novel Evaluation Metric: The paper proposes using the compression ratio as a generic evaluation metric for LLMs. They argue that, unlike traditional metrics which are task-specific, compression ratios offer a more unified measure of a model’s generalization ability, thus simplifying comparative evaluations across different NLP tasks.
3. Practical Implementation: The paper applies its methodology to five popular LLMs—LLaMA 2 7B, Mistral 7B, OPT-IML 1.3B, GPT-2-XL 1.5B, and GPT-2 774M—using the Text8 dataset to calculate compression ratios. The subsequent application of these models to three NLP tasks, namely sentence completion, question answering, and coreference resolution, highlights the practical applicability and validity of the compression ratio as a performance metric.

Numerical Findings

• The experimentation results underscore the positive correlation between compression ratios and model performance on NLP tasks, with algorithms yielding higher compression ratios generally performing better across the board.
• Notably, models like Mistral 7B, showcasing a superior compression ratio of 9.266, also achieved high accuracy in sentence completion tasks compared to others.

Implications and Speculation

The implications of this research extend to both the practical and theoretical realms:

• Theoretically, it fortifies the understanding that LLMs' ability to efficiently compress data indicates their robust generalization and comprehension capabilities.
• Practically, the adoption of compression ratios as a universal metric could streamline LLM evaluation, potentially influencing benchmarking standards and guiding model development.

Looking towards the future, this paper's findings suggest that as LLMs evolve, focusing on improving their intrinsic ability to compress information may yield enhancements in overall performance across diverse applications. Advances in neural compression techniques could further refine such metrics, suggesting an avenue for ongoing research exploration.

Conclusion

This paper establishes a credible link between compression efficiency and model comprehension in LLMs, proposing a shift towards using compression ratios as a holistic performance metric. By integrating arithmetic coding principles and focusing on lossless data compression, the research advocates for a reformed perspective in comparing LLMs, potentially simplifying cross-task evaluations and fostering the development of more versatile LLMs.