LLMZip: Lossless Text Compression using Large Language Models (2306.04050v2)

Abstract: We provide new estimates of an asymptotic upper bound on the entropy of English using the LLM LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the LLM with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.

• The paper introduces a novel lossless compression algorithm that integrates LLaMA-7B predictions with arithmetic coding, achieving 0.7101 bits/character on a 1MB text8 dataset.
• The paper presents new entropy estimates for English at 0.709 bits/character on text8 and 0.85 bits/character on a Project Gutenberg sample, substantially lower than earlier models.
• The paper combines theoretical insights with practical compression techniques, promising significant storage and bandwidth savings for large-scale digital applications.

Utilizing LLaMA-7B for Estimating English Language Entropy and Text Compression

The paper investigates using the LLaMA-7B LLM to provide new estimates for the asymptotic upper bound on the entropy of the English language. This research endeavors to achieve superior performance in lossless text compression by capitalizing on the enhanced predictive capabilities of modern LLMs, specifically focusing on LLaMA-7B. Previous estimates made by Cover and King (1978) and more recent models indicated higher upper bounds, whereas this paper reports significantly lower estimates.

Summary of Contributions

The key contributions of the paper are as follows:

1. Entropy Estimation:
   • The authors estimate the asymptotic upper bound on the entropy of English to be 0.709 bits/character using LLaMA-7B on a 1MB section of the text8 dataset.
   • The estimate changes to 0.85 bits/character when a 100KB text from Project Gutenberg is used.
2. Compression Techniques:
   • The paper introduces a novel lossless compression algorithm that integrates predictions from LLaMA-7B with arithmetic coding.
   • Preliminary experimental results suggest the proposed scheme outperforms leading text compression algorithms like BSC, ZPAQ, and paq8h.
3. Algorithmic Approach:
   • The proposed method leverages the rank-ordered predictions of tokens by LLaMA-7B, computing the ranks of actual next tokens during encoding.
   • For encoding, a combination of standard lossless compression algorithms and arithmetic coding is employed to convert these predictions into compressed bit sequences efficiently.

Detailed Analysis and Interpretation

1. Relationship Between Learning, Prediction, and Compression:

The researchers elucidate the interplay between prediction and compression, grounded in the foundational work of Shannon (1951). By enhancing prediction accuracy using LLMs like LLaMA-7B, the compression algorithm can more effectively reduce redundancy in text. Here, the LLaMA-7B model predicts the next token based on prior context, which directly aids the compression mechanism.

2. Entropy and its Upper Bound:

Estimating the entropy upper bound involves calculating the average amount of information required to predict a token. The researchers' estimate of 0.709 bits/character for the text8 dataset suggests that LLaMA-7B's predictions are significantly more efficient than earlier models. This figure represents a lower bound than Cover and King’s 1.3 bits/character estimate and Lutati et al.’s more recent models, indicating a substantial improvement in predictive efficiency.

3. Compression Algorithms:

The paper proposes three different compression techniques:

• LLaMA+zlib: Uses zlib to compress the sequence of ranks from LLaMA-7B predictions.
• Token-by-Token Compression: Involves creating a prefix-free code, leveraging token probabilities from LLaMA-7B for compression.
• Arithmetic Coding with LLaMA-7B: Merges arithmetic coding with LLaMA’s predictive probabilities to achieve near-optimal compression.

Among these methods, arithmetic coding combined with LLaMA-7B yields the best performance, achieving a compression ratio of 0.7101 bits/character on a 1MB section of the text8 dataset—outperforming state-of-the-art schemes like ZPAQ and paq8h.

Practical and Theoretical Implications

Practical Implications:

The practical applications of this research extend primarily to storage and data transmission. The demonstrated compression efficiency implies significant savings in storage space and bandwidth, particularly in large-scale deployments such as digital libraries, cloud storage services, and real-time communication systems.

Theoretical Implications:

The improved entropy estimate suggests a theoretical advancement in understanding the limits of text predictability. It underscores the potential of LLMs to approach the inherent entropy limit of natural languages more closely than traditional methods.

Future Directions

Future developments in this area can explore several avenues:

• Scaling Up: Evaluating the performance of LLaMA-7B-based compression on larger and more diverse datasets.
• Model Enhancements: Investigating the impact of even larger models and alternative architectures on entropy estimation and compression efficiency.
• Algorithm Optimization: Further refining the integration of arithmetic coding with LLM predictions to push the compression ratios closer to the theoretical limits.
• Cross-linguistic Applications: Extending these methodologies to other languages to understand how language structure influences entropy and compression potential.

In conclusion, the paper provides substantial improvements in estimating the entropy of the English language using LLaMA-7B. The innovative integration of this LLM with lossless compression techniques showcases the enhanced predictive power of modern LLMs and their potential in achieving more efficient text compression.