Compressed-Language Models for Understanding Compressed File Formats: a JPEG Exploration (2405.17146v1)

Abstract: This study investigates whether Compressed-LLMs (CLMs), i.e. LLMs operating on raw byte streams from Compressed File Formats~(CFFs), can understand files compressed by CFFs. We focus on the JPEG format as a representative CFF, given its commonality and its representativeness of key concepts in compression, such as entropy coding and run-length encoding. We test if CLMs understand the JPEG format by probing their capabilities to perform along three axes: recognition of inherent file properties, handling of files with anomalies, and generation of new files. Our findings demonstrate that CLMs can effectively perform these tasks. These results suggest that CLMs can understand the semantics of compressed data when directly operating on the byte streams of files produced by CFFs. The possibility to directly operate on raw compressed files offers the promise to leverage some of their remarkable characteristics, such as their ubiquity, compactness, multi-modality and segment-nature.

• The paper demonstrates that CLMs accurately recognize JPEG file properties, achieving 100% accuracy in quality detection.
• The model reliably identifies and corrects single-byte anomalies in JPEG files with over 95% accuracy.
• Greedy decoding enables CLMs to generate visually coherent JPEG images with a success rate between 97% and 99%.

An Examination of the Capabilities of Compressed-LLMs on JPEG Data

In this paper, the authors evaluate whether Compressed-LLMs (CLMs)—LLMs trained for next-token prediction directly on raw byte streams produced by Compressed File Formats (CFFs)—can comprehend and manipulate the byte streams of JPEG encoded files. The paper is geared toward investigating three main tasks: recognizing file properties, handling files with anomalies, and generating new files. The choice of JPEG format is motivated by its ubiquity, representativeness of key compression concepts, and the ease with which generated outputs can be evaluated through visual inspection.

Introduction

CFFs are prevalent in modern computing due to their efficiency in storing and transmitting data. However, unlike traditional formats that are directly readable, CFFs require decoding to revert the data to its usable form. The paper proposes leveraging the language-like properties of CFFs to train LLMs directly on these byte streams. This approach could grant significant advantages such as universality, compactness, and the capability to bypass large-scale redundancy reduction techniques commonly employed in models trained on uncompressed data.

Preliminaries

JPEG is selected for its widespread use and because it exemplifies fundamental concepts of compression: lossy compression, quantization, and entropy coding. Despite these attributes, the paper acknowledges several challenges when modeling JPEG files. These include sensitivity to modifications, bit-level operations like Huffman coding, and complex dependencies introduced by techniques such as the Discrete Cosine Transform (DCT) and quantization.

Methodology

The authors evaluate the model's understanding capabilities across three axes:

1. File Recognition: The model is tested for its ability to recognize the properties of compressed files such as JPEG quality and semantic class from the byte sequence.
2. File Anomaly Handling: The model's capacity to detect, locate, and correct anomalies in JPEG files is assessed.
3. File Generation: The ability to generate new, visually coherent JPEG files while maintaining specified quality parameters is evaluated.

The input vocabulary consists of 256 byte values with additional tokens indicating JPEG quality and semantic class. For training, the model employs a decoder-only Transformer architecture optimized for next-token prediction.

Experimental Results

File Recognition

The model achieves high accuracy in recognizing both JPEG quality and semantic class, particularly excelling at recognizing JPEG quality with 100% accuracy. Semantic class recognition shows room for improvement, especially on more complex datasets like CIFAR-10, where accuracy can be enhanced through fine-tuning.

File Anomaly Handling

For anomaly detection, the model shows remarkable sensitivity to even single-byte perturbations, consistently assigning higher likelihoods to original files. The accuracy in locating anomalies within a file exceeds 95% for broken files, and the model proves capable of correcting these anomalies effectively. These capabilities underscore the model's fine-grained understanding of JPEG byte streams.

File Generation

Sampling from the model with greedy decoding results in a 97-99% success rate for generating valid JPEG files with the correct quality parameter. The visual quality of the generated images further validates the model's competence in generating coherent data consistent with the provided prompts.

Discussion

The findings indicate that vanilla decoder-only Transformer models can effectively handle compressed byte streams like those of JPEGs. This capability manifests in strong performance across file recognition, anomaly handling, and file generation tasks. The implications are far-reaching: models trained on compressed formats can harness the compactness and efficiency of these formats, opening avenues for more efficient data handling in various AI applications.

Future research should explore generalizing these findings to other CFFs such as MP3 or ZIP, and evaluate the performance of larger, more complex models on a broader spectrum of compressed data types. Additionally, combining this approach with advances in byte-level modeling architectures, such as MegaByte or bGPT, could further enhance the capabilities and efficiency of CLMs.

Conclusion

The paper establishes a robust framework for understanding and manipulating JPEG files directly from their compressed byte streams using LLMs. The demonstrated proficiency of CLMs in tasks such as file recognition, anomaly handling, and generation, underscores the potential of leveraging these models in practical and theoretical AI domains. The approach introduces a versatile and efficient method for handling a wide range of compressed data formats, making significant strides in the field of generative AI.