Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data (2410.05078v2)

Abstract: Foundation models are strong data compressors, but when accounting for their parameter size, their compression ratios are inferior to standard compression algorithms. Naively reducing the parameter count does not necessarily help as it deteriorates predictions and, accordingly, compression. We conduct a large-scale empirical study to find a sweet spot where pre-trained vanilla transformers can achieve competitive compression ratios. To this end, we train models on 165GB of raw byte sequences of either text, image, or audio data (and all possible combinations of the three) and then compress 1GB of out-of-distribution (OOD) data from each modality. We find that relatively small models (millions of parameters) can outperform standard general-purpose compression algorithms (gzip, LZMA2) and even domain-specific compressors (PNG, JPEG-XL, FLAC) $\unicode{x2013}$ even when accounting for parameter size. We achieve, e.g., the lowest compression ratio of 0.49 on OOD audio data (vs. 0.54 for FLAC). We conduct extensive ablations and hyperparameter sweeps to study the impact of model- and dataset scale, and we investigate the effect of unimodal versus multimodal training. We find that even small models can be trained to perform well on multiple modalities, but unlike large-scale foundation models, transfer to unseen modalities is generally weak.

• The paper demonstrates that relatively small pre-trained transformers can outperform standard and specialized compressors by achieving competitive compression ratios on multimodal data.
• It reveals that multimodal training preserves performance across modalities by only slightly diminishing unimodal results while enhancing overall versatility.
• It highlights complex scaling dynamics and context window trade-offs, underscoring the need for future improvements in computational efficiency and cross-modal transfer.

Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data

The paper explores the capability of pre-trained transformers, particularly smaller-scale models, as competitive data compressors across various data modalities. The work seeks to determine whether these models can achieve compression ratios surpassing standard algorithms, including both general-purpose and domain-specific compressors, even when accounting for model size.

Key Findings

1. Model Efficacy with Reduced Parameters: The research demonstrates that relatively small transformer models, with millions of parameters, can outperform traditional compression tools like gzip and LZMA2, and even specialized compressors like FLAC for audio data. Notably, they achieved a compression ratio of 0.49 on out-of-distribution (OOD) audio data, compared to 0.54 for FLAC.
2. Multimodal Training: The paper indicates that training on multimodal data only slightly diminishes the performance on individual modalities compared to unimodal training. However, it significantly enhances capabilities in compressing multimodal data, provided all evaluation modalities are present in the training mixture. Despite expectations based on large-scale models, transfer to entirely unseen modalities was minimal.
3. Scaling Dynamics: Results also show a complex relationship between model size, dataset size, and training compute budget. Optimal performance emerges when both model and dataset scales are increased systematically, reflecting principles observed in known scaling laws for LLMs.
4. Context Window Trade-offs: The research investigates the balance between model size and context size. It finds modality-specific preferences, with text compression benefiting from smaller contexts and larger models, whereas image data favors larger context windows. This highlights the intricacies involved in tuning transformers for diverse data types.

Implications and Future Directions

The findings suggest that pre-trained transformers, even with relatively small configurations, hold significant promise for data compression tasks across multiple domains. This capability might be harnessed to create more versatile and efficient compression systems in the future, moving beyond traditional algorithmic approaches.

However, these models presently come with a substantial computational footprint, making them impractical when considering real-world efficiency metrics such as processing speed and computational resources. Future work might focus on reducing these computational demands, possibly through more efficient model architectures or advanced quantization methods.

Moreover, the limited transfer to unseen modalities indicates a potential area for further exploration. Enhancements in cross-modal transfer could dramatically expand the applicability of these models, mirroring some of the broader ambitions seen in foundational AI research, such as constructing more generalized and adaptive AI systems.

Conclusion

The paper provides significant insights into the compression capabilities of pre-trained transformers, particularly within a constrained parameter regime. While these models are not yet practical replacements for current algorithms in terms of runtime efficiency, they offer a glimpse into the potential for future breakthroughs in universal compression methodologies that might simplify integration across diverse data environments.