An Introduction to Neural Data Compression (2202.06533v3)

Abstract: Neural compression is the application of neural networks and other machine learning methods to data compression. Recent advances in statistical machine learning have opened up new possibilities for data compression, allowing compression algorithms to be learned end-to-end from data using powerful generative models such as normalizing flows, variational autoencoders, diffusion probabilistic models, and generative adversarial networks. The present article aims to introduce this field of research to a broader machine learning audience by reviewing the necessary background in information theory (e.g., entropy coding, rate-distortion theory) and computer vision (e.g., image quality assessment, perceptual metrics), and providing a curated guide through the essential ideas and methods in the literature thus far.

• The paper presents a comprehensive overview of neural data compression, detailing both lossless and lossy methods enhanced by neural networks.
• It explains how neural models like autoregressive and latent variable approaches improve entropy coding and optimize the rate-distortion trade-off.
• The study outlines emerging strategies such as quantization-free compression and perceptual losses, paving the way for future research in efficient data encoding.

Understanding Neural Data Compression

Introduction

Data compression is about efficiently reducing the number of bits needed to represent information. Neural compression, also called learned compression, leverages neural networks and machine learning techniques to enhance this process. Recent advances in generative models such as GANs, VAEs, normalizing flows, and diffusion models have significantly impacted this field. This article dives into the key concepts and methods in neural data compression, both lossless and lossy, explaining the major developments and implications.

Lossless Compression

Fundamentals

Lossless compression aims to represent data as compactly as possible without losing any information—that is, the original data can be perfectly reconstructed. The core tools of lossless compression are entropy coding techniques like Huffman coding, arithmetic coding, and asymmetric numeral systems (ANS).

1. Entropy and Information: Entropy measures the uncertainty or information content in a data source. Coding schemes aim to minimize the expected message length, which ideally is close to the entropy of the data.
2. Huffman Coding: A well-known prefix-free code that assigns shorter codes to more frequent symbols. Although efficient, it incurs a bit overhead due to its symbol-by-symbol approach.
3. Arithmetic Coding: More efficient for sequences, this method assigns intervals to symbols based on their probabilities. Encoding a message compresses it closer to its entropy, reducing overhead.
4. ANS: This modern method operates like a stack, differing from arithmetic coding’s queue approach. It efficiently handles non-uniform distributions with low computational overhead.

Neural Techniques

Neural networks enhance traditional methods by modeling high-dimensional data distributions more accurately.

• Autoregressive Models: These models predict each data point conditioned on previous ones, making them suitable for sequence data (e.g., texts).
• Latent Variable Models: The data distribution is modeled using hidden variables, with VAEs being a prominent example. Bits-back coding leverages this by encoding auxiliary bits alongside the main data.

Lossy Compression

Fundamental Concepts

Lossy compression accepts some fidelity loss for higher compression rates. The balance between bit-rate (the amount of compression) and distortion (loss of quality) is crucial.

1. Rate-Distortion Theory: This theory provides limits on the performance of compression algorithms. The rate-distortion function quantifies the best achievable trade-offs.
2. Vector Quantization: Groups data into clusters and represents each cluster with a codebook vector. It's efficient but struggles with high-dimensional data.
3. Transform Coding: Transforms data into a space where it's easier to compress. JPEG is a classic example using discrete cosine transform (DCT).

Neural Approaches

Recent methods train neural networks end-to-end to learn optimal transforms and entropy models.

1. Neural Transform Coding: Uses deep networks for the analysis and synthesis transforms, adapting flexibly to data patterns. Common architectures are convolutional neural networks (CNNs) for images.
2. Quantization Techniques: To address non-differentiability, methods like straight-through estimators (STE), additive uniform noise, and stochastic Gumbel annealing provide gradient approximations for backpropagation.
3. Entropy Models: Leveraging powerful models from lossless compression, these models (e.g., autoregressive models, hyperpriors) predict probabilities for efficient entropy coding.
4. Connections to VAEs: Many neural compression models can be framed as VAEs, including rate-distortion VAEs for training network parameters.

Emerging Techniques

Compression Without Quantization

Avoiding quantization altogether, methods like stochastic coding can transmit continuous latent representations efficiently.

• Dithered Quantization: Uses noise addition and subtraction to simulate quantization, making encoding differentiable.
• Minimal Random Coding: Transmits samples from a probabilistic encoder using techniques from importance sampling.

Perceptual Losses

Optimal perceptual quality in reconstructed data is crucial.

1. Metrics like SSIM and MS-SSIM: These evaluate similarity based on structural information rather than pixel-wise differences, aligning better with human perception.
2. Adversarial Models: GANs are employed to ensure realistic and high-quality reconstructions by training discriminators to distinguish between real and compressed data.

Task-Oriented Compression

For data processed primarily by algorithms, task-specific loss functions ensure the compressed representation preserves performance on relevant tasks (e.g., image classification).

Video Compression

Neural video codecs leverage motion compensation and residual compression, often using hierarchical and recurrent models. These methods are evolving to integrate deeper models like transformers for better performance.

Implications and Future Directions

Neural compression offers significant improvements over traditional methods but faces challenges including computational demands and efficient encoding for new data types. Future research will likely focus on:

• Reducing computational overhead for real-world applications.
• Enhancing loss functions for better perceptual quality and realism.
• Expanding neural methods to novel data modalities like 3D content and point clouds.

Neural compression is transforming how we approach data storage and transmission, and continuing advances will further solidify its role in diverse applications from media to scientific data.