Equivalence of Compression and Model Training

• Equivalence of Compression and Model Training is defined as the deep connection between data/model compression techniques and predictive training, underpinned by information theory and source coding principles.
• The topic leverages metrics like cross-entropy, bits-per-character, and rate-distortion theory to provide a unified evaluation framework that correlates compression efficiency with model performance.
• It demonstrates that compression-aware training, including model pruning, quantization, and optimal resource trade-offs, guides efficient data curation and improves generalization in modern ML systems.

The equivalence of compression and model training refers to the deep and principled connections—both theoretical and empirical—between the tasks of compressing data or models and the objectives underlying the training and evaluation of modern machine learning systems. This equivalence permeates sequence modeling, vision, model selection, data curation, and model architecture optimization. The topic involves insights from information theory, rate-distortion theory, Shannon’s source coding theorem, empirical trends in neural network performance, and practical techniques for both lossless and lossy compression as well as model parameter reduction.

1. Information-Theoretic Foundations: Compression and Predictive Modeling

The core theoretical insight, formalized by Shannon’s source coding theorem, is that the minimum expected code length for lossless compression of a data source is its entropy. For a sequence $x_{1:n}$ drawn from a probabilistic model $\rho$, the optimal codelength is $-\log_2 \rho(x_{1:n})$ (Delétang et al., 2023). This is operationalized in language and time series modeling by maximizing the log-likelihood (minimizing negative log-likelihood): training via cross-entropy loss is mathematically equivalent to learning a coding distribution that achieves near-optimal compression. This equivalence holds in any scenario where the probabilistic model $P_\theta(x_t \mid x_{<t})$ approximates the data distribution.

For any model $Q_\theta$, the average code length for data $X$ is

$$L^*(X) = -\sum_{t=1}^T \log_2 Q_\theta(x_t \mid x_{<t})$$

which is the model’s likelihood evaluated on the sequence and, via arithmetic or entropy coding, corresponds exactly to the compressed file size (Wan et al., 25 Sep 2025).

2. Compression Metrics as Unsupervised Evaluation and Proxy for Intelligence

Compression-centric evaluation metrics—including bits-per-character (BPC), compression ratio, and bits-per-byte—have emerged as robust, unsupervised proxies for model capability and generalization. For LLMs, empirical studies demonstrate strong positive correlations (Pearson $r \sim -0.94$ to $-0.96$) between compression efficiency and aggregate benchmark scores across knowledge, code, and mathematical reasoning tasks (Huang et al., 15 Apr 2024). The same equivalence extends to time series models, where lossless compression metrics are strictly equivalent to NLL and capture modeling capacity over the complete generative distribution, outperforming task-specific metrics that may not penalize misspecification (Wan et al., 25 Sep 2025).

Table: Evaluation Equivalence

Model Task	Metric	Theoretical Equivalence
Language Modeling	Cross-entropy / NLL	Optimal compression length
Compression	Arithmetic code-length / ratio	Negative log-likelihood on data
Time Series	Bits-per-byte (bpb)	Sequence-level NLL

A significant implication is that compression metrics are task-agnostic, label-free, and resistant to prompt engineering and data contamination. They can be computed directly using model outputs and do not require actual bitstream generation (Guo et al., 20 Jun 2024).

3. Limits and Refinements of Compression–Intelligence Correspondence

Empirical work in code intelligence challenges the universality and the precise form of the compression–intelligence relationship. While prior studies observed an approximately linear relationship between compression and benchmark performance (especially for LLMs in general NLP), comprehensive evaluation on code LLMs across multi-language, multi-task settings reveals a logarithmic relationship:

$$\log C \propto -\text{BPC} \ C \propto e^{-a \cdot \text{BPC}}$$

where $C$ is log-averaged code intelligence and BPC is code compression (bits-per-character) (Luo et al., 16 May 2025). The claimed linearity is shown to be an artifact of examining only the high-performance (low BPC) regime of a fundamentally logarithmic dependency. Thus, small improvements in compression at low BPC yield diminishing returns on intelligence, while large BPC changes at high BPC have little effect.

Compression metrics cannot capture all critical aspects of intelligence such as syntactic and semantic code validity, especially in specialized domains. Additionally, good in-sample compression is insufficient to guarantee generalization under distribution shift (e.g., post-training cutoff) (Li et al., 1 Feb 2024).

4. Compression-Centric Approaches to Model and Data Optimization

Model Compression and Rate Distortion

Model compression—pruning, quantization, low-rank factorization—can be analyzed using rate-distortion theory. The rate-distortion function for compressing model parameters $W$ under performance loss $D$ is:

$$R(D) = \min_{P_{\hat{W} | W} : \mathbb{E}[d(W, \hat{W})] \leq D} I(W; \hat{W})$$

where $d$ measures output deviation, and $I$ is mutual information (Gao et al., 2018). For linear models, this is tight and achievable via probabilistic Gaussian noise addition. For deep networks, quadratic approximations (weighted parameter error) inform practical compression objectives, but model compression is not equivalent to re-training: the optimal compressed representation depends on network function geometry, not just loss minimization.

Compression-Aware Training

Training objectives can be explicitly designed to yield compressible models. Regularizing neural network weights (e.g., penalizing nuclear norm for low-rank, or group sparsity for neuron removal) during training aligns model weights with the requirements of downstream compression, resulting in compact models with no accuracy loss compared to post hoc compression (Alvarez et al., 2017).

Data Selection via Compression

Data curation for training LLMs also leverages compression metrics. The Zip algorithm selects maximally informative samples by constructing datasets that minimize compression ratio, thereby maximizing entropy and diversity (Yin et al., 9 Jul 2024). This results in higher downstream model performance than classical sample-quality-based selection, affirming that dataset information density (as measured by compression) is a key driver of learning efficacy. Performance risk can also be predicted via compression ratio and first-epoch training loss.

5. Compression and Resource Trade-offs in Model Training

Scaling laws for training with lossy compressed data formalize the substitutability of data quantity (sample size) and quality (bits-per-sample) under resource (e.g., storage) constraints. In vision tasks, the storage scaling law quantifies test error as a joint function of the number of training samples $n$ and number of bits per image $L$:

$$Err(n, L) \approx Err^* + A n^{-\alpha} + B L^{-\beta}$$

with exponents $\alpha$, $\beta$ fitted empirically (Mentzer et al., 25 Jul 2024). For fixed storage $s$, optimal allocation solves $n \cdot L = s$ for lowest error, often favoring more low-quality images over fewer high-quality ones, up to a point. Thus, compressed data can directly substitute for more data in training, with the error drop determined by the harmonic mean of scaling exponents.

6. Model Training as Structured Compression: Entropy and Randomness

Advanced frameworks interpret model training—especially iterative pruning and compress-train cycles—as a process of structured entropy reduction, relating the preservation of information content to guided randomness. The Gibbs randomness-compression proposition formalizes this: the success of compression (decreasing number of active parameters without loss of function) is mirrored by a directed decrease in Gibbs entropy over measurement vectors associated with weights (Süzen, 29 May 2025). This observation supports the hypothesis that deep learning training, particularly with compressive cycles, is equivalent to performing lossy compression that preserves functional entropy.

Table: Compression–Entropy Mapping in Neural Training

Cycle Stage	Measured Quantity	Interpretation
Post-pruning	Gibbs entropy of weight vectors	Degree of randomness
Post-retraining	Test accuracy	Preserved model function

The near-perfect correlation between entropy and performance demonstrates that compression and model utility retention are governed by the same underlying statistical physics principles.

7. Practical Applications and Implications

The compression–training equivalence informs both evaluation and optimization across multiple axes:

• Universal Benchmarks: Compression metrics provide task-agnostic, contamination-resistant benchmarks for models in language, time series, and vision (Wan et al., 25 Sep 2025, Li et al., 1 Feb 2024).
• Data Curation: Compression or entropy metrics guide principled data selection without labels or model-specific heuristics (Yin et al., 9 Jul 2024).
• Efficient Training: Training for compressibility yields models better aligned with deployment constraints (Alvarez et al., 2017).
• Optimal Resource Use: Storage scaling laws allow for principled choice of compression ratio and sample size given storage or compute constraints (Mentzer et al., 25 Jul 2024).
• Understanding Generalization: Out-of-distribution generalization is most faithfully assessed via post-cutoff compression rates (Li et al., 1 Feb 2024).

However, this equivalence is not absolute: metrics such as BPC fail to capture all aspects of code correctness, robustness to distribution shift, or task-specific requirements. Empirical relationships between compression and "intelligence" (e.g., code LLMs) may be logarithmic rather than linear, indicating diminishing returns for optimization via compression alone at frontier performance levels (Luo et al., 16 May 2025). Compression is thus a stringent, but not exclusive, criterion for model evaluation and selection.