Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

Abstract: Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.

• The paper presents requential coding, achieving orders-of-magnitude improvements by coding only the additional bits reflecting teacher-student KL divergence.
• It employs a teacher-student framework with relative entropy coding to compress neural networks independently of parameter count and dataset entropy.
• Empirical benchmarks on language and image datasets demonstrate significant compression gains and improved generalization bounds via PAC-Bayes principles.

Requential Coding: Efficient Model Compression via Self-Generated Training Data

Introduction and Motivation

The paper "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data" (2607.11883) introduces a fundamentally new approach to neural network compression. Traditional parameter-based compressors such as pruning and quantization produce model codes scaling linearly with parameter count. Prequential coding, a data-based alternative, compresses the training trajectory but still codes the entire data sequence, paying for both dataset entropy and parameter inefficiency. These approaches fail to capture the effective information learned by large models and overstate model complexity, especially as scale increases.

Requential coding circumvents these limitations by coding only the additional bits required when a teacher selects samples from the student’s own distribution. Using relative entropy coding (REC), each training data batch is transmitted as an index, incurring a bit cost proportional to the teacher-student KL divergence. The compressed model size is thus decoupled from both parameter count and dataset entropy, yielding orders-of-magnitude improvements in compression, with further advantage as model scale grows.

Figure 1: Requential coding achieves strong model compression; the student samples candidates from its distribution, with a teacher selecting one via REC, yielding a code length independent of parameter count and data entropy.

Method and Implementation

Requential coding operates as follows: At each training step, the student model $P_t$ generates candidate batches using a PRNG with a shared seed. The teacher $Q_t$ selects from these candidates according to REC so that the accepted sample $X_t$ is distributed as $Q_t$. The code records only the index of the accepted proposal, costing approximately $\mathrm{KL}(Q_t\|P_t)$ bits. Student and teacher trajectories are synchronized without explicit transmission of the teacher, and the code length concentrates tightly around its mean (see Appendix).

This coding scheme is parameter-agnostic and flexible in teacher selection. Optimizations such as teacher smoothing (EMA over teacher checkpoints) and iso-loss projection (resetting teacher to student and recovering pre-reset loss) further reduce KL and enhance compression. The runtime for evaluating compressed size scales moderately ($2.33\times$ normal training), but actual encoding is computationally intensive due to the exponential search cost in block size; amortizing REC costs over large batches is therefore crucial.

Figure 2: Code length amortization and encoding/decoding time tradeoff; code length drops as block overhead is amortized, and encoding grows exponentially with block size.

Compression and Benchmarking

Empirical results demonstrate requential coding’s superiority. On language (OpenWebText, FineWeb) and image (CIFAR-5M) datasets, student transformers compressed via requential coding achieve code lengths substantially below parameter-based quantization (even idealized 4-bit PTQ) and prequential coding; the gap increases with scale and dataset entropy.

Figure 3: Requential coding dominates the Pareto frontier of loss vs compressed model size, outperforming prequential coding and PTQ across datasets.

Notably, requential coding enables the accurate decomposition of learnable information (epiplexity) from dataset entropy, sharply distinguishing between model structure and random content. In practical scenarios, the requential code for even billion-parameter LLMs can fall below one bit per parameter, and the method is robust to dataset modality and entropy.

Figure 4: Larger models compress to smaller sizes despite higher parameter counts, holding loss fixed; compression improves orders-of-magnitude.

Theoretical Implications: Generalization and Scaling

Compression is closely tied to generalization via PAC-Bayes and minimum description length principles. The requential code feeds into generalization bounds as a valid prefix-free code, certifying improved bounds for large-scale LLMs in both fixed-data and compute-optimal scaling regimes ($D=20N$).

Figure 5: PAC-Bayes generalization bounds improve with requential coding, tightening with model scale and outperforming idealized PTQ.

The bound’s gap decays as a power law in model size; extrapolation indicates that, in compute-optimal scaling, the generalization gap vanishes with scale, consistent with empirical loss improvements. Requential coding reveals that compression and generalization coincide: larger models and ensembles are more compressible and generalize provably better, countering common heuristics based on parameter-based complexity.

Dynamic Effects and Data Structure

The technique further predicts subtle learning phenomena, e.g., gradual overfitting as training epochs increase: compressed model size grows, reflecting memorization, while training loss continues to decrease. Requential coding also quantifies the learnable contents of various datasets, ranking structured text far above random strings and images, in agreement with epiplexity theory.

Figure 6: Realized requential codelength tightly concentrates around its mean, establishing reliability of code length as a compression certificate.

Figure 7: Generalization bounds on OpenWebText follow trends observed for CIFAR-5M and FineWeb, confirming robustness across modalities.

Discussion and Future Directions

Requential coding is a theoretical and practical advancement in model compression, decoupling code length from model scale and dataset entropy, and operationalizing PAC-Bayes bounds for large neural networks. It is primarily an analysis tool rather than a practical transmission scheme owing to the computational cost of REC search, but its certification of compressibility and generalization has strong implications for understanding deep learning efficiency and overparameterization.

Open questions include compression of forgotten information during training, improved teacher trajectories, and application to more complex distillation and ensemble regimes. Requential coding enables principled evaluation of training methods and can inform future practices targeting improved generalization at scale.

Conclusion

Requential coding establishes a new standard for model compression, outperforming parameter-based and prequential approaches by leveraging self-generated data and efficient relative entropy coding. Its implications extend to generalization guarantees and dataset structure analysis, providing a rigorous framework for understanding the limits and phenomena of deep neural network learning and scale.