Requential Coding: Sequential Model Compression

• Requential Coding is a sequential model compression technique that encodes the divergence between a teacher and a student using self-generated samples.
• It leverages relative entropy coding to control code lengths based solely on teacher–student disagreement rather than raw data entropy or parameter count.
• The method underpins tighter PAC-Bayes generalization bounds and aligns with progressive source-coding, noisy-feedback communication, and DNA reconstruction.

Requential coding denotes, in its most explicit current usage, a model-compression scheme in which a student generative model is transmitted by coding its training on self-generated samples selected by a stronger teacher, so that the expected code length is controlled by teacher–student disagreement rather than parameter count or raw data entropy (Qiu et al., 13 Jul 2026). Across the broader arXiv literature, the same label or close interpretive variants are attached to several distinct sequential coding constructions, including linear feedback communication, profile-based DNA reconstruction codes, temporal predictive coding, and progressive source-coding schemes. This suggests that the term is best understood as a family resemblance centered on sequential generation, refinement, or reconstruction, rather than as a single classical coding-theoretic object.

1. Core formulation in model compression

In the 2026 formulation, the student is a generative model $P_t$ trained iteratively. At each step $t$, the student proposes candidate samples $Y_t^{(0)}, Y_t^{(1)}, \dots$ i.i.d. $\sim P_t$ using shared randomness, and the teacher $Q_t$ selects an index $i_t^\star$ via a relative entropy coding (REC) acceptance rule so that the accepted sample $X_t := Y_t^{(i_t^\star)}$ is marginally distributed as $Q_t$. Both encoder and decoder apply the same update rule $G$ to obtain $P_{t+1} = G(P_t, X_t)$. The code records only the selected index at each step; the teacher sequence is used only on the encoder side and is never transmitted (Qiu et al., 13 Jul 2026).

This construction is explicitly contrasted with two older compression paradigms. Parameter-based compression, such as post-training quantization, produces code lengths that scale with parameter count. Prequential coding instead compresses the training trajectory, but it codes the exact data sequence and therefore pays for the entropy of the data even when the model has learned most of its regularities. Requential coding removes both dependencies: the leading term in its code length contains neither model size nor raw data entropy, only teacher–student divergence. In the paper’s framing, it therefore describes “how the student differs from the teacher,” not the final parameter vector and not the original dataset (Qiu et al., 13 Jul 2026).

A central operational premise is self-generated training data. The student samples its own candidates, and the teacher chooses among them. Because the decoder can regenerate the same candidate stream from the shared pseudorandom seed and the current student $t$0, transmitting the accepted index is sufficient to reconstruct the exact synthetic batch and hence the full training trajectory. The resulting compressed description is a code for the trained student, not for the teacher itself (Qiu et al., 13 Jul 2026).

2. Relative entropy coding mechanism and code-length bounds

The formal encoder–decoder pair is defined by a shared student initialization $t$1, a shared update rule $t$2, a shared PRNG seed $t$3, and a fixed number of steps $t$4. At step $t$5, the encoder computes $t$6, decodes $t$7, and updates $t$8; the decoder repeats the same decode-and-update procedure from the transmitted messages $t$9. In the rejection-sampling illustration, proposals $Y_t^{(0)}, Y_t^{(1)}, \dots$0 are accepted when

$Y_t^{(0)}, Y_t^{(1)}, \dots$1

with $Y_t^{(0)}, Y_t^{(1)}, \dots$2, and the transmitted message is a prefix-free code for the accepted index $Y_t^{(0)}, Y_t^{(1)}, \dots$3 (Qiu et al., 13 Jul 2026).

The cumulative conditional expected code length is bounded by

$Y_t^{(0)}, Y_t^{(1)}, \dots$4

with $Y_t^{(0)}, Y_t^{(1)}, \dots$5. For large batches, the lower-order logarithmic and constant terms are negligible, so $Y_t^{(0)}, Y_t^{(1)}, \dots$6 in practice (Qiu et al., 13 Jul 2026).

The bound is implemented with a universal integer code. Using Elias delta coding for the index,

$Y_t^{(0)}, Y_t^{(1)}, \dots$7

together with the PFR log-index bound

$Y_t^{(0)}, Y_t^{(1)}, \dots$8

yields the stated per-step complexity control (Qiu et al., 13 Jul 2026). More broadly, REC is the coding primitive that makes the scheme possible: REC algorithms encode a sample from a target distribution $Y_t^{(0)}, Y_t^{(1)}, \dots$9 using a proposal distribution $\sim P_t$0, with expected codelength on the order of $\sim P_t$1. A related line of work develops A*-based REC algorithms, including AS* and AD*, with expected codelength $\sim P_t$2 and favorable runtime guarantees under unimodality assumptions (Flamich et al., 2022).

The realized code length is controlled by a martingale fluctuation identity. If $\sim P_t$3 is the realized length and $\sim P_t$4 its cumulative conditional mean, then

$\sim P_t$5

With PFR and Elias delta encoding, the one-step conditional variances are bounded, and the paper gives a high-probability upper bound of the form

$\sim P_t$6

where $\sim P_t$7 is the summed KL term and $\sim P_t$8 the summed lower-order overhead (Qiu et al., 13 Jul 2026).

3. Relation to prequential coding and parameter-based compression

The main conceptual contrast is with prequential coding. For batches $\sim P_t$9, initialized model $Q_t$0, and update rule $Q_t$1, the prequential code length is

$Q_t$2

Its expectation over draws from a true distribution decomposes as

$Q_t$3

Prequential coding therefore scales linearly with dataset size and pays for irreducible entropy even when the predictor is already close to optimal (Qiu et al., 13 Jul 2026).

Requential coding avoids this because it does not transmit the exact real-data sequence. Instead, it transmits only the teacher’s selections among student-generated proposals. The expected cost is set by $Q_t$4, so bits are paid only where teacher and student disagree. When $Q_t$5, the encoder can transmit a constant message and the decoder simply uses the first proposal; no entropy cost is incurred. This is the sense in which the requential code length is independent of raw data entropy (Qiu et al., 13 Jul 2026).

The same section of the literature also contrasts requential coding with parameter-based methods such as post-training quantization. PTQ codes the final parameter array, with code length scaling with parameter count and remaining insensitive to how much information the model has actually extracted from its training experience. In the formulation of requential coding, this is precisely the undesirable dependence that the teacher–student protocol removes: no bits are paid for dormant parameter capacity, only for persistent predictive disagreement during training (Qiu et al., 13 Jul 2026).

A recurring misconception is therefore to treat requential coding as a synonym for prequential coding. The two procedures share a sequential training narrative, but their objects of compression differ fundamentally. Prequential coding compresses the observed data through the evolving model; requential coding compresses the evolving model through teacher-selected synthetic data (Qiu et al., 13 Jul 2026).

4. Generalization bounds, scaling laws, and empirical regime

A major application of the requential code is PAC-Bayes generalization. For bounded losses, the paper states a standard PAC-Bayes-style inequality

$Q_t$6

and for autoregressive LLMs with categorical NLL and prediction smoothing it uses the adapted bound

$Q_t$7

Because all complexity-controlled terms are monotone in $Q_t$8, a shorter code directly tightens the certified generalization bound (Qiu et al., 13 Jul 2026).

The empirical studies use OpenWebText, CIFAR-5M, and FineWeb, with GPT-2-style transformers over sequence length $Q_t$9. Scaling experiments range from approximately $i_t^\star$0M to $i_t^\star$1M parameters on OpenWebText and CIFAR-5M, and from $i_t^\star$2M to $i_t^\star$3B on FineWeb. In ensembles, $i_t^\star$4 models of approximately $i_t^\star$5M parameters each share a teacher, while the REC reference is the averaged student prediction (Qiu et al., 13 Jul 2026).

Several findings are emphasized. Requential per-token costs are one to two orders of magnitude below prequential per-token costs. Holding loss fixed, larger models and larger ensembles compress to substantially smaller sizes despite more parameters. Under the compute-optimal regime $i_t^\star$6, requential PAC-Bayes bounds beat idealized lossless 4-bit PTQ baselines and tighten with scale. On OpenWebText at fixed $i_t^\star$7B tokens, the bound drops from $i_t^\star$8 to $i_t^\star$9 nats as $X_t := Y_t^{(i_t^\star)}$0 grows from $X_t := Y_t^{(i_t^\star)}$1M to $X_t := Y_t^{(i_t^\star)}$2M, with a certified gap at the Chinchilla budget down to $X_t := Y_t^{(i_t^\star)}$3 nats by $X_t := Y_t^{(i_t^\star)}$4M parameters. The paper further reports that compressed size per parameter $X_t := Y_t^{(i_t^\star)}$5 decays as a power law for larger models, implying a shrinking certified generalization gap if the trend persists (Qiu et al., 13 Jul 2026).

The same code is used to probe multi-epoch training and dataset structure. As data are repeated across epochs, the bound predicts a gradual divergence between train and test losses, with the best bound appearing near one epoch. For matched budgets of $X_t := Y_t^{(i_t^\star)}$6B tokens and one epoch, the compressed model sizes rank uniform strings $X_t := Y_t^{(i_t^\star)}$7 trivial repeats $X_t := Y_t^{(i_t^\star)}$8 images $X_t := Y_t^{(i_t^\star)}$9 text, which the paper interprets as isolating learnable information from unpredictable content (Qiu et al., 13 Jul 2026).

5. Other technical uses of the term and closely related constructions

Outside model compression, “requential coding” and closely related sequential formulations appear in several technically distinct settings.

Domain	Construction	Technical core
Learned source coding	Successive refinement of the Wyner–Ziv problem (Joukovsky et al., 2023)	Layered RNN encoders/decoders, ideal Slepian–Wolf coding, progressive distortions $Q_t$0
Noisy-feedback communication	Sequential linear coding for AWGN with noisy output feedback (Mishra et al., 2021)	Scalar transmitter state, Kalman decoding, DP-optimal feedback gains
DNA storage	Codes for DNA sequence profiles (Kiah et al., 2015)	$Q_t$1-mer profile vectors, asymmetric distance, restricted de Bruijn graphs, Ehrhart counting
Weakly constrained coding	Error-resilient row-by-row coding (Mishra et al., 2024)	Sequential array construction, fixed-order stitching, transition rows, exact edge frequencies
Anytime communication	Sequential semi-orthogonal code on the infinite-bandwidth AWGN channel [0610151]	Streaming bits, delay-dependent reliability, semi-orthogonal signaling

In the learned Wyner–Ziv setting, the sequential aspect takes the form of successive refinement. The encoder produces a layered description in $Q_t$2 stages, and the decoder reconstructs with progressively improving distortions while using side information $Q_t$3. Under ideal Slepian–Wolf assumptions, the learned layered scheme recovers binning behavior akin to scalable nested quantization and achieves rate–distortion performance close to the Wyner–Ziv bound (Joukovsky et al., 2023).

In the AWGN noisy-feedback setting, the term refers to a causal linear strategy in which the transmitter maintains a scalar state

$Q_t$4

and the receiver applies Kalman filtering. The paper derives closed-form optimal coefficients within the proposed sequential class through dynamic programming and shows that the scheme coincides with Schalkwijk–Kailath in the noiseless-feedback limit (Mishra et al., 2021).

In DNA storage, requential coding is explicitly identified with $Q_t$5-gram or $Q_t$6-mer reconstruction coding. Codewords are distinguished by their profile vectors rather than by raw sequence identity, and decoding proceeds by reconstructing a representative sequence from a corrected profile via Eulerian traversal on a restricted de Bruijn graph. The relevant metric is an asymmetric distance on profile vectors, tailored to synthesis substitutions, sequencing substitutions, and undersampling losses (Kiah et al., 2015).

In weakly constrained coding, the sequential feature is row-by-row generation of a two-dimensional array whose columns are concatenated to form the transmitted word. The error-resilient variant replaces variable-order stitching with a constant number of deterministic transition rows, fixes the concatenation order, and preserves exact empirical edge frequencies while improving robustness to symbol errors in the structural rows (Mishra et al., 2024).

The infinite-bandwidth AWGN work presents an anytime code that is sequential in nature: bits are streamed without block buffering, and maximum-likelihood decoding yields an exponentially small probability of error as a function of tolerated receiver delay. Although the supplied material does not reproduce the paper’s detailed proofs, it situates the construction as a canonical example of sequential coding in communication over noisy channels [0610151].

6. Predictive-coding and representational interpretations, limitations, and terminology

Two additional arXiv usages are explicitly interpretive rather than terminologically canonical. In temporal predictive coding, “Requential Coding” is used as a label for sequential or temporal predictive coding. The single-layer model minimizes

$Q_t$7

updates weights by

$Q_t$8

and during recall evolves by

$Q_t$9

For identity nonlinearity, the paper proves that retrieval is equivalent to a whitened asymmetric Hopfield network,

$G$0

which ties the “sequential coding” interpretation to implicit statistical whitening and sequence memory (Tang et al., 2023).

In a separate Bayesian analysis of wide neural networks, the relevant phrase is “coding scheme,” interpreted as the structure by which neurons represent class information in the non-lazy regime. Linear networks exhibit analog coding, sigmoidal networks exhibit redundant coding through spontaneous symmetry breaking, and ReLU networks exhibit sparse coding via a small number of outlier neurons with $G$1 amplitudes. The query term is therefore interpreted there as representational coding rather than as a standalone named method (Meegen et al., 2024).

These heterogeneous usages delimit the term’s scope. In the strictest sense, “Requential Coding” is the 2026 teacher–student compression method. In a broader arXiv sense, it is an umbrella label occasionally applied to sequential reconstruction, progressive refinement, or emergent representational schemes. This broader reading is interpretive rather than standardized.

The 2026 scheme itself also has explicit limitations. It assumes discrete generative models, shared randomness, deterministic shared updates, and the ability to evaluate teacher and student likelihoods. Encoding time can be prohibitive because actual REC search may require exponentially many proposals in the KL divergence; the paper therefore treats requential coding primarily as a tool for evaluating compressed model size and obtaining PAC-Bayes certificates rather than as a practical deployment-time transmission protocol. The code is for the student, not the teacher, its length only grows across training, and compression weakens if the teacher is poorly calibrated or too far ahead of the student (Qiu et al., 13 Jul 2026).