Language Modeling is Monotone Compression

Abstract: A long-standing hypothesis in artificial intelligence and neuroscience posits that intelligence is closely related to compression: the ability to compress information efficiently intuitively reflects capacities associated with intelligence and learning. Indeed, recent experimental works verify this intuition by showing connections between the capabilities of LLMs and their ability as compressors: for instance, Deletang et al. (ICLR'24) demonstrate that LLMs can be used as powerful compressors, and Huang et al. (COLM'24) show that the compression ability of LLMs is highly correlated with their performance on benchmarks for knowledge and reasoning. In this work, we initiate a theoretical study of this connection. Our main result is that LLMs (formally modeled as next-token predictors) are equivalent to monotone (a.k.a. order-preserving) compression algorithms---namely, compression algorithms where the encoding process preserves the ordering of the inputs---in the sense that the one can be constructed from the other while preserving the same error up to an additive gap of 2. We next show that the monotonicity is required for this equivalence to hold if and only if cryptographic (infinitely-often) one-way functions exist. As a direct corollary, we get a cryptographic result of independent interest: the notion of next-bit pseudoentropy (a computational analogue of entropy) of a distribution is equivalent to monotone incompressibility of the distribution. (Previously, it was only known (Haitner et al., ITCS'23) that incompressibility implies next-bit pseudoentropy.)

• The paper proves an efficient equivalence between autoregressive models and monotone compression, enabling bidirectional transformations with a constant loss.
• The authors argue that monotonicity is both sufficient and often necessary for maintaining predictive accuracy when transitioning between compression and autoregressive modeling.
• The findings align compression with next-bit pseudoentropy, linking inorder preservation to both efficient autoregressive modeling and predictability of compressed distributions.

Problem formulation and conceptual contribution

“Language Modeling is Monotone Compression” (2610.11031) gives a complexity-theoretic characterization of the relationship between autoregressive language modeling and lossless data compression. The paper formalizes a LLM as an efficiently computable next-token predictor and measures its quality by expected negative log-likelihood, equivalently the cross-entropy between a source distribution $D$ and the distribution induced by autoregressive sampling from the model. Compression is formalized through efficient prefix-free encodings whose expected code length is compared with the Shannon entropy $H(D)$.

The central claim is not that arbitrary compression and language modeling are equivalent. Rather, the paper identifies monotone compression as the exact structural condition needed for an efficient equivalence. An encoding is monotone when it preserves a specified lexicographic ordering: if $x$ precedes $y$ in the source ordering, then the binary codeword for $x$ precedes the codeword for $y$. Under this condition, the paper proves an efficient two-way transformation:

• a $\delta$-optimal autoregressive model yields a $(\delta+2)$-optimal monotone compressor;
• a $\delta$-optimal monotone compressor yields a $\delta$-optimal autoregressive model.

Thus, the two representations have the same statistical efficiency up to a constant additive gap of two bits in one direction. This is a stronger statement than the informal observation that arithmetic coding can turn model probabilities into compressed representations. It establishes that the converse is also possible, but only when the encoding has sufficient order structure to support efficient recovery of prefix probabilities.

The result also explains an empirical asymmetry in prior work. Compression systems built from autoregressive predictors can be highly effective because arithmetic coding directly exploits the model’s conditional probabilities. Conversely, heuristic predictors constructed from general compressors can perform substantially worse. The paper attributes this disparity not merely to engineering differences, but to a formal computational obstruction: non-monotone compression can hide predictive structure in a way that efficient next-token prediction cannot recover.

Formal setup

Let $H(D)$0 be a distribution over token sequences, represented without loss of generality as fixed-length strings using an end-of-sequence token and padding. An autoregressive model $H(D)$1 assigns a conditional distribution $H(D)$2 to every next token. Its induced distribution over complete sequences is

$H(D)$3

The training objective is the expected negative log-likelihood,

$H(D)$4

which equals the cross-entropy $H(D)$5. Since

$H(D)$6

excess language-modeling loss is exactly a KL-divergence penalty, relative to the optimum $H(D)$7.

A compression scheme consists of an injective, prefix-free encoding $H(D)$8 and its decoder. Its objective is to minimize

$H(D)$9

Shannon’s source coding lower bound gives $x$0 for binary prefix-free codes. A compressor is $x$1-optimal if its expected code length is at most $x$2.

The monotonicity condition is imposed over the entire input domain, not merely over the support of $x$3. This is important for efficiency: monotonicity allows decoding and ranking by binary search, and it induces a tractable relationship between codeword intervals and source-prefix intervals. In particular, all strings sharing a fixed token prefix form a contiguous lexicographic interval, and monotonicity maps that source interval to a contiguous interval of binary random seeds or codewords.

The paper works in a non-uniform polynomial-time setting. Input lengths, alphabet sizes, and sequence lengths are polynomially bounded, and the relevant algorithms receive polynomial-size advice. These assumptions are essential to the efficient transformations and should not be conflated with claims about practical neural-network training.

From autoregressive models to monotone compression

The forward transformation uses arithmetic encoding with respect to the model-induced distribution $x$4. For each complete output $x$5, arithmetic coding assigns a codeword satisfying

$x$6

Averaging over samples from the true distribution $x$7 gives

$x$8

Since cross-entropy decomposes into entropy plus KL divergence,

$x$9

a model with excess loss at most $y$0 produces a compressor with expected length at most $y$1. The additive two-bit term is the arithmetic-coding overhead used by the construction. The paper notes that some constant overhead is unavoidable in general prefix-free coding; even non-monotone codes cannot uniformly achieve zero additive redundancy for arbitrary distributions.

The monotonicity of the resulting compressor is inherited from arithmetic coding. Arithmetic encoding orders the probability intervals associated with outputs according to the source ordering. The construction is therefore not simply a generic entropy coder: its interval structure preserves lexicographic order.

Efficiency requires more than efficient next-token evaluation. Arithmetic encoding also requires efficient computation of the cumulative distribution function

$y$2

The paper shows that this CDF can be computed recursively from the model’s next-token probabilities. For a candidate sequence $y$3, one sums the probabilities of all outputs that first diverge from $y$4 at each position by choosing a smaller token, then adds the probability of $y$5 itself. Because the alphabet and sequence length are polynomially bounded, only polynomially many conditional-probability evaluations are needed.

This result identifies a precise computational property underlying the common practice of compressing with LLMs: efficient autoregressive prediction implies efficient arithmetic encoding because the model’s local probabilities suffice to compute global lexicographic cumulative probabilities.

From monotone compression to autoregressive prediction

The reverse direction is technically more involved. Given a prefix-free monotone encoding $y$6, the authors construct a distribution $y$7 by decoding uniformly random binary strings. Informally, a random seed is examined from left to right; if one of its prefixes is a valid codeword, the corresponding source string is output. If no valid codeword is encountered, the procedure returns an appropriate source string determined by the order-preserving encoding.

For every source string $y$8,

$y$9

The reason is direct: every random seed beginning with $x$0 causes the sampler to output $x$1, and this event has probability $x$2. Consequently,

$x$3

Taking expectations under $x$4 yields

$x$5

Therefore, if $x$6 is $x$7-optimal, then

$x$8

so the induced distribution has $x$9. The corresponding autoregressive model is consequently $y$0-optimal.

The central algorithmic issue is computing the conditional next-token probabilities of $y$1. A naive approach would enumerate all random seeds, which is exponential in the code length. Monotonicity avoids this. The sampler is weakly monotone: larger binary seeds produce lexicographically no-smaller outputs. Therefore, the set of seeds producing outputs at most a given sequence $y$2 is an initial interval of the seed space. Its endpoint can be located by binary search using only polynomially many sampler evaluations.

Once the CDF of $y$3 is available, prefix probabilities follow by subtracting CDF values at the endpoints of lexicographic prefix intervals. Conditional next-token probabilities are then ratios of prefix probabilities. This supplies an efficient autoregressive representation of the distribution generated by the compressor.

The construction is significant because it preserves the compression loss globally. The model is not repeatedly approximating local token probabilities from the code lengths of prefixes. Instead, it defines one coherent distribution over complete outputs and derives all next-token probabilities from that distribution. The paper contrasts this with a prior heuristic that assigns a token weight based on the incremental change in compressed length. Such a method can incur a constant KL error at every generation step, producing total loss linear in sequence length. The present construction avoids that accumulation by performing the approximation once at the level of complete sequences.

Why monotonicity is necessary

The paper’s strongest structural claim is that monotonicity is not merely a convenient sufficient condition. Under standard cryptographic assumptions, it is necessary for any strong equivalence between efficient compression and efficient autoregressive prediction.

Assuming one-way functions, and hence pseudorandom generators, the authors construct an efficiently samplable distribution

$y$4

where $y$5 expands an $y$6-bit seed to a $y$7-bit pseudorandom string. A non-monotone compressor can encode such a sample using only the seed $y$8, achieving expected length exactly $y$9, which equals the entropy of the distribution. Encoding and decoding remain efficient: the compressor outputs the second component, and the decoder recomputes the generator output.

However, any efficient autoregressive model that predicts the distribution substantially better than chance would enable distinguishing pseudorandom generator outputs from uniformly random strings. Specifically, if an autoregressive model achieved KL divergence below $\delta$0, then its probability assignments to the first $\delta$1 bits could be thresholded to distinguish $\delta$2 from $\delta$3 with constant advantage. This contradicts pseudorandomness.

The resulting lower bound is quantitatively strong:

For every constant $\delta$4, every efficient autoregressive model has KL divergence at least $\delta$5 for all sufficiently large lengths, or infinitely many lengths in the infinitely-often formulation, while the non-monotone compressor achieves exactly $\delta$6.

Thus, under one-way functions, the gap between optimal non-monotone compression and efficient language modeling is linear in the sequence length. No transformation with constant or even sublinear additive loss can exist for arbitrary efficient compressors.

The construction exposes the role of order preservation. The compressor extracts the hidden seed from a pseudorandom-looking output, but the encoding destroys the lexicographic geometry needed to infer local conditional probabilities. Efficient decoding alone is insufficient: an autoregressive model requires a way to aggregate codeword mass over all outputs sharing a prefix, and a non-monotone encoding can make those sets computationally inaccessible.

The converse complexity-theoretic result is also established. If infinitely-often one-way functions do not exist, then the monotonicity condition can be dropped, although only approximately. The authors invoke the Impagliazzo–Luby next-bit prediction framework [(2610.11031) cites the 1989 result] to show that efficiently samplable distributions admit approximate next-bit predictors under the relevant nonexistence assumption. Given an arbitrary efficient $\delta$7-optimal compressor, uniform decoding of codewords defines a distribution within KL divergence $\delta$8 of the source. The next-bit prediction theorem then yields an autoregressive model statistically close to an optimal predictor for that auxiliary distribution.

This establishes a sharp dependence on cryptographic hardness:

• with infinitely-often one-way functions, monotone and non-monotone compression separate;
• without infinitely-often one-way functions, arbitrary compression admits an approximate prediction-to-compression equivalence.

The result is conditional and asymmetrical. The positive result without one-way functions incurs statistical error, unlike the exact monotone construction. It therefore does not establish exact equivalence between all compression and autoregressive modeling.

Connection to pseudoentropy

The paper applies the equivalence to next-bit pseudoentropy, a computational analogue of Shannon entropy. For a binary sequence, next-bit pseudoentropy measures whether each bit remains computationally unpredictable given its preceding prefix, aggregated across positions.

Prior work showed that incompressibility implies next-bit pseudoentropy, up to a constant, but the converse fails for unrestricted efficient compression when one-way functions exist. The present paper identifies the missing structural restriction: next-bit pseudoentropy is equivalent, up to an additive gap of two, to monotone incompressibility.

For a distribution family $\delta$9 and threshold $(\delta+2)$0, the paper establishes the equivalence between:

• the inability of every efficient monotone prefix-free encoding to achieve expected length below $(\delta+2)$1;
• next-bit pseudoentropy at least $(\delta+2)$2.

The forward implication follows by converting any efficient next-token predictor into a monotone arithmetic code. If the distribution had insufficient pseudoentropy, the corresponding predictor would achieve low KL divergence, and arithmetic encoding would yield a monotone compressor violating the incompressibility assumption. The reverse implication follows by converting a monotone compressor into an equally good autoregressive model and applying the characterization of pseudoentropy through KL-hardness.

This corollary gives a cryptographic interpretation of the main theorem. Monotone compression is not only an information-theoretic counterpart of autoregressive prediction; it is also the coding-theoretic form of next-bit unpredictability. The restriction to monotone codes is essential because unrestricted codes can exploit pseudorandom structure that is compressible but not efficiently predictable one bit at a time.

Limitations and open questions

The paper’s claims concern efficiently computable abstract distributions and encodings, not trained neural networks or practical compression ratios. The model class is a black-box family of next-token probability functions, and the results do not provide bounds relating transformer size, optimization dynamics, finite-sample estimation, or generalization to the existence of a $(\delta+2)$3-optimal model. Consequently, the equivalence should not be interpreted as an empirical claim that any high-performing compressor can be directly converted into a competitive LLM.

The constructions also rely on several technical assumptions: polynomially bounded alphabets and sequence lengths, prefix-free encodings, efficient exact or sufficiently precise probability computations, and non-uniform polynomial-time algorithms. The additive gap of two bits arises from arithmetic encoding and is constant rather than asymptotically significant, but the paper does not establish whether a fully general efficient monotone transformation can reduce this overhead to the optimal one-bit coding redundancy.

The negative result is conditional on the existence of infinitely-often one-way functions. Since that assumption is unresolved, the unconditional status of the separation between monotone and non-monotone compression remains open. Conversely, when infinitely-often one-way functions do not exist, the paper obtains approximate rather than exact prediction, with statistical error introduced by next-bit reconstruction. The precise tradeoff between this error, computational resources, and compression redundancy is not fully optimized.

Finally, the formal monotonicity requirement is defined relative to a fixed efficiently computable lexicographic ordering. It remains open how robust the equivalence is under alternative orderings, randomized or adaptive encodings, approximate monotonicity, and practical entropy coders whose arithmetic precision and probability representations are finite. These questions bear directly on whether the theoretical distinction between monotone and non-monotone compression corresponds to a broad practical principle or primarily to the exact computational model used in the paper.

Conclusion

The paper establishes that efficient autoregressive prediction and efficient monotone compression are quantitatively equivalent representations of the same distributional structure. Arithmetic coding gives the forward reduction with a two-bit overhead, while monotone decoding of uniformly sampled codewords gives the reverse reduction without additional cross-entropy loss. Monotonicity is also shown to be computationally substantive: assuming one-way functions, optimal non-monotone compression can coexist with linear next-token prediction error. The resulting equivalence between monotone incompressibility and next-bit pseudoentropy extends the modeling–compression connection into cryptographic complexity theory and identifies order preservation as the key condition governing efficient reversibility.