Large language models and the entropy of English

Abstract: We use LLMs to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to decrease with context length at least to $N\sim 10^4$ characters, implying that there are direct dependencies or interactions across these distances. A corollary is that there are small but significant correlations between characters at these separations, as we show from the data independent of models. The distribution of code lengths reveals an emergent certainty about an increasing fraction of characters at large $N$. Over the course of model training, we observe different dynamics at long and short context lengths, suggesting that long-ranged structure is learned only gradually. Our results constrain efforts to build statistical physics models of LLMs or language itself.

• The paper demonstrates that increasing context length leads to sustained decline in conditional entropy and code length for written English.
• It employs open-source transformer-based LLMs to measure long-range correlations and mutual information over diverse text corpora.
• Findings challenge classical models by showing persistent long-range dependencies, impacting future language modeling and compression.

LLMs and the Entropy of English

Overview

This work investigates long-range dependencies and information-theoretic properties of written English using current LLMs as analytical instruments. Evaluating conditional entropy and code length up to $N \sim 10^4$ characters on multiple corpora, the study demonstrates persistent decline in both metrics with increasing context length, indicating robust long-distance interactions not previously accessible to either human guessing games or earlier computational models. Distributional analyses reveal emergent certainty in prediction for large contexts, and direct measurements uncover low, but statistically significant, mutual information between widely separated characters.

Methodology

The analysis leverages four open-source transformer-based LLMs (OLMo 2 1B, Llama 3.2 1B, Qwen3 8B, and DCLM 1.7B) to compute per-token code lengths and estimate conditional entropy in next-token prediction conditioned on long contexts. Datasets include the C4 internet corpus, English Wikipedia, BBC News, BookSum narratives, and poetry (Gutenberg and Poetry Foundation), ensuring a diverse range of genres and styles. The models' output distributions are employed to calculate the code length $L(N)$ as an upper bound on the true conditional entropy.

Figure 1: Code length vs. context length for the C4 corpus, showing consistent decrease in $L(N)$ up to $N\sim 10^4$ characters across models.

Code Length and Entropy Scaling

Examining the behavior of $L(N)$ across corpora and models, the authors report strong agreement for $N > 10^3$, with no evidence of a plateau in code length even at the largest context lengths accessible. This extends prior conjectures (Hilberg, 1990) of sub-extensive entropy in natural language to substantially longer scales than previous studies. Notably, genre differences are pronounced; for example, poetry deviates by approaching a plateau far earlier than prose, suggesting different structural dependency regimes.

Figure 2: Code length scaling for C4, Wikipedia, and Gutenberg Poetry Corpus; poems exhibit longer code lengths and early plateauing compared to prose.

Conditional entropy distributions, as a function of context length, develop a pronounced power-law peak near zero for large $N$, indicating the emergence of almost perfect predictability for a subset of tokens. The overall decrease in mean entropy is evenly attributed to both bulk distribution shift and peak emergence.

Figure 3: Distributional evolution of conditional entropy and code length with context length, highlighting the rise of near-zero entropy predictions.

Information Structure and Learning Dynamics

Direct computation of mutual information between characters separated by distance $d$ (using sizable samples for statistical reliability) confirms the existence of long-range power-law correlations, with mutual information $I(d) \propto d^{-\alpha}$, where $\alpha\approx 0.12$ for C4. This observation supports the interpretation of written English as possessing interactions that persist up to thousands of characters, challenging classical finite-range models.

Figure 4: Mutual information between characters decays according to a power law for C4, confirming the presence of long-range correlations.

Model training dynamics reveal that short-range structure is assimilated rapidly, while the capacity to exploit long-range dependencies improves only gradually over extended training. This underscores the progressive nature of learning global, contextual phenomena compared to local rules in LLMs.

Figure 5: Code length improvement during model training, with delayed progress at large context lengths relative to rapid convergence at short ranges.

Theoretical Implications

The key result—a persistent, slow decline in conditional entropy and code length as context grows—implies that English (at least in written corpora) is best characterized by either direct long-range interactions or mean-field–like global variables mediating semantic and thematic coherence. Traditional bag-of-words models, finite-range Markov models, and even classical grammatical hierarchies do not account for the observed extended decay. These findings invite the development of new statistical physics–inspired models for natural language, incorporating long-range and hierarchical effects possibly analogous to latent field models.

Figure 6: Empirical comparison of conditional entropy and code length across models and corpora, illustrating persistent trends.

Practical Consequences and Future Directions

For AI research, this study constrains the theoretical space for accurate language modeling and compression: effective next-token distribution prediction demands architectures and training objectives that exploit and encode long-distance dependencies. The differences in behavior across genres further motivate bespoke approaches for narrative, expository, and poetic texts. Additionally, the findings suggest that improvements in LLMs' handling of extensive contexts may yield diminishing returns only at even larger, as-yet-unexplored scales.

The analysis also flags the need for new toy models capable of reflecting the continuous fall-off in entropy and long-distance correlation structure. Such models could provide insights not only for LLM design but for fundamental questions about information, meaning, and organization in natural language.

Conclusion

This paper presents a quantitatively rich study of English text predictability through LLMs, demonstrating sustained long-range interactions and a slow approach to certainty in character prediction. The results challenge standard finite-range models, motivate new theoretical frameworks, and have implications for both AI system design and our understanding of linguistic structure. Persistent decay of code length and mutual information at large context lengths reveal that natural language texts retain complex dependencies over thousands of characters, calling for reevaluation of underlying assumptions in statistical modeling of language.