TextEntropy: Entropy Metrics in Text

• TextEntropy is a family of entropy-based measures that quantify uncertainty and structural patterns in textual data using diverse representations.
• It employs methodologies like Shannon, block, conditional, and attention-derived entropy to assess language complexity, lexical diversity, and decoding performance.
• It enables practical applications in keyword extraction, model regulation, steganography, watermarking, and hallucination detection in speech and neural text generation.

TEXTENTROPY denotes a family of entropy-based constructs used to quantify uncertainty, diversity, structure, or information flow in textual systems. In the literature provided here, the term spans several distinct but related usages: Shannon entropy over word distributions in natural languages, block and source entropy for sequential text, entropy of word-length series, entropy-guided decoding and watermarking in autoregressive LLMs, entropy-controlled steganographic generation, and—most specifically in a SpeechLLM setting—“the entropy of re-normalised attention weights over text input tokens” (Waldendorf et al., 21 Apr 2026). This breadth suggests that TEXTENTROPY is best understood not as a single invariant formula, but as a technical label for entropy computed over a text-relevant representation, with the representation varying by task, level of organization, and model architecture.

1. Definitions and conceptual scope

In the supplied research, the most standard formulation is Shannon entropy over a vocabulary distribution,

$H(T) = - \sum_{i=1}^V p(w_i)\log_2 p(w_i),$

where $p(w_i)$ is the probability of word type $w_i$ in text $T$ (Bentz et al., 2016). In that setting, entropy is interpreted as average uncertainty, or average information content, associated with words. Closely related formulations appear in studies of lexical diversity in large corpora,

$H = -\sum_{i=1}^V p_i \log p_i,$

and in normalized word-frequency entropy,

$$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$

where $L$ is text length, $D$ is the number of distinct symbols, and $f_r$ is the frequency of the symbol at rank $r$ (Rosillo-Rodes et al., 2024, Febres et al., 2013, Febres et al., 2014).

Other papers redefine the underlying random variable. In word-length analysis, a text becomes a sequence

$p(w_i)$0

with $p(w_i)$1 the length of the $p(w_i)$2-th word, and entropy is computed over $p(w_i)$3-grams of these lengths (Kalimeri et al., 2014). In complexity–entropy analyses of written language, entropy rate is estimated at the character-sequence level, while excess entropy measures information shared between past and future portions of a text (Estevez-Rams et al., 2019). In hallucination detection for SpeechLLMs, TextEntropy is explicitly defined as the entropy of renormalized attention weights over text input tokens rather than over lexical frequencies (Waldendorf et al., 21 Apr 2026).

A concise comparison is useful.

Setting	Entropy object	Technical role
Natural-language word distributions	$p(w_i)$4 over vocabulary	Average uncertainty / information content
Word-length series	$p(w_i)$5-grams of word lengths	Language and genre discrimination
Autoregressive decoding	Conditional next-token distribution	Degeneration control and UID-style smoothing
SpeechLLM attention	Renormalized attention over text tokens	Hallucination detection feature

This range of definitions implies that TEXTENTROPY is representation-dependent. A plausible implication is that cross-paper comparisons are meaningful only after specifying the symbolization level—word, character, block, sentence pattern, token distribution, or attention slice.

2. Corpus-level entropy of words and symbols

A major line of work treats entropy as a corpus statistic of word or symbol distributions. Using parallel texts in 21 languages and then a massively parallel corpus spanning more than 1000 languages, one study distinguishes between block entropy, estimated from unigram frequencies, and source entropy, estimated with a Lempel–Ziv-style longest-match procedure (Bentz et al., 2016). The source-entropy estimator is written as

$p(w_i)$6

with $p(w_i)$7 the longest previous match length. The paper reports convergence below 100K tokens for all 21 languages, with average convergence points of about 35K tokens for source entropy and 38K tokens for block entropy, and uses 100K tokens as a cutoff in the Parallel Bible Corpus. It also reports that source entropies are systematically lower than block entropies, with average values roughly around 5.97 bits and 9.26 bits respectively, and that the two are strongly correlated, $p(w_i)$8, $p(w_i)$9, with fitted linear relation

$w_i$0

(Bentz et al., 2016).

Related work on gigaword corpora in English, Spanish, and Turkish studies entropy jointly with type-token ratio. There, entropy is again Shannon entropy over the empirical word distribution, and the central empirical result is a robust functional relation between entropy and TTR across books, news/web text, and tweets (Rosillo-Rodes et al., 2024). The paper derives, in the large-vocabulary limit,

$w_i$1

fits

$w_i$2

and then obtains an explicit entropy–TTR relation,

$w_i$3

This analysis is explicitly asymptotic and is stated to be most trustworthy for large texts and relatively small TTR (Rosillo-Rodes et al., 2024).

Bias in entropy estimation is itself a methodological issue. An efficient algorithm for Zhang’s estimator rewrites the estimate in terms of the frequency spectrum $w_i$4, the number of types occurring exactly $w_i$5 times, reducing computation from $w_i$6 to $w_i$7, where $w_i$8 is the number of distinct occupied frequencies (Lozano et al., 2017). Across multilingual corpora, the paper reports average $w_i$9 of 0.057 for UDHR and 0.037 for PBC, supporting the practical value of spectrum-based acceleration (Lozano et al., 2017).

Several studies apply entropy directly to literary or language-comparison problems. Written Spanish has been analyzed with direct counting for characters, digrams, trigrams, and $T$0-word symbols up to $T$1, yielding weighted averages of 4.40 bits/character, 7.96 bits/digram, and 11.05 bits/trigram for natural Spanish, and an approximate entropy rate of about 0.90 bits/character with redundancy near 86% (0901.4784). Another study compares English and Spanish texts and computer code through normalized entropy and specific diversity, reporting that artificial/code text generally has higher entropy than natural text of similar length, while Spanish shows higher diversity than English (Febres et al., 2013). A separate analysis of Nobel and non-Nobel English and Spanish texts reports that Nobel laureate texts tend to have higher specific diversity and lower entropy than comparison texts, and combines relative diversity, relative entropy, and Zipf deviation into a Writing Quality Scale (Febres et al., 2014).

3. Sequence structure, block entropy, and multiscale organization

TEXTENTROPY is not restricted to bag-of-words distributions. Several papers emphasize sequential dependence and multiscale organization. In word-length series, block entropies are defined over gliding $T$2-grams,

$T$3

with $T$4 estimated by relative frequency in a sliding window (Kalimeri et al., 2014). For $T$5-word segments, Greek texts are reported to have higher $T$6 than English texts, and the genre ordering is generally politics/economy $T$7 sports $T$8 literature. The paper attributes most of the effect to the unigram distribution of word lengths, especially the near-uniform probabilities of lengths five to ten, and shows via shuffled surrogates that real texts have slightly lower entropy than shuffled ones, indicating weak but detectable short-range word-length correlations (Kalimeri et al., 2014).

At the character-sequence level, complexity–entropy analysis distinguishes entropy rate $T$9 from excess entropy $H = -\sum_{i=1}^V p_i \log p_i,$0. For a stationary symbolic process,

$H = -\sum_{i=1}^V p_i \log p_i,$1

and $H = -\sum_{i=1}^V p_i \log p_i,$2 is also interpreted as mutual information between the infinite past and infinite future (Estevez-Rams et al., 2019). Using Lempel–Ziv estimators, one study maps works by Shakespeare, Conan Doyle, and Jacob Abbott into a complexity–entropy plane and reports distinct authorial regions: Abbott with lower entropy rate and higher excess entropy, Shakespeare with higher entropy rate and lower excess entropy, and Doyle intermediate (Estevez-Rams et al., 2019). Sentence-, word-, and character-randomization experiments further decompose contributions from different organizational levels.

A related theoretical framework studies excess entropy as block mutual information,

$H = -\sum_{i=1}^V p_i \log p_i,$3

and links power-law growth of excess entropy to Herdan’s law for vocabulary growth (Dębowski, 2011). The paper’s theorems state that, under stationary, finite-energy, strongly nonergodic processes, power-law growth in the number of inferable facts or in excess entropy implies lower bounds on vocabulary growth of admissibly minimal grammars, up to logarithmic factors. This suggests an information-theoretic route from long-range shared structure to lexical scaling (Dębowski, 2011).

Another correlation-centered paper models symbolic sequences as weakly correlated additive high-order stationary ergodic Markov chains and derives, for differential entropy,

$H = -\sum_{i=1}^V p_i \log p_i,$4

where $H = -\sum_{i=1}^V p_i \log p_i,$5 is the symbolic pair correlator (Melnik et al., 2014). The resulting interpretation is that entropy decomposes into an uncorrelated baseline plus a negative quadratic correction from correlations; both persistent and antipersistent correlations reduce entropy. Applied to written English, the paper reports one-letter entropy around $H = -\sum_{i=1}^V p_i \log p_i,$6 and an effective correlation scale roughly $H = -\sum_{i=1}^V p_i \log p_i,$7–$H = -\sum_{i=1}^V p_i \log p_i,$8 symbols (Melnik et al., 2014).

4. Entropy, lexical diversity, and maximum-entropy frequency models

A recurring theme is the relation between entropy and lexical diversity. The type-token ratio,

$H = -\sum_{i=1}^V p_i \log p_i,$9

is treated as a simpler diversity statistic, but one paper explicitly argues that TTR and entropy should not be treated as fully independent indicators in large natural-language corpora because both are constrained by Zipf and Heaps scaling (Rosillo-Rodes et al., 2024). This claim is empirical and analytic within the asymptotic regime developed there.

Another line of work uses maximum-entropy reasoning at the level of frequency distributions. The random group formation model predicts the frequency-of-frequency distribution $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$0 by maximum entropy, with generic form

$$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$1

determined by coarse state variables such as total token count $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$2, number of distinct types $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$3, and the maximum repetition count $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$4 (Yan et al., 2014). That paper applies the model to Chinese texts represented as words and as characters, arguing that both are well predicted by their respective coarse state variables, and treats systematic deviations from the neutral RGF prediction as evidence for additional language-specific structure, in particular multiple meanings of Chinese characters (Yan et al., 2014). A plausible implication is that entropy-maximizing baselines are valuable precisely because deviations from them can be interpreted linguistically.

Entropy has also been used for keyword detection in literary corpora. The abstract of one paper states that Shannon’s entropy of information is used as a tool for automatic keyword extraction, with special attention to the random shuffled text as a calibration device for ranking indices, and with Darwin’s The Origin of Species used as a representative sample [0701028]. Because only the abstract is available here, more specific methodological claims about that work cannot be made on the present evidence.

A different, more idiosyncratic formulation appears in a study of Chinese and English text complexity. There, entropy is defined over hypothetical and realized sentence configurations, with realized entropy for short sentence classes contributing most strongly, and lower entropy is interpreted as higher complexity (Xie et al., 2016). The paper distinguishes character-based and word-based Chinese analysis, uses the jiebaR package in R for Chinese segmentation, and normalizes specific entropy per 10,000 units. It also reports stylistic differences in debate transcripts, including an application contrasting Hillary Clinton and Donald Trump (Xie et al., 2016). This framework is conceptually distinct from Shannon entropy over empirical token distributions, even though it uses Shannon-like formulas.

5. Entropy in neural text generation, UID, steganography, and watermarking

In modern language generation, TEXTENTROPY often denotes entropy of the conditional next-token distribution. One paper defines

$$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$5

and proposes the Stable Entropy Hypothesis: human-like generations usually lie in a narrow and nearly flat entropy band (Arora et al., 2023). The corresponding stable entropy zone is taken as $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$6 standard deviations around a data-derived baseline and is reported to cover approximately 87% of smoothed conditional entropies under the target distribution. The paper further reports strong correlations between entropy-zone violations and degeneration metrics, including around $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$7 between Mauve and entropy violation ratio, around $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$8 between Repeat Score@5 and lower-bound violation ratio, and around $$h(F(f_r)) = -\sum_{r=1}^{D} \frac{f_r}{L}\,\log_D\!\left(\frac{f_r}{L}\right),$$9 between F1 and upper-bound violation ratio (Arora et al., 2023). Entropy-Aware Decoding then intervenes when entropy moves outside the calibrated band.

A 2025 decoding proposal combines entropy with Uniform Information Density. It defines next-token entropy,

$L$0

token surprisal,

$L$1

and a combined score,

$L$2

with selection rule

$L$3

(Shou, 20 Feb 2025). Using GPT-2 on WikiText-2, OpenWebText, and WMT, the paper reports that Entropy-UID attains the lowest entropy standard deviation, around 2.8, and relatively low average surprisal, around 5.7, with smoother information density than GPT-2, entropy-only, or UID-only baselines (Shou, 20 Feb 2025).

Entropy also functions as a control variable in text steganography. ADLM-stega formulates an entropy window

$L$4

and a confidence-like normalization

$L$5

using adaptive truncation of GPT-2 XL candidate pools (Qin et al., 2024). The paper reports substantially lower perplexity than RNN-stega and PPLM-stega, for example 36.73 at 1 BPW versus 1593.47 and 94.17 respectively, and much worse PPL after ablating the confidence module, which it presents as evidence that entropy/confidence control is essential (Qin et al., 2024).

RTMStega uses normalized entropy over the top-$L$6 candidate tokens,

$L$7

with embedding permitted only when

$L$8

(Jiang et al., 27 Oct 2025). The paper positions entropy as a gate for payload capacity and imperceptibility, arguing that low-entropy token distributions are a central bottleneck in generative steganography.

Watermarking work uses yet another operational definition. Entropy-Guided Watermarking accumulates a watermark entropy budget and delays watermark insertion until a threshold $L$9 is exceeded (Cai et al., 16 Apr 2025). The paper instantiates token entropy contribution as $D$0, leaves low-entropy prefixes unwatermarked, and then applies a secret-key-guided sampling function compatible with inverse transform sampling or binary sampling. It reports average degradation of 13.34% versus 15.81% for ITS in open-ended and semantic-quality settings, and large improvements on long-answer QA, for example on Llama-3.2-1B: MATH 94.7 for ITS versus 8.7 for the proposed method, GSM8K 92.1 versus 4.3, and BFCL 58.7 versus 0.0 (Cai et al., 16 Apr 2025). Under paraphrase attack, the proposed method is reported to incur only about a 10% AUC drop, whereas a binary-sampling scheme falls from about 0.98 to 0.35 (Cai et al., 16 Apr 2025).

6. TEXTENTROPY as a named attention metric in SpeechLLMs and related model analyses

The most explicit named metric called TextEntropy in the provided literature appears in hallucination detection for SpeechLLMs. There, TextEntropy is defined verbatim as “the entropy of re-normalised attention weights over text input tokens” (Waldendorf et al., 21 Apr 2026). For a given layer $D$1, head $D$2, and decoding step $D$3, the procedure is to select the text-token slice of the attention vector, renormalize it, compute its entropy, average over decoding steps, and concatenate the resulting layer-head values into a feature vector of size $D$4. MinMax scaling is applied to AudioEntropy and TextEntropy so that all feature values lie in $D$5 (Waldendorf et al., 21 Apr 2026).

In the same work, TextEntropy is one of four attention-derived hallucination features, alongside AudioRatio, AudioConsistency, and AudioEntropy. Empirically, it is not the strongest signal. The stable-feature analysis retains 39 AudioRatio features, 36 AudioConsistency features, 20 AudioEntropy features, and only 4 TextEntropy features. Cross-task overlap of top-50 heads is reported as 14% for both Qwen-2-Audio and Voxtral-3B, and the paper states that entropy-based metrics exhibit substantially lower consistency across ASR and speech-to-text translation tasks (Waldendorf et al., 21 Apr 2026). The interpretation given there is that TextEntropy is more task-specific and less transferable than the dominant audio-focused attention features.

A broader LLM paper considers model output itself as a stationary information source and estimates entropy rates over generated text (Scharringhausen, 23 Feb 2026). It assumes a constant random distribution, estimates conditional entropy up to context length 6, and extrapolates with

$D$6

taking $D$7 as the asymptotic entropy rate (Scharringhausen, 23 Feb 2026). Using the Open American National Corpus as a baseline, the paper reports 0.716 bit/word for written OANC, 1.255 bit/word for spoken OANC, and lower accumulated LLM entropies of 0.574 bit/word at $D$8 and 0.618 bit/word at $D$9. It concludes that LLM text is statistically more predictable per word than both written and spoken natural language in that setup (Scharringhausen, 23 Feb 2026). This suggests a distinct sense in which TEXTENTROPY can serve as a comparative diagnostic of synthetic versus human-generated language.

Across these model-centric applications, the shared pattern is operational rather than purely descriptive: entropy is used to detect hallucination, regulate decoding, allocate steganographic payload, or schedule watermark insertion. That shift from descriptive corpus statistic to online control signal is one of the clearest developments in the supplied literature.

7. Applications, controversies, and methodological constraints

The applications of TEXTENTROPY in these papers are broad: keyword extraction in literary corpora [0701028], multilingual comparison and translation difficulty (Bentz et al., 2016), lexical-diversity analysis at gigaword scale (Rosillo-Rodes et al., 2024), language and genre discrimination from word-length series (Kalimeri et al., 2014), quantitative style and writing-quality assessment (Febres et al., 2014), degeneration control in open-ended generation (Arora et al., 2023), UID-informed decoding (Shou, 20 Feb 2025), steganography (Qin et al., 2024, Jiang et al., 27 Oct 2025), watermarking (Cai et al., 16 Apr 2025), hallucination detection in SpeechLLMs (Waldendorf et al., 21 Apr 2026), and comparison of LLM outputs with natural corpora (Scharringhausen, 23 Feb 2026).

At the same time, the literature makes clear that entropy is not a single universally comparable scalar. Several papers work with unigram distributions, whereas others use conditional entropy, entropy rate, excess entropy, sentence-configuration entropy, normalized entropy over top-$f_r$0 candidate sets, or attention-distribution entropy. Some studies normalize entropy to $f_r$1, others report bits/word or bits/character, and others emphasize only relative changes. This implies that “higher” or “lower” TEXTENTROPY cannot be interpreted without the underlying representation, estimator, and normalization scheme.

The interpretation of low entropy is also not uniform. In the Stable Entropy Hypothesis, entropy that is too low signals degeneration through overconfidence and repetition (Arora et al., 2023). In the Chinese–English complexity study, lower entropy is interpreted as higher complexity because only a more selective subset of hypothetical configurations is realized (Xie et al., 2016). In literary-quality work, Nobel texts are associated with lower entropy together with higher diversity (Febres et al., 2014). In SpeechLLM hallucination detection, TextEntropy is merely one feature among several and is not the dominant one (Waldendorf et al., 21 Apr 2026). These contrasts are not contradictions at the level of formalism; they arise because the random variable and the research objective differ.

A final methodological constraint is estimator dependence. The supplied papers mention plug-in, NSB, Zhang’s estimator, Lempel–Ziv-style source entropy, direct counting, block shuffling, longest-match methods, and fitted asymptotic extrapolation (Rosillo-Rodes et al., 2024, Lozano et al., 2017, Bentz et al., 2016, Estevez-Rams et al., 2019, Scharringhausen, 23 Feb 2026). This suggests that any encyclopedia-level account of TEXTENTROPY must treat estimation strategy as part of the definition rather than as an afterthought. In the literature summarized here, entropy is not merely a measure of disorder; it is a task-conditioned, representation-specific instrument for probing and controlling text.