Semantic–Structural Entropy (S²-Entropy)

• Semantic–Structural Entropy (S²-Entropy) is a composite metric that decomposes linguistic uncertainty into additive structural and semantic components.
• It is derived from information-theoretic principles using measures like KL divergence and mutual information to contrast true text order against randomized baselines.
• Its applications range from corpus-level statistical universality analysis to enhancing uncertainty estimates in generative language models.

Semantic–Structural Entropy (S²-Entropy) is a composite information-theoretic metric designed to quantify both the structural ordering and the semantic context-dependence within linguistic sequences. Developed through two complementary lines of research—statistical word order analysis (Montemurro et al., 2015) and fine-grained semantic similarity-based uncertainty quantification (Nguyen et al., 30 May 2025)—S²-Entropy generalizes traditional entropy metrics by decomposing the overall uncertainty into additive structural and semantic components. The metric has been formulated rigorously for applications ranging from corpus-level statistical universality to generative uncertainty quantification in LLMs.

1. Formal Definition and Mathematical Foundations

Let $N$ be the total number of word tokens and $K$ the vocabulary size. For a word-type $w$, let $n_w$ denote its frequency in the corpus. S²-Entropy at scale $s$ is defined as: $S^2(s)\;=\;D_s\;+\;\Delta I(s)$ where:

• $D_s$ is the structural entropy term (relative entropy of ordering):

$D_s = H_s - H$

$H$ is the empirical entropy rate of the text (estimated e.g. via Lempel–Ziv), and $H_s$ is the entropy rate under random shuffling of word tokens (Boltzmann entropy).

• $K$0 is the semantic specificity term (bias-corrected mutual information between blocks and word identities):

$K$1

$K$2 measures the mutual information between block index $K$3 (partitioning the text into $K$4 blocks) and word identity $K$5; the expectation $K$6 removes finite-size bias from random permutations.

For generative models, an alternative formulation replaces hard clustering of outputs with a continuous, pairwise similarity kernel: $K$7 where $K$8 are model-generated answers, $K$9 is a semantic similarity kernel, and $w$0 is a scale parameter.

2. Information-Theoretic Derivation

Both terms in S²-Entropy are derived from first principles in information theory:

• Structural Component ($w$1): This is a Kullback–Leibler divergence between the true sequential process $w$2 and the random bag-of-words model $w$3. The entropy rate $w$4 is estimated using nonparametric string-matching or compression measures, while $w$5 exploits the combinatorial entropy of constrained permutations.
• Semantic Component ($w$6): Partition the text into equal-length blocks and compute the empirical mutual information $w$7 between block indices and word types. Analytical correction $w$8 removes bias due to finite sample sizes.

In the case of model outputs, S²-Entropy applies a nearest-neighbor style entropy estimator, replacing discrete clusters with kernel-weighted affinity sums to account for both within-cluster spread and between-cluster distances.

3. Decomposition: Structural vs. Semantic Uncertainty

S²-Entropy quantifies two orthogonal channels of order:

• Structural Order ($w$9): Measures long-range syntactic, grammatical, and ordering constraints not captured by vocabulary statistics alone. Empirical studies show a universal mean $n_w$0 bits/word across diverse languages (Montemurro et al., 2015).
• Semantic Order ($n_w$1): Isolates topical, context-dependent variability. $n_w$2 is maximized at characteristic scales ($n_w$3 words) corresponding to lexical domains and topic spans.

A plausible implication is that the additive form $n_w$4 allows direct attribution of uncertainty in text or model outputs to either global syntactic structuring or local domain-specific semantic clustering.

4. Methodological Procedure and Estimation Algorithms

Corpus-Based Estimation (Montemurro et al., 2015)

1. Compute $n_w$5: Use a string-matching compressor to estimate empirical entropy rate.
2. Compute $n_w$6: Analytical calculation using word frequencies:

$n_w$7

1. Calculate $n_w$8:

$n_w$9

1. Partition text: Divide into $s$0 blocks of size $s$1.
2. Empirical Mutual Information: for each word $s$2, compute the block distribution $s$3 and entropy $s$4.
3. Finite-Size Correction: Analytical expectation $s$5 under random shuffle.
4. Aggregate: Compute

$s$6

Generative Model Estimation (Nguyen et al., 30 May 2025)

1. Sample Outputs: Generate answers $s$7 for a prompt $s$8.
2. Pairwise Similarity: Compute $s$9 for all $S^2(s)\;=\;D_s\;+\;\Delta I(s)$0 (cosine of embeddings, ROUGE-L, entailment scores).
3. Nearest-Neighbor Entropy: For black-box models, average LogSumExp over kernel similarities; for white-box, weight by normalized model probabilities.

Pseudocode

$D_s = H_s - H$3

This procedure generalizes hard semantic clustering to continuous affinity-based uncertainty estimates.

5. Key Properties and Assumptions

• $S^2(s)\;=\;D_s\;+\;\Delta I(s)$1 is remarkably invariant across typologically diverse corpora (mean $S^2(s)\;=\;D_s\;+\;\Delta I(s)$2 bits/word, SD $S^2(s)\;=\;D_s\;+\;\Delta I(s)$3 bits/word over 24 language families).
• $S^2(s)\;=\;D_s\;+\;\Delta I(s)$4 peaks at a finite scale corresponding to topical domains, generally $S^2(s)\;=\;D_s\;+\;\Delta I(s)$5 words.
• Both terms assume stationarity and ergodicity in word-type statistics and require no lexicon or grammar annotation.
• In generative settings, S²-Entropy strictly generalizes semantic entropy (SE): cluster-based SE is the limiting case when the pairwise similarity kernel is degenerate.
• Robustness to temperature parameter $S^2(s)\;=\;D_s\;+\;\Delta I(s)$6 and similarity metrics (ROUGE-L, embedding cosine) has been established empirically.

6. Empirical Performance and Illustrative Example

Corpus Example

Given the toy text "dog eats dog bone and dog eats bone" with $S^2(s)\;=\;D_s\;+\;\Delta I(s)$7 tokens and $S^2(s)\;=\;D_s\;+\;\Delta I(s)$8 word-types:

• $S^2(s)\;=\;D_s\;+\;\Delta I(s)$9 bits/word, $D_s$0 bits/word $D_s$1 bits/word.
• Partitioned into $D_s$2 blocks: semantic term $D_s$3 bits/word.
• $D_s$4 bits/word.

In typical real corpora, $D_s$5 bits/word, $D_s$6–$D_s$7 bits/word, yielding $D_s$8–$D_s$9 bits/word (Montemurro et al., 2015).

Generative Model Evaluation

S²-Entropy, both black-box and white-box, consistently outperforms semantic entropy and baseline uncertainty metrics by 2–5 AUROC points in question-answering and by 3–7 points on summarization/translation precision-recall rate (Nguyen et al., 30 May 2025). Its advantages are most pronounced for long, semantically diverse outputs.

7. Theoretical Generalization and Limiting Cases

Two formal results establish S²-Entropy as a strict generalization of cluster-based semantic entropy:

• Discrete SE as a Special Case: If $D_s = H_s - H$0 is constant within clusters and $D_s = H_s - H$1 between clusters, S²-Entropy reduces to semantic cluster entropy.
• Weighted SE Recovery: When $D_s = H_s - H$2 is proportional to the log of normalized model probabilities within clusters, white-box S²-Entropy matches probability-weighted semantic entropy.

A plausible implication is that S²-Entropy unifies information-theoretic order quantification across both static texts and dynamic model-generated outputs, allowing for expressive measurement of uncertainty that subsumes existing cluster-based approaches.