Normalized Bi-Semantic Entropy

• Normalized Bi-Semantic Entropy is a family of measures that quantify semantic uncertainty by embedding explicit semantic structures into the sample space.
• It generalizes traditional entropy estimators like SE and DSE through pairwise similarity metrics (e.g., SNNE and WSNNE) to address challenges in long, single-sentence outputs.
• Normalization in these constructions removes trivial dependencies (such as sample size and temperature), enabling robust comparisons across diverse LLM tasks and uncertainty metrics.

Searching arXiv for the cited papers to ground the article and confirm metadata. [arXiv search] Query: (Nguyen et al., 30 May 2025) Normalized Bi-Semantic Entropy denotes a family of normalized entropy constructions in which uncertainty is measured after introducing an explicitly semantic structure into the sample space. In recent arXiv work, one formulation is a normalized pairwise uncertainty score for LLM outputs, obtained by normalizing Semantic Nearest Neighbor Entropy (SNNE/WSNNE) built from pairwise semantic similarities among sampled generations (Nguyen et al., 30 May 2025). Distinct constructions also define a bi-semantic entropy through the joint use of statistical and logical probabilities in a P–T framework (Lu, 2021), through normalization of semantic entropy over semantic classes or over a binary semantic partition in natural language generation (Kuhn et al., 2023), through normalized spatial Rényi entropy equal to normalized fractal dimension (Chen, 2016), and through a Jensen–Shannon functional over identity and independence couplings that is bounded by one (Çamkıran, 2022). Taken together, these constructions suggest a shared concern with scale-free semantic uncertainty, but not a single universally fixed definition.

1. Pairwise semantic uncertainty in language generation

For LLM uncertainty quantification, the construction in “Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity” begins from Semantic Entropy (SE). Given a prompt $q$, one generates $n$ responses $\{a_1,\dots,a_n\}$. In the white-box setting, the sequence log-probability is

$$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$

and the length-normalized score is

$$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$

SE clusters the responses into $M$ semantic classes $\{C_k\}_{k=1}^M$ using a bidirectional entailment NLI model, forms cluster masses

$$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$

and computes

$SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$

In the black-box setting, Discrete SE (DSE) replaces probabilities with empirical fractions,

$DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$

The motivation for moving beyond SE is tied to long single-sentence generations. The cited work states that modern models such as Llama-3.1-8B and Phi-3-mini produce longer one-sentence outputs, and that the number of semantic clusters correlates strongly with response length, with Spearman $n$0 on SQuAD for Llama-3.1-8B. When $n$1, DSE saturates to a near-constant value because clusters become nearly singleton; even when $n$2 is small, SE ignores intra-cluster dispersion and inter-cluster separation because it only depends on cluster masses $n$3. This is the specific regime in which pairwise semantic methods are introduced (Nguyen et al., 30 May 2025).

2. SNNE, WSNNE, and the normalized bi-semantic score

The pairwise construction works directly in a semantic space. Let responses $n$4 have embeddings $n$5, or let similarity be defined at the string level. The similarity function $n$6 may be instantiated by ROUGE-L scores in $n$7, NLI entailment probabilities $n$8, or cosine similarity on sentence embeddings rescaled to $n$9 by $\{a_1,\dots,a_n\}$0. The full pairwise matrix $\{a_1,\dots,a_n\}$1 has entries

$\{a_1,\dots,a_n\}$2

Large $\{a_1,\dots,a_n\}$3 encode intra-cluster similarity, while small $\{a_1,\dots,a_n\}$4 encode inter-cluster separation.

The black-box Semantic Nearest Neighbor Entropy is

$\{a_1,\dots,a_n\}$5

with temperature $\{a_1,\dots,a_n\}$6. The white-box version inserts length-normalized probability weights,

$\{a_1,\dots,a_n\}$7

As $\{a_1,\dots,a_n\}$8, the inner LogSumExp approaches a soft nearest-neighbor similarity via $\{a_1,\dots,a_n\}$9; as $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$0 increases, it averages across neighbors. The construction is described as a nearest-neighbor-style entropy estimator in semantic space that is less sensitive to outliers than sum-of-similarity graph methods.

Normalization is introduced because the magnitude of SNNE/WSNNE depends on $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$1, $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$2, and the range of $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$3. If $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$4, then

$$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$5

which yields

$$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$6

The normalized bi-semantic entropy is then

$$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$7

The white-box normalization is

$$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$8

For ROUGE-L and NLI probabilities, the prescribed choice is $$P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{<j}),$$9, $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$0; cosine similarity is first rescaled to $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$1. Under this normalization, $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$2 when pairwise similarities are near $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$3, and $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$4 when similarities are near $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$5 (Nguyen et al., 30 May 2025).

3. Generalization of semantic entropy and nearest-neighbor structure

A central theoretical claim is that SNNE and WSNNE strictly generalize DSE and SE. The cited work proves a DSE recovery theorem: if

$$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$6

whenever $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$7 for some $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$8, and $$\widetilde{P}(a\mid q)=\frac{P(a\mid q)}{\operatorname{len}(a)}.$$9 otherwise, then

$M$0

It also proves an SE recovery theorem: if

$M$1

and

$M$2

whenever $M$3, with $M$4 otherwise, then

$M$5

The black-box and white-box pairwise estimators therefore reduce exactly to cluster-level semantic entropy under special similarity functions.

Although SNNE does not require explicit clustering, the pairwise form admits an implicit decomposition. Given a clustering $M$6, for $M$7,

$M$8

The first term represents intra-cluster cohesion; the second represents inter-cluster overlap or separation. SNNE averages $M$9 over $\{C_k\}_{k=1}^M$0.

The relation to classical $\{C_k\}_{k=1}^M$1-NN entropy estimation is also explicit. The Kozachenko–Leonenko estimator in $\{C_k\}_{k=1}^M$2 uses nearest-neighbor radii $\{C_k\}_{k=1}^M$3,

$\{C_k\}_{k=1}^M$4

In the semantic construction, one may define a distance $\{C_k\}_{k=1}^M$5 for $\{C_k\}_{k=1}^M$6, so that $\{C_k\}_{k=1}^M$7. SNNE replaces the hard minimum with LogSumExp on similarities, thereby inheriting nearest-neighbor behavior without Euclidean geometric constants or embedding-dimension calibration (Nguyen et al., 30 May 2025).

4. Estimation workflow, calibration, and empirical behavior

The black-box workflow is specified as follows. One samples $\{C_k\}_{k=1}^M$8 outputs at a chosen generation temperature, computes all pairwise similarities $\{C_k\}_{k=1}^M$9, evaluates

$$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$0

and normalizes it to $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$1. The white-box variant first computes $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$2, forms $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$3, evaluates $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$4, and then normalizes to $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$5. The default hyperparameters reported in the paper are $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$6 and $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$7. For long single-sentence summarization and translation, ROUGE-L is recommended; for QA, NLI entailment or embedding cosine is recommended. Full pairwise evaluation costs $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$8 similarity computations and $$P(C_k)=\sum_{i: a_i\in C_k}\widetilde{P}(a_i\mid q), \qquad \bar p_k=\frac{P(C_k)}{\sum_j P(C_j)},$$9 memory if the similarity matrix is stored; $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$0-NN truncation reduces aggregation to $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$1 after neighbor selection (Nguyen et al., 30 May 2025).

Practical guidance is also explicit. The work states that AUROC improves with $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$2 but saturates around $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$3. Extremely low $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$4 yields overly conservative samples, and extremely high $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$5 yields excessively diverse outputs; degraded performance is reported near $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$6 and $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$7, while the primary setting uses $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$8 with $SE(q)=-\sum_{k=1}^{M}\bar p_k \log \bar p_k.$9. Thresholds for hallucination detection are to be set on a development set, with $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$0 used as a threshold-friendly score.

Across two LLMs and three tasks, the paper reports consistent gains over SE/DSE and other baselines. The reported examples are summarized below.

Setting	Metric	Reported values
QA, Llama-3.1-8B	AUROC	SNNE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$1, SE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$2, WSNNE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$3, KLE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$4
QA, Phi-3-mini	AUROC	SNNE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$5, SE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$6, WSNNE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$7, KLE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$8
Summarization, Phi-3-mini, ROUGE-L correctness	PRR	SNNE $DSE(q)=-\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}.$9, DSE $n$00, WSNNE $n$01, LexSim $n$02
Summarization, Phi-3-mini, BERTScore correctness	PRR	SNNE $n$03, SE $n$04, WSNNE $n$05, LexSim $n$06
Translation, Phi-3-mini, ROUGE-L correctness	PRR	SNNE $n$07, SE $n$08, WSNNE $n$09
Translation, Phi-3-mini, BERTScore correctness	PRR	SNNE $n$10, SE $n$11, WSNNE $n$12

These results are paired with ablations showing that ROUGE-L is often best for long generations, embedding or NLI similarities are competitive for translation under BERTScore, and $n$13 is a robust default. The implementation described in the paper uses ROUGE-L via Google’s implementation, models and datasets from Hugging Face, NVIDIA RTX A6000 hardware, and three runs per experiment; code is provided at https://github.com/BigML-CS-UCLA/SNNE (Nguyen et al., 30 May 2025).

5. Alternative formalizations of bi-semantic normalization

The phrase “bi-semantic” is also used in a distinct P–T framework in which two semantic systems coexist: statistical semantics, represented by $n$14, $n$15, $n$16, $n$17, and $n$18; and logical or truth-function semantics, represented by $n$19, $n$20, semantic cross-entropy $n$21, fuzzy entropy $n$22, and semantic mutual information $n$23. In that setting, the conditional bi-semantic entropy is

$n$24

the marginal bi-semantic entropy is

$n$25

and the associated information gain is

$n$26

Two unit-range normalizations are defined: $n$27 and

$n$28

This construction is tied to semantic Bayes’ formulae, truth functions, NEFs, logical probabilities, and rate–distortion reinterpretations (Lu, 2021).

Within semantic uncertainty for natural language generation, a simpler normalization operates directly on semantic classes. If $n$29 is the number of semantic classes with non-zero mass, then

$n$30

With sampled clusters $n$31, the practical estimator becomes

$n$32

In the binary case $n$33, with $n$34 and $n$35,

$n$36

Operationally, this binary case may arise either from naturally binary tasks or from collapsing discovered clusters into a dominant meaning versus all alternatives (Kuhn et al., 2023).

A third formalization appears in spatial multifractal analysis. There the two “semantics” are macrostate Boltzmann entropy, $n$37, and information/Shannon/Rényi entropy, $n$38. The normalized quantity is

$n$39

with $n$40 and $n$41. For regular monofractals the equality is exact; for empirical prefractals, the paper states that it becomes “infinitely approximate” as $n$42 within the scaling range (Chen, 2016).

A fourth construction uses Jensen–Shannon divergence. Given a distribution $n$43 on $n$44, define two joint distributions on $n$45: $n$46 The entropy functional is

$n$47

with base-2 logarithms, so $n$48. The paper also gives the closed form

$n$49

This functional is strictly concave, equals $n$50 iff $n$51 is degenerate, is maximized on a fixed finite alphabet by the uniform distribution, and is strictly increasing in alphabet size under uniformity (Çamkıran, 2022).

6. Interpretation, scope, and recurrent misconceptions

A recurrent misconception is to treat all normalized semantic entropies as normalized cluster counts. The pairwise LLM construction is explicitly different: it does not require clustering, and its uncertainty signal depends on pairwise semantic similarities rather than only on the masses or frequencies of semantic bins. This is why it is introduced as a remedy for regimes in which cluster proliferation causes DSE saturation and SE loses discriminative power (Nguyen et al., 30 May 2025).

A second misconception is that normalization alone determines comparability. In the pairwise LLM formulation, normalization removes the trivial dependence on $n$52 and on the scale induced by $n$53, but the score still depends on the chosen similarity function $n$54. The paper therefore prescribes the codomain-based bounds $n$55, recommends ROUGE-L for long single-sentence summarization and translation, and recommends NLI entailment or embedding cosine for QA-like answers. A plausible implication is that “normalized” here means scale-stable under fixed semantic geometry, not geometry-independent.

A third misconception is that semantic entropy and pairwise semantic entropy quantify only one type of uncertainty. The LLM literature describes sampling-based semantic UQ as primarily capturing epistemic uncertainty by the variation of model outputs across samples, while also noting that aleatoric noise in the data can manifest as dispersion. In the earlier semantic entropy framework, paraphrase invariance is the central property: if many surface forms express the same meaning, semantic entropy decreases relative to token-level predictive entropy because probability mass is aggregated at the level of meanings rather than sequences (Kuhn et al., 2023).

Across the broader literature, the phrase “bi-semantic” is not tied to one invariant mathematical object. In one line of work it refers to pairwise semantic similarity among generated responses; in another it refers to the simultaneous use of Shannon probabilities and logical truth functions; in another it denotes the alignment of macrostate and information-theoretic spatial entropies through normalization; and in another it refers to the pair $n$56 of identity and independence couplings. This suggests that the stable conceptual core lies less in a single formula than in a recurring program: encode two semantic structures, normalize the resulting entropy, and obtain a bounded or scale-free quantity suitable for comparison across prompts, datasets, labels, or scales.