--- title: Normalized Bi-Semantic Entropy url: https://www.emergentmind.com/topics/normalized-bi-semantic-entropy type: topic --- # Normalized Bi-Semantic Entropy Searching arXiv for the cited papers to ground the article and confirm metadata. [arXiv search] Query: 2506.00245 Normalized Bi-Semantic Entropy denotes a family of normalized entropy constructions in which uncertainty is measured after introducing an explicitly semantic structure into the sample space. In recent arXiv work, one formulation is a normalized pairwise uncertainty score for large language model outputs, obtained by normalizing Semantic Nearest Neighbor Entropy (SNNE/WSNNE) built from pairwise semantic similarities among sampled generations [2506.00245]. Distinct constructions also define a bi-semantic entropy through the joint use of statistical and logical probabilities in a P–T framework [2110.07769], through normalization of semantic entropy over semantic classes or over a binary semantic partition in natural language generation [2302.09664], through normalized spatial Rényi entropy equal to normalized fractal dimension [1608.02054], and through a Jensen–Shannon functional over identity and independence couplings that is bounded by one [2204.09723]. Taken together, these constructions suggest a shared concern with scale-free semantic uncertainty, but not a single universally fixed definition. ## 1. Pairwise semantic uncertainty in language generation For LLM uncertainty quantification, the construction in “Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity” begins from Semantic Entropy (SE). Given a prompt \(q\), one generates \(n\) responses \(\{a_1,\dots,a_n\}\). In the white-box setting, the sequence log-probability is \[ P(a\mid q)=\sum_j \log p(a_j \mid q \oplus a_{0\). The white-box version inserts length-normalized probability weights, \[ WSNNE(q)= -\sum_{i=1}^{n}\bar w_i \log \sum_{j=1}^{n}\exp\!\left(\frac{f(a_i,a_j\mid q)}{\tau}\right), \qquad \bar w_i=\frac{\widetilde{P}(a_i\mid q)}{\sum_j \widetilde{P}(a_j\mid q)}. \] As \(\tau\to 0\), the inner LogSumExp approaches a soft nearest-neighbor similarity via \(\max_j f(a_i,a_j\mid q)\); as \(\tau\) increases, it averages across neighbors. The construction is described as a nearest-neighbor-style entropy estimator in semantic space that is less sensitive to outliers than sum-of-similarity graph methods. Normalization is introduced because the magnitude of SNNE/WSNNE depends on \(n\), \(\tau\), and the range of \(f\). If \(f\in[f_{\min},f_{\max}]\), then \[ n\exp(f_{\min}/\tau)\le \sum_j \exp(f_{ij}/\tau)\le n\exp(f_{\max}/\tau), \] which yields \[ SNNE \in \left[-\log n-\frac{f_{\max}}{\tau},\; -\log n-\frac{f_{\min}}{\tau}\right]. \] The normalized bi-semantic entropy is then \[ NBSE(q)= \frac{SNNE(q)-\left(-\log n-\frac{f_{\max}}{\tau}\right)} {\left(-\log n-\frac{f_{\min}}{\tau}\right)-\left(-\log n-\frac{f_{\max}}{\tau}\right)} = \frac{SNNE(q)+\log n+\frac{f_{\max}}{\tau}} {\frac{f_{\max}-f_{\min}}{\tau}}. \] The white-box normalization is \[ NBSE_w(q)= \frac{WSNNE(q)+\log n+\frac{f_{\max}}{\tau}} {\frac{f_{\max}-f_{\min}}{\tau}}. \] For ROUGE-L and NLI probabilities, the prescribed choice is \(f_{\min}=0\), \(f_{\max}=1\); cosine similarity is first rescaled to \([0,1]\). Under this normalization, \(NBSE\approx 0\) when pairwise similarities are near \(f_{\max}\), and \(NBSE\approx 1\) when similarities are near \(f_{\min}\) [2506.00245]. ## 3. Generalization of semantic entropy and nearest-neighbor structure A central theoretical claim is that SNNE and WSNNE strictly generalize DSE and SE. The cited work proves a DSE recovery theorem: if \[ f(a_i,a_j\mid q)=\tau\log(1/n) \] whenever \(a_i,a_j\in C_k\) for some \(k\), and \(f=-\infty\) otherwise, then \[ SNNE(q)= -\sum_k \frac{|C_k|}{n}\log\frac{|C_k|}{n}=DSE(q). \] It also proves an SE recovery theorem: if \[ Q=\sum_i \widetilde{P}(a_i\mid q), \] and \[ f(a_i,a_j\mid q)=\tau\log\!\left(\frac{\widetilde{P}(a_j\mid q)}{Q}\right) \] whenever \(a_i,a_j\in C_k\), with \(f=-\infty\) otherwise, then \[ WSNNE(q)= -\sum_k \bar p_k \log \bar p_k = SE(q). \] The black-box and white-box pairwise estimators therefore reduce exactly to cluster-level semantic entropy under special similarity functions. Although SNNE does not require explicit clustering, the pairwise form admits an implicit decomposition. Given a clustering \(\{C_k\}\), for \(i\in C_k\), \[ S_i=\log\!\left[ \sum_{j\in C_k}\exp(f_{ij}/\tau) + \sum_{\ell\ne k}\sum_{j\in C_\ell}\exp(f_{ij}/\tau) \right]. \] The first term represents intra-cluster cohesion; the second represents inter-cluster overlap or separation. SNNE averages \(-S_i\) over \(i\). The relation to classical \(k\)-NN entropy estimation is also explicit. The Kozachenko–Leonenko estimator in \(\mathbb{R}^d\) uses nearest-neighbor radii \(\epsilon_i\), \[ H(X)\approx \frac{d}{n}\sum_i \log \epsilon_i + \log V_d + \psi(n). \] In the semantic construction, one may define a distance \(d(a_i,a_j)=1-f(a_i,a_j)\) for \(f\in[0,1]\), so that \(\epsilon_i=\min_{j\ne i} d(a_i,a_j)\). SNNE replaces the hard minimum with LogSumExp on similarities, thereby inheriting nearest-neighbor behavior without Euclidean geometric constants or embedding-dimension calibration [2506.00245]. ## 4. Estimation workflow, calibration, and empirical behavior The black-box workflow is specified as follows. One samples \(n\) outputs at a chosen generation temperature, computes all pairwise similarities \(f_{ij}\), evaluates \[ SNNE(q)= -\frac{1}{n}\sum_i \log \sum_j \exp(f_{ij}/\tau), \] and normalizes it to \(NBSE(q)\). The white-box variant first computes \(\widetilde{P}(a_i\mid q)\), forms \(\bar w_i\), evaluates \(WSNNE(q)\), and then normalizes to \(NBSE_w(q)\). The default hyperparameters reported in the paper are \(n=10\) and \(\tau=1\). For long single-sentence summarization and translation, ROUGE-L is recommended; for QA, NLI entailment or embedding cosine is recommended. Full pairwise evaluation costs \(O(n^2)\) similarity computations and \(O(n^2)\) memory if the similarity matrix is stored; \(k\)-NN truncation reduces aggregation to \(O(nk)\) after neighbor selection [2506.00245]. Practical guidance is also explicit. The work states that AUROC improves with \(n\) but saturates around \(n\approx 10\). Extremely low \(T\) yields overly conservative samples, and extremely high \(T\) yields excessively diverse outputs; degraded performance is reported near \(T\approx 0.5\) and \(T\approx 2.0\), while the primary setting uses \(n=10\) with \(T\approx 1.0\). Thresholds for hallucination detection are to be set on a development set, with \(NBSE\in[0,1]\) used as a threshold-friendly score. Across two LLMs and three tasks, the paper reports consistent gains over SE/DSE and other baselines. The reported examples are summarized below. | Setting | Metric | Reported values | |---|---|---| | QA, Llama-3.1-8B | AUROC | SNNE \(\approx 0.83\), SE \(\approx 0.79\), WSNNE \(\approx 0.83\), KLE \(\approx 0.80\) | | QA, Phi-3-mini | AUROC | SNNE \(\approx 0.84\), SE \(\approx 0.80\), WSNNE \(\approx 0.84\), KLE \(\approx 0.81\) | | Summarization, Phi-3-mini, ROUGE-L correctness | PRR | SNNE \(\approx 0.26\), DSE \(\approx 0.14\), WSNNE \(\approx 0.27\), LexSim \(\approx 0.23\) | | Summarization, Phi-3-mini, BERTScore correctness | PRR | SNNE \(\approx 0.64\), SE \(\approx 0.36\), WSNNE \(\approx 0.65\), LexSim \(\approx 0.58\) | | Translation, Phi-3-mini, ROUGE-L correctness | PRR | SNNE \(\approx 0.63\), SE \(\approx 0.57\), WSNNE \(\approx 0.63\) | | Translation, Phi-3-mini, BERTScore correctness | PRR | SNNE \(\approx 0.75\), SE \(\approx 0.70\), WSNNE \(\approx 0.75\) | These results are paired with ablations showing that ROUGE-L is often best for long generations, embedding or NLI similarities are competitive for translation under BERTScore, and \(\tau=1\) is a robust default. The implementation described in the paper uses ROUGE-L via Google’s implementation, models and datasets from Hugging Face, NVIDIA RTX A6000 hardware, and three runs per experiment; code is provided at `https://github.com/BigML-CS-UCLA/SNNE` [2506.00245]. ## 5. Alternative formalizations of bi-semantic normalization The phrase “bi-semantic” is also used in a distinct P–T framework in which two semantic systems coexist: statistical semantics, represented by \(P(y)\), \(P(y\mid x)\), \(H(Y)\), \(H(Y\mid X)\), and \(I(X;Y)\); and logical or truth-function semantics, represented by \(T(y)\), \(T(y\mid x)\), semantic cross-entropy \(H(Y_\theta)\), fuzzy entropy \(H(Y_\theta\mid X)\), and semantic mutual information \(I(X;Y_\theta)\). In that setting, the conditional bi-semantic entropy is \[ HBS(Y\mid X)=H(Y\mid X)+H(Y_\theta\mid X), \] the marginal bi-semantic entropy is \[ HBS(Y)=H(Y)+H(Y_\theta), \] and the associated information gain is \[ IBS(X;Y)=I(X;Y)+I(X;Y_\theta)=HBS(Y)-HBS(Y\mid X). \] Two unit-range normalizations are defined: \[ NBS1=\frac{H(Y\mid X)+H(Y_\theta\mid X)}{H(Y)+H(Y_\theta)}, \] and \[ NBS2= 1-\left[\alpha \frac{I(X;Y)}{H(Y)}+(1-\alpha)\frac{I(X;Y_\theta)}{H(Y_\theta)}\right], \qquad \alpha\in[0,1]. \] This construction is tied to semantic Bayes’ formulae, truth functions, NEFs, logical probabilities, and rate–distortion reinterpretations [2110.07769]. Within semantic uncertainty for natural language generation, a simpler normalization operates directly on semantic classes. If \(K_x\) is the number of semantic classes with non-zero mass, then \[ SE_{\mathrm{norm}}(x)=\frac{SE(x)}{\log K_x}. \] With sampled clusters \(C\), the practical estimator becomes \[ \widehat{SE}_{\mathrm{norm}}(x)= \frac{-\sum_{c\in C}\hat p(c\mid x)\log \hat p(c\mid x)}{\log |C|}. \] In the binary case \(K=2\), with \(p=p(c_1\mid x)\) and \(1-p=p(c_2\mid x)\), \[ H_{bi}(x)= -p\log p -(1-p)\log(1-p), \qquad H_{bi,\mathrm{norm}}(x)=\frac{H_{bi}(x)}{\log 2}. \] Operationally, this binary case may arise either from naturally binary tasks or from collapsing discovered clusters into a dominant meaning versus all alternatives [2302.09664]. A third formalization appears in spatial multifractal analysis. There the two “semantics” are macrostate Boltzmann entropy, \(M_0(\epsilon)=\ln N(\epsilon)\), and information/Shannon/Rényi entropy, \(M_q(\epsilon)\). The normalized quantity is \[ E_q^{(\mathrm{norm})}(\epsilon)\equiv \frac{M_q(\epsilon)}{M_{\max}(\epsilon)}=\frac{D_q}{D_{\max}}\in[0,1], \] with \(M_{\max}(\epsilon)=\ln N_T(\epsilon)\) and \(D_{\max}=d\). For regular monofractals the equality is exact; for empirical prefractals, the paper states that it becomes “infinitely approximate” as \(\epsilon\to 0\) within the scaling range [1608.02054]. A fourth construction uses Jensen–Shannon divergence. Given a distribution \(p\) on \(\mathcal X\), define two joint distributions on \(\mathcal X\times\mathcal X\): \[ \delta(x,x')=p(x)\ \text{if }x=x',\ \text{and }0\ \text{otherwise}, \qquad \pi(x,x')=p(x)p(x'). \] The entropy functional is \[ H^*(p):=\operatorname{JSD}(\delta\Vert \pi), \] with base-2 logarithms, so \(H^*(p)\le 1\). The paper also gives the closed form \[ H^*(p)=\sum_x p(x) I^*[p(x)], \qquad I^*(\alpha)=\frac{1}{2}\log_2\!\left(\frac{4\,\alpha^\alpha}{(\alpha+1)^{\alpha+1}}\right). \] This functional is strictly concave, equals \(0\) iff \(p\) is degenerate, is maximized on a fixed finite alphabet by the uniform distribution, and is strictly increasing in alphabet size under uniformity [2204.09723]. ## 6. Interpretation, scope, and recurrent misconceptions A recurrent misconception is to treat all normalized semantic entropies as normalized cluster counts. The pairwise LLM construction is explicitly different: it does not require clustering, and its uncertainty signal depends on pairwise semantic similarities rather than only on the masses or frequencies of semantic bins. This is why it is introduced as a remedy for regimes in which cluster proliferation causes DSE saturation and SE loses discriminative power [2506.00245]. A second misconception is that normalization alone determines comparability. In the pairwise LLM formulation, normalization removes the trivial dependence on \(n\) and on the scale induced by \(\tau\), but the score still depends on the chosen similarity function \(f\). The paper therefore prescribes the codomain-based bounds \([f_{\min},f_{\max}]\), recommends ROUGE-L for long single-sentence summarization and translation, and recommends NLI entailment or embedding cosine for QA-like answers. A plausible implication is that “normalized” here means scale-stable under fixed semantic geometry, not geometry-independent. A third misconception is that semantic entropy and pairwise semantic entropy quantify only one type of uncertainty. The LLM literature describes sampling-based semantic UQ as primarily capturing epistemic uncertainty by the variation of model outputs across samples, while also noting that aleatoric noise in the data can manifest as dispersion. In the earlier semantic entropy framework, paraphrase invariance is the central property: if many surface forms express the same meaning, semantic entropy decreases relative to token-level predictive entropy because probability mass is aggregated at the level of meanings rather than sequences [2302.09664]. Across the broader literature, the phrase “bi-semantic” is not tied to one invariant mathematical object. In one line of work it refers to pairwise semantic similarity among generated responses; in another it refers to the simultaneous use of Shannon probabilities and logical truth functions; in another it denotes the alignment of macrostate and information-theoretic spatial entropies through normalization; and in another it refers to the pair \((\delta,\pi)\) of identity and independence couplings. This suggests that the stable conceptual core lies less in a single formula than in a recurring program: encode two semantic structures, normalize the resulting entropy, and obtain a bounded or scale-free quantity suitable for comparison across prompts, datasets, labels, or scales. Source: https://www.emergentmind.com/topics/normalized-bi-semantic-entropy