Semantic Reformulation Entropy (SRE)

• Semantic Reformulation Entropy (SRE) is an uncertainty metric computed by clustering semantically equivalent answers and applying Shannon entropy over the resulting distribution.
• It assesses model consistency by comparing multiple reformulated inputs and outputs to detect semantic dispersion and potential hallucinations.
• SRE has shown robust performance in QA benchmarks, outperforming traditional measures like perplexity with significant AUROC gains.

to=arxiv_search.search 天天中彩票可以 json_object
{"query":"all:\"Semantic Reformulation Entropy\" OR all:\"semantic entropy\" large language models hallucination detection", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search.search 天天中彩票在 ＿国产 code 񎔊ppjson_object
{"query":"all:\"Semantic Reformulation Entropy\" OR all:\"semantic entropy\" hallucination", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search 天天中彩票软件 天天爱彩票网站 天天中彩票怎么买json
{"query":"Semantic Reformulation Entropy semantic entropy hallucination", "max_results": 10}
Semantic Reformulation Entropy (SRE) is an uncertainty measure defined over meanings rather than token strings. In its contemporary large-language-model usage, SRE is computed by generating multiple answers—or, in a stronger formulation, answers to multiple semantically equivalent reformulations of the same question—grouping those outputs into semantic-equivalent clusters, and applying Shannon entropy to the resulting cluster distribution. High SRE indicates dispersion across incompatible meanings and is therefore interpreted as high epistemic uncertainty; low SRE indicates concentration on a single dominant semantic interpretation. The term is explicit in question answering work on hallucination detection [2509.17445], while closely related papers use the label semantic entropy for what they explicitly describe as the same construct: entropy over semantic reformulations [2302.09664, 2503.00269].

1. Conceptual foundations

In the semantic-uncertainty literature, the central move is to replace the standard event space of token sequences with an event space of semantic equivalence classes. For an input (x), a language model defines a distribution over sequences (p(s\mid x)), but SRE aggregates that mass over all sequences that “mean the same thing.” In the formalization introduced for semantic uncertainty, one defines an equivalence relation (E(s,s')) and corresponding semantic classes (c), then sets
[
p(c\mid x)=\sum_{s\in c} p(s\mid x),
\qquad
SE(x)=-\sum_c p(c\mid x)\log p(c\mid x).
]
This is Shannon entropy over meanings rather than over strings [2302.09664].

A closely aligned formulation appears in clinical LLM evaluation, where “semantic entropy” is described as the same object that people refer to as Semantic Reformulation Entropy: multiple sampled answers to one question are clustered by semantic equivalence, and entropy is computed over the cluster frequencies. In that interpretation, “semantic” identifies the level of abstraction, “reformulation” identifies the sampled paraphrastic answer set, and “entropy” identifies the Shannon functional [2503.00269].

This formulation differs sharply from perplexity and related token-level confidence measures. Perplexity is sensitive to the lexical probability of one sequence, whereas SRE is sensitive to meaning-level disagreement across samples. A model can therefore assign high token probability to a hallucinated answer while still exhibiting high SRE if repeated samples diverge semantically; conversely, a cluster of paraphrases expressing the same answer yields low SRE even when the surface forms differ [2302.09664].

2. Formal definition and computation

The simplest empirical definition starts from (M) sampled answers (y_1,\dots,y_M) to a question (q). After semantic clustering into (K) clusters (C_1,\dots,C_K), cluster probabilities are estimated as
[
p_i=\frac{|C_i|}{M},\qquad i=1,\dots,K,
]
and semantic entropy is
[
H_{\text{semantic}}(q)=-\sum_{i=1}^{K} p_i\log p_i.
]
This is the canonical SRE: entropy over semantic reformulation clusters [2503.00269].

A more general formulation, used in robust QA hallucination detection, augments output sampling with input-side semantic reformulations. For an original question (x), one first generates a set of reformulated questions
[
\mathcal{R}(x)={r_1,\dots,r_N},
]
filters them to a high-quality subset (\mathcal{R}^*(x)), samples (K) answers per reformulation, pools the answer set (\mathcal{Y}_x), clusters all answers jointly into (\mathcal{C}_x={C_1,\dots,C_m}), and defines
[

\mathcal{H}_{\mathrm{SRE}}(x,M)

-\sum_{C\in\mathcal{C}_x} p_f(C)\log p_f(C),
\qquad
p_f(C)=\frac{|C|}{|\mathcal{Y}_x|}.
]
In this formulation, SRE measures how semantically inconsistent the model is across both paraphrased inputs and sampled outputs [2509.17445].

The practical behavior of SRE therefore depends on two components: the quality of the semantic-equivalence relation and the stability of clustering. Early semantic-entropy work clusters sampled answers by bi-directional entailment, approximating “same meaning” by requiring (x+s\Rightarrow x+s') and (x+s'\Rightarrow x+s) [2302.09664]. Later work retains that logic but adds stronger pre-processing and clustering machinery: Sentence-BERT filtering of reformulated questions with cosine similarity (s(r_i,x)\in[\tau_{\min},\tau_{\max}]), exact-match grouping, embedding similarity, bidirectional NLI, and energy-based boundary refinement [2509.17445].

3. Generation, reformulation, and clustering protocols

In the original semantic-uncertainty pipeline, a single model samples multiple outputs for one input and clusters them by bi-directional entailment. The method is explicitly unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In the reported QA studies, multinomial sampling at intermediate temperature, typically (T=0.5), and sample counts below 20 were sufficient for useful uncertainty estimation [2302.09664].

Clinical applications adopt a similar structure but make the clustering protocol more explicit. In the women’s health study, GPT‑4o was sampled (M=10) times per question at (T=1.0), while semantic equivalence was judged deterministically at temperature (0.0) using bi-directional entailment. Clustering was iterative: each new response was compared with an existing cluster representative, added if both entailment directions held, and otherwise used to initialize a new cluster [2503.00269].

The QA-specific SRE formulation strengthens both sides of the pipeline. On the input side, semantically faithful paraphrases are generated by few-shot prompting and filtered for equivalence and diversity. On the output side, Hybrid Semantic Clustering (HSC) performs progressive grouping by exact match, embedding similarity, and DeBERTa-v2-xlarge-MNLI entailment/contradiction scores, then refines cluster boundaries using an energy function
[

E(i,j)

\mathbf{1}{\text{same}}(i,j)\Big[1-(\alpha\,\mathrm{sim}(y_i,y_j)+\beta\,\mathrm{entail}(y_i,y_j))\Big]
+
(1-\mathbf{1}{\text{same}}(i,j))\Big[1-\gamma\,\mathrm{contra}(y_i,y_j)\Big].
]
The reported operating point uses (N=3) reformulations, (K=8) samples per reformulation, (T=0.8), (\tau_{\mathrm{emb}}=0.92), and (\tau_{\mathrm{nli}}=0.8) [2509.17445].

These protocols instantiate two different views of SRE. The first treats SRE as entropy over sampled answer meanings for one prompt. The second treats SRE as entropy over meanings induced by equivalent questions plus sampled answers, thereby probing model consistency under input paraphrase invariance as well as output variability.

4. Empirical behavior in hallucination detection and clinical QA

Across free-form question answering, semantic entropy and SRE behave as uncertainty scores: incorrect answers exhibit more semantic dispersion than correct ones. In the foundational QA study, semantic entropy achieved an AUROC of about (0.83) on TriviaQA and about (0.77) on CoQA for OPT‑30B, outperforming predictive-entropy and lexical baselines; incorrect answers also produced more semantic clusters on average than correct answers [2302.09664].

In women’s health question answering, the same basic construct substantially outperformed perplexity. On 1,644 MRCOG questions, semantic entropy reached AUROC (0.76) versus (0.62) for perplexity, and in expert validation it reached AUROC (0.97) versus (0.57). Expert review also found that accuracy was highest when all sampled answers formed a single cluster, and dropped sharply as the number of clusters increased [2503.00269].

The explicit SRE formulation further improves robustness in QA benchmarks. On SQuAD-v2 and TriviaQA, the full method combining Semantic Reformulation and Hybrid Semantic Clustering, denoted SRE (SR+HSC), outperformed strong baselines including token-level negative log-likelihood, token entropy, embedding variance, (P(\mathrm{True})), and vanilla semantic entropy. On TriviaQA with Qwen3-14B, SRE (SR+HSC) reached AUROC (0.887), AURAC (0.685), and F1@Best (0.743); the paper attributes the strongest gains to HSC, with input-side reformulation providing additional improvement [2509.17445].

Paper	Setting	Representative finding
[2302.09664]	TriviaQA and CoQA free-form QA	Semantic entropy reached about (0.83) AUROC on TriviaQA and about (0.77) on CoQA
[2503.00269]	MRCOG women’s health QA	Semantic entropy reached (0.76) AUROC versus (0.62) for perplexity; expert validation reached (0.97)
[2509.17445]	SQuAD-v2 and TriviaQA	Full SRE (SR+HSC) reached AUROC (0.887) on TriviaQA with Qwen3-14B

Taken together, these results position SRE as a semantic-level uncertainty signal whose practical value derives from a simple empirical regularity: when a model repeatedly settles on incompatible meanings for equivalent tasks, hallucination risk increases.

5. Extensions beyond cluster-count entropy

Subsequent work argues that plain cluster-level entropy omits two important geometric factors: intra-cluster similarity and inter-cluster similarity. Semantic Nearest Neighbor Entropy (SNNE) replaces hard cluster counts by a pairwise-similarity functional
[

SNNE(q)

-\frac{1}{n}\sum_{i=1}^{n}\log\left(\sum_{j=1}^{n}\exp!\left(\frac{f(a^i,a^j\mid q)}{\tau}\right)\right),
]
with a white-box weighted variant
[

WSNNE(q)

-\sum_{i=1}^{n}\bar{P}(a^i\mid q)\log\left(\sum_{j=1}^{n}\exp!\left(\frac{f(a^i,a^j\mid q)}{\tau}\right)\right).
]
The paper proves that DSE is a special case of SNNE and SE a special case of WSNNE, thereby presenting SNNE and WSNNE as strict generalizations of cluster-based semantic entropy [2506.00245].

A different extension replaces entropy by energy while preserving semantic clustering. Semantic Energy uses logits from the penultimate layer, treating token energy as
[
E(x_t^{(i)},\theta)=-z_\theta(x_t^{(i)}),
]
and aggregates negative logits over tokens and semantic clusters. Its main criticism of semantic entropy is that when all sampled responses fall into a single semantic cluster, entropy collapses to zero even if the cluster is uniformly wrong. On single-cluster questions, the reported semantic-entropy AUROC is (50.0\%), whereas Semantic Energy remains discriminative [2508.14496].

In retrieval-augmented generation, SEReDeEP approximates semantic entropy without repeated sampling by training linear probes on hidden states. It defines External Context Entropy (ECE) as a Z-score of the current token’s predicted semantic entropy relative to attended context-token entropies, and Parametric Knowledge Entropy (PKE) as the absolute entropy difference across the FFN injection. The resulting context–parameter fusion improves hallucination detection over ReDeEP and SEP on RAGTruth and HalluRAG [2505.07528].

These extensions do not alter the central definition of SRE as entropy over semantic reformulations, but they show that the original cluster-count form is not the only way to operationalize semantic uncertainty. A plausible implication is that SRE is better understood as a family of semantic-level uncertainty functionals than as a single fixed estimator.

6. Broader entropy-theoretic context, limitations, and ambiguity of the acronym

SRE sits within a broader information-theoretic tradition in language. Word-entropy work defines entropy as average uncertainty per word token,
[
H(T)=-\sum_{i=1}^{V} p(w_i)\log_2 p(w_i),
]
distinguishes block entropy from source entropy, and shows that entropy estimates converge with relatively modest token counts in parallel corpora. That work does not define SRE, but it explicitly suggests an analogous entropy over semantic paraphrases or translations,
[
H_{\mathrm{SRE}}(x)=-\sum_j p(r_j\mid x)\log_2 p(r_j\mid x),
]
and proposes normalization by language-specific source entropy for cross-linguistic comparability [1606.06996]. A related semantic-tree model of language treats reformulation entropy as the residual or conditional entropy of token realizations given a semantic tree; this is presented as a conceptual extension rather than as an implemented SRE estimator [2602.13194].

Several limitations recur across the literature. The women’s health study reports that strict semantic clustering was successful in only about (30\%) of cases, largely because LLM-based entailment judgments are noisy and iterative clustering is order-dependent [2503.00269]. The explicit SRE method identifies sensitivity to paraphrase quality, clustering thresholds, and computational cost, since it requires multiple reformulated inputs, multiple sampled outputs, and pairwise NLI or embedding computations [2509.17445]. Semantic Energy, in turn, shows that semantic entropy can fail catastrophically when a model produces a single coherent but incorrect semantic cluster [2508.14496].

A further source of confusion is terminological. In radio astronomy, SRE denotes spectral relative entropy, the Kullback–Leibler divergence between an empirical voltage distribution and a fitted Gaussian reference distribution for RFI detection [2408.06488]. In quantum many-body physics, SRE denotes stabilizer Rényi entropy, a magic monotone defined from Pauli or Majorana expectation values [2509.17417, 2508.03534]. In wireless semantic communications, semantic entropy refers to the minimum expected number of semantic symbols needed to preserve task performance, not entropy over semantic reformulations [2402.02950]. Within LLM uncertainty estimation, by contrast, SRE refers specifically to Shannon entropy over semantic reformulation clusters.

Under that usage, SRE is best understood as a semantic-level uncertainty measure for generative models: sample meanings, collapse paraphrastic invariances, quantify dispersion, and interpret that dispersion as a proxy for hallucination risk. The literature now shows both the power of that idea and the technical fragility of its implementation.