Discrete Semantic Entropy

• Discrete semantic entropy is a measure that abstracts raw symbols into discrete semantic units—such as feature maps, tokens, or geometric symbols—to capture task-specific meaning.
• It encompasses diverse formulations, including count-based minimal representations, Shannon entropy over semantic clusters, and predictive uncertainty in token distributions.
• This concept underpins adaptive communication strategies, enabling compression-control, physical layer security, and improved estimation through hybrid and coverage-adjusted approaches.

In the cited literature, discrete semantic entropy is not operationalized by a single universal formula. Instead, it is instantiated as an entropy-like quantity over discrete units that are intended to preserve task-relevant meaning: feature-map channels in wireless semantic communication, discrete speech tokens or token groups, discrete emergent-language messages, semantic equivalence classes of sampled model outputs, and geometric configuration symbols in time-series analysis (Rong et al., 2024, Zuo et al., 30 Aug 2025, Kharitonov et al., 2019, Wienholt et al., 10 Oct 2025, Majumdar et al., 2018). Across these formulations, the common theme is that entropy is shifted away from raw surface symbols toward discrete structures that are claimed to be semantically or task-relevantly sufficient.

1. Operational scope and principal formulations

The main formulations in the literature differ in what counts as a “semantic unit” and in whether entropy is interpreted as a Shannon entropy, a predictive uncertainty, or a minimum sufficient representation size.

Setting	Discrete semantic units	Quantity
Wireless semantic communications	Feature maps $F_i$	Minimum expected count of selected maps
Semantic speech representation	HuBERT token stream segments	Predictive entropy over next-token distributions
Emergent communication	Messages $M$	$H(M)=I(X_s;M)$
Black-box VLM uncertainty	Semantic answer clusters $C_i$	Shannon entropy over cluster frequencies
Time-series semantics	13 geometric configurations	Shannon entropy over configuration frequencies

One line of work defines semantic entropy as the minimum expected number of semantic symbols sufficient for a task, rather than as a Shannon entropy over probabilities. In that formulation,

$$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$

and the quantity is explicitly task-oriented and representation-based (Rong et al., 2024). Other work instead uses standard Shannon form over discrete semantic objects, such as speech-token predictions, semantic answer clusters, message symbols, or symbolic geometric configurations (Zuo et al., 30 Aug 2025, Wienholt et al., 10 Oct 2025, Kharitonov et al., 2019, Majumdar et al., 2018).

This divergence matters. It means that “discrete semantic entropy” can denote at least three distinct constructions: a dimensional bottleneck over discrete semantic units, a Shannon entropy over semantically clustered outcomes, or an information-theoretic complexity of a discrete semantic code. The literature therefore supports a family resemblance rather than a canonical definition.

2. Task-oriented minimal representations

In wireless semantic communication, the most explicit dimensional formulation begins with a general definition of semantic entropy as the minimum expected number of semantic symbols needed to preserve the task-conditional distribution. Because the optimal semantic encoder is intractable, the paper introduces a deep-learning approximation,

$$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$

and then specializes it to image semantic communication by treating encoder feature maps as discrete semantic units (Rong et al., 2024).

For an input image $\mathbf{p}\in\mathbb{R}^{H\times W\times 3}$, the encoder produces

$$\mathbf{Feat}=En(\mathbf{p},\zeta), \qquad \mathbf{Feat}\in\mathbb{R}^{H'\times W'\times N},$$

where each channel $F_i$ is treated as one semantic unit. Semantic importance is quantified through a gradient-based score

$$w_i^c = \sum_m \sum_n \alpha_{mn}^{ic}\,\mathrm{relu}\left(\frac{\partial z^c}{\partial F_{m,n}^i}\right),$$

followed by $M$0. The resulting task-specific approximation is

$M$1

with $M$2, the number of selected feature maps.

This formulation is discrete in a specific sense: the internal representation is continuous-valued, but semantic entropy is measured by how many feature-map slots are transmitted. The paper explicitly contrasts this with Shannon entropy and notes that there is no explicit $M$3 term in this construction (Rong et al., 2024). A common misconception is therefore that discrete semantic entropy must always be a Shannon entropy over semantic categories; this formulation shows that the literature also uses a count-based minimum-sufficiency notion.

3. Adaptive transmission and physical-layer security

The same wireless formulation is operationalized in SemEntropy for both adaptive transmission and channel security. After ranking feature maps by importance, the semantic selection module is written as

$M$4

with $M$5. The selected semantic units are then mapped to OFDM subcarriers according to channel gains

$M$6

through a joint assignment

$M$7

so that higher-importance semantic information is placed on better subchannels (Rong et al., 2024).

The paper’s Algorithm 1 computes feature maps under a semantic-entropy constraint, scores them, estimates CSI, channel-encodes each selected map, encrypts it, and allocates it to subcarriers. Lower semantic entropy corresponds to fewer selected feature maps, fewer carriers, and fewer transmitted bits; higher semantic entropy corresponds to the opposite. In experiments, the paper reports that “SemEntropy can keep the semantic accuracy remain 95% with 60% less transmission,” and that for classification at $M$8 transmission delay is reduced to about 25% of baselines with almost no accuracy loss across SNRs (Rong et al., 2024).

SemEntropy also uses semantic importance scores for key generation. The semantic key stream is defined as

$M$9

and is combined with a conventional Physical Layer Key. The paper emphasizes three randomness sources: semantic randomness from nonlinear task-dependent feature importance, channel randomness from reciprocal channel measurements, and subcarrier randomization from adaptive allocation. Under the reported evaluation, BER for legitimate users nearly matches the plaintext system, whereas BER for eavesdroppers is close to $H(M)=I(X_s;M)$0, and constellation plots show clean 16-QAM for legitimate users and scrambled constellations for eavesdroppers (Rong et al., 2024).

The significance of this line is not merely definitional. It demonstrates that a discrete semantic-entropy variable can act simultaneously as a compression-control variable and a security primitive. In that setting, “semantic entropy” is neither purely epistemic uncertainty nor purely Shannon source entropy; it is a control parameter over task-sufficient discrete semantic resources.

4. Predictive and output-space uncertainty formulations

A second family of formulations uses Shannon entropy over predictive distributions or semantic output clusters. In semantic speech representation learning, an autoregressive LLM over HuBERT token sequences defines token-wise predictive entropy

$H(M)=I(X_s;M)$1

which is then used to mark segment boundaries when $H(M)=I(X_s;M)$2 or when $H(M)=I(X_s;M)$3 (Zuo et al., 30 Aug 2025). Low-entropy spans are aggregated into longer segments, while high-entropy positions indicate likely phoneme or word boundaries. The paper compares M1, M2, and M3 thresholding strategies, adopts global-threshold M1, and reports that 15 Hz compressed representations often perform best on ASR and ST, while 7 Hz representations align best with word boundaries and 15 Hz representations align best with phoneme boundaries.

This is a Shannon-style discrete semantic entropy, but it is predictive rather than output-cluster based. The entropy is not over final task labels; it is over the next discrete token in a semantic speech stream. The operational role is segmentation and compression, not uncertainty filtering alone.

In black-box radiology VLMs, discrete semantic entropy is instead defined over semantic clusters of sampled answers. For a fixed image-question pair $H(M)=I(X_s;M)$4, the model is queried $H(M)=I(X_s;M)$5 times at temperature $H(M)=I(X_s;M)$6, answers are grouped into meaning-equivalent clusters $H(M)=I(X_s;M)$7 using bidirectional entailment, and

$H(M)=I(X_s;M)$8

is computed from the empirical cluster frequencies (Wienholt et al., 10 Oct 2025). With $H(M)=I(X_s;M)$9, the maximum possible value is $C_i$0, reached when each answer forms its own cluster. On 706 image-question pairs, baseline low-temperature accuracy was 51.7% for GPT-4o and 54.8% for GPT-4.1; after filtering out questions with DSE $C_i$1, retained-question accuracy rose to 76.3% for GPT-4o and 63.8% for GPT-4.1, both with $C_i$2 (Wienholt et al., 10 Oct 2025).

The same paper also states an important limitation: DSE measures semantic consistency, not correctness. It can fail on confident hallucinations, such as repeated wrong answers that all land in a single semantic cluster and therefore yield DSE near zero (Wienholt et al., 10 Oct 2025). This limitation is central to later work on improved estimators and calibration.

5. Emergent communication and minimal message complexity

In emergent-language signaling games, discrete semantic entropy appears as the entropy of the message distribution itself. With deterministic sender and receiver after training,

$C_i$3

and because $C_i$4 is a deterministic function of $C_i$5,

$C_i$6

The paper therefore identifies the information-theoretic complexity of the emergent language with message entropy $C_i$7 (Kharitonov et al., 2019).

The analysis establishes the bounds

$C_i$8

so the message must encode at least the task information unavailable to the receiver, but no more than the sender input. In the Guess Number task, the minimal required entropy is $C_i$9 bits when the receiver already knows $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$0 bits. In the two-digit MNIST classification task, the lower bound is $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$1. Empirically, the observed $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$2 tracks these task-imposed lower bounds closely: the emergent language is “as simple as the task allows” (Kharitonov et al., 2019).

The paper also reports that increased channel discreteness strengthens this entropy minimization pressure. Lower Gumbel–Softmax temperature drives $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$3 closer to the lower bound, reduces overfitting to noisy labels, and improves adversarial robustness. In the random-label experiments, GS with low temperature fails to memorize noisy labels that continuous or less discrete baselines can fit; under FGSM attacks, lower temperature yields higher robustness (Kharitonov et al., 2019).

This formulation is again distinct from the cluster-based and predictive variants. Here, discrete semantic entropy is message complexity under a communicative task constraint, and the central claim is that discrete channels induce an implicit information bottleneck.

6. Estimation, sample coverage, and alternative lineages

A major issue in black-box language-model uncertainty estimation is that canonical discrete semantic entropy is usually estimated from very few samples. In the standard black-box formulation, one samples $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$4 answers, clusters them into semantic classes $$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$5, and computes the plug-in estimator

$$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$6

Recent work shows that this underestimates the true semantic entropy because unseen semantic types receive zero probability mass (McCabe et al., 17 Sep 2025). To address this, the paper estimates semantic alphabet size using Good–Turing coverage,

$$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$7

combines it with the spectral estimator

$$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$8

and defines

$$H(X;Y) \triangleq \min_{E_S} \mathbb{E}\left(\dim(Code^{E_S}(X))\right) \quad \text{s.t.}\quad P(Y\mid Code^{E_S}(X)) = P(Y\mid X),$$9

A Chao–Shen-style coverage-adjusted hybrid entropy estimator then uses $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$0 as an estimated coverage factor. At $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$1, this hybrid estimator has the lowest MSE against the reference $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$2 across the reported model-dataset combinations, and the hybrid semantic alphabet size itself ranks jointly at the top with KLE in Bradley–Terry analysis of incorrectness detection (McCabe et al., 17 Sep 2025).

SENECA pushes the same issue into the general small-sample discrete entropy-estimation regime. It introduces a self-consistent missing-mass estimate $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$3 as a fixed point of a function $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$4, and then plugs this into a coverage-adjusted entropy estimator. In LLM uncertainty experiments, SENECA-M and SENECA obtain the second- and third-highest aggregated AUROC among the compared black- and gray-box uncertainty methods, and the paper explicitly presents SENECA as a drop-in replacement for small-sample discrete semantic entropy estimation (McCabe et al., 1 May 2026).

A separate lineage uses “semantic entropy” in a geometric-symbolic sense. For discrete time series, a 3-point neighborhood can realize precisely 13 geometric configurations, producing a 13-symbol alphabet; semantic entropy is then

$$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$5

where $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$6 is the frequency of configuration $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$7 (Majumdar et al., 2018). The same work defines information power from the discrete analog of $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$8 and uses the ratio $$\tilde{H}(X;Y) \triangleq \min \mathbb{E}\left(\dim(Code^{E_{\mathrm{DL}}}(X))\right) \quad \text{s.t.}\quad P(Y\mid X)-P(Y\mid Code^{E_{\mathrm{DL}}}(X))<\varepsilon,$$9 as an indicator of synchronous behavior in EEG, reporting that in 72 out of 87 seizures $\mathbf{p}\in\mathbb{R}^{H\times W\times 3}$0 reaches its lowest value during seizure (Majumdar et al., 2018). Earlier work in the same line had already introduced the 13-configuration alphabet, a DFA for all finite digital signals, and a WFST for tasks such as action-potential identification and speaker discrimination (Majumdar et al., 2016).

This older geometric usage broadens the term’s scope. In that line, “semantic” refers to local shape rather than task labels, equivalence classes, or communicative meanings. The literature on discrete semantic entropy therefore spans heterogeneous semantics: task sufficiency, predictive structure, answer meaning, emergent-message complexity, and local geometric form. That heterogeneity is a definitional fact of the field rather than an anomaly.