Per-Token Entropy: Concepts & Applications

• Per-token entropy is a measure of uncertainty computed for individual tokens, with applications ranging from autoregressive decoding to multimodal feature alignment.
• It serves as a key signal for calibration, reinforcement learning, and dynamic token selection, enabling adaptive inference and improved prediction sets.
• Recent methods leverage per-token entropy for efficient decoding, masking, and pruning by assessing local information density and activation sparsity.

Per-token entropy denotes an entropy quantity attached to an individual token or token position, but recent work uses the term for several non-equivalent objects. In autoregressive language modeling, it most commonly means the Shannon entropy of the next-token distribution at a decode step; in sequence-level uncertainty estimation it may be accumulated across positions; in multimodal and vision settings it can instead mean entropy over a token’s channel activations or over a token-specific covariance spectrum. Across these settings, per-token entropy functions as a local uncertainty, informativeness, sparsity, or compression signal, and is used for calibration, reinforcement learning, adaptive inference, multimodal alignment, masking, pruning, and safety analysis (Xu, 30 Aug 2025, Chen et al., 29 Jan 2026, Wang et al., 19 Feb 2026, Clark et al., 29 Jul 2025).

1. Formal meanings and mathematical forms

A central point in the literature is that “per-token entropy” is not a single invariant definition. The same phrase refers to at least five distinct constructions: entropy of the next-token distribution at a position; cumulative entropy along a generated sequence; entropy of a token’s feature vector across channels; matrix entropy of a visual token’s representation; and first-subword entropy used as an approximation to word entropy in psycholinguistics (Xu, 30 Aug 2025, Chen et al., 29 Jan 2026, Wang et al., 19 Feb 2026, Clark et al., 29 Jul 2025).

Setting	Token-level object	Representative papers
Autoregressive decoding	Shannon entropy of the next-token distribution at position $t$	(Chawla et al., 3 Jun 2026, Luo et al., 17 Jun 2026, Lv et al., 30 Jun 2026)
Candidate-level UQ	Sum of token entropies over a generated response	(Xu, 30 Aug 2025)
Multimodal feature learning	Shannon entropy of a token’s channel-wise SoftMax distribution	(Chen et al., 29 Jan 2026)
Visual token pruning	Matrix entropy / von Neumann entropy of token covariance spectrum	(Wang et al., 19 Feb 2026)
Psycholinguistic prediction	First-subword entropy used as a lower-bound approximation to word entropy	(Clark et al., 29 Jul 2025)

In autoregressive language modeling, the standard local form is the Shannon entropy of the predictive distribution. TECP writes the candidate-level score as

$$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$

where higher cumulative token entropy implies greater uncertainty in the generated answer (Xu, 30 Aug 2025). AH-GRPO uses a normalized per-position form,

$$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$

so that entropy is directly comparable across vocabularies and models (Chawla et al., 3 Jun 2026). STARE likewise treats per-token entropy as the entropy of the next-token policy at a specific context $c=(x,o_{<t})$,

$H = -\sum_v \pi_v \ln \pi_v,$

and studies its local training dynamics under GRPO-style updates (Luo et al., 17 Jun 2026).

Other domains redefine the token object itself. Token Entropy Regularization computes entropy after applying SoftMax along a token’s channel dimension,

$$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$

so high entropy means diffuse channel activations and low entropy means concentrated, sparse activations (Chen et al., 29 Jan 2026). EntropyPrune uses matrix entropy of a visual token’s head-wise covariance,

$$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$

interpreting higher spectral entropy as greater token informativeness (Wang et al., 19 Feb 2026).

The psycholinguistic literature adds a separate caution. First-token entropy over a subword vocabulary is widely used as a proxy for contextual word entropy, but it is a lower bound on true word entropy and can distort the predictor because multi-token words carry additional uncertainty beyond the first subword (Clark et al., 29 Jul 2025). This establishes that granularity is not a merely notational choice; it changes the quantity being measured.

2. Uncertainty quantification and calibrated prediction

Per-token entropy has become a practical uncertainty signal for open-ended generation, especially when model internals are unavailable. TECP introduces Token-Entropy Conformal Prediction for black-box LLMs, using cumulative token entropy as a logit-free, reference-free uncertainty score and then calibrating it with split conformal prediction to obtain prediction sets with finite-sample coverage guarantees (Xu, 30 Aug 2025).

The TECP pipeline begins with $M$ sampled candidate outputs $\hat{y}_1,\ldots,\hat{y}_M$ for an input $x$. It computes one entropy-derived score per candidate and places the scores of semantically correct calibration candidates into a multiset $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$0. For target coverage $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$1, it selects an empirical quantile threshold with a “higher” interpolation rule and retains only candidates whose entropy score is below that threshold:

$$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$2

Under exchangeability, the paper states that this yields nominal coverage of at least $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$3 (Xu, 30 Aug 2025).

The methodological significance of TECP lies in its contrast with self-consistency or frequency-based uncertainty. The paper argues that frequency alone can be misleading when a model is confidently wrong and repeatedly emits the same hallucinated answer; repetition then appears to signal confidence even though the answer is inaccurate. Token entropy instead reflects uncertainty in the autoregressive token-selection process, and TECP reports lower variance across random seeds and calibration/test splits than the ConU baseline, with risk curves aligning more closely to the theoretical line $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$4 (Xu, 30 Aug 2025).

Empirically, TECP reports stable coverage and compact prediction sets across six LLMs on TriviaQA and CoQA. In calibration/test split ablations, coverage remains within roughly $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$5 for most models. Prediction set size shows the intended monotonic shrinkage as $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$6 increases, from about $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$7 at $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$8 to about $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$9 at $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$0, with stronger models often producing more selective sets at medium-to-high risk levels (Xu, 30 Aug 2025).

The same paper also exposes an important limitation that recurs across the broader literature: although the method is framed as black-box and logit-free at the system level, the entropy formula presupposes access to token predictive distributions. TECP therefore assumes that entropy-derived scores are reliably extractable and stable across models and tasks; the authors note that this assumption may fail under distribution shift, non-standard decoding, or training regimes such as instruction tuning or reinforcement learning that distort token-probability behavior (Xu, 30 Aug 2025).

3. Reinforcement learning, exploration, and entropy collapse

In reinforcement learning for LLMs, per-token entropy is primarily treated as a signal for exploration stability and token-level credit assignment. ETPO formalizes this perspective by applying entropy-regularized RL at the token level rather than at the whole-sequence level. Its token-conditioned soft $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$1-value,

$$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$2

supports a per-token soft Bellman update in which KL regularization to a reference policy is applied at each token generation step. The paper argues that this decomposition matches autoregressive generation, improves credit assignment, and reduces exploration cost from exponential $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$3 sequence search to linear $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$4 token-wise exploration (Wen et al., 2024).

Subsequent RLVR work shifts from token-level regularization to token-level entropy dynamics. DLER identifies entropy collapse as one of three key RL failure modes under truncation-based length penalties, alongside large bias in advantage estimation and sparse reward signal. It characterizes clipped tokens as often low-probability, high-entropy transitional tokens such as “Wait,” “Hmm,” “Alternatively,” “thus,” and “also,” and argues that suppressing their gradients prematurely narrows exploration. Its higher clipping threshold is reported to alleviate entropy collapse: average token entropy does not vanish and can even rise after an initial drop, while the token entropy distribution remains right-skewed, with a small high-entropy minority carrying much of the exploratory burden (Liu et al., 16 Oct 2025).

AH-GRPO and SA-AH-GRPO turn per-token entropy directly into a policy-gradient weight. They define a cumulative entropy-based discount

$$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$5

so uncertain prefixes shorten the effective learning horizon by down-weighting later token contributions. SA-AH-GRPO applies this attenuation only to negative-advantage rollouts, leaving successful trajectories unattenuated. On the 3B model, this yields training variance $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$6, a $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$7 reduction relative to GRPO’s $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$8, while matching peak accuracy (Chawla et al., 3 Jun 2026).

STARE provides a more explicit token-level analysis of entropy drift under GRPO. Starting from the local logit update $$H_t^{(i)} = \frac{H(\pi_\theta(\cdot \mid q,\, o_{i,<t}))}{\log V} \in [0,1],$$9, it derives the first-order entropy variation

$c=(x,o_{<t})$0

where $c=(x,o_{<t})$1 is the sampled-token probability and $c=(x,o_{<t})$2 is an entropy sensitivity function (Luo et al., 17 Jun 2026). This yields a four-quadrant structure: positive-advantage low-surprisal tokens decrease entropy, positive-advantage high-surprisal tokens increase entropy, negative-advantage low-surprisal tokens increase entropy, and negative-advantage high-surprisal tokens decrease entropy. Because low-surprisal tokens are sampled more frequently, vanilla GRPO develops a systematic bias toward entropy decrease. STARE reweights the entropy-increasing minority and adds a target-entropy gate, reporting that even mild weights such as $c=(x,o_{<t})$3 can mitigate collapse, while the default $c=(x,o_{<t})$4, $c=(x,o_{<t})$5, $c=(x,o_{<t})$6 stabilizes training over thousands of steps (Luo et al., 17 Jun 2026).

A recurring implication across these methods is that entropy is not only a descriptive uncertainty statistic. In RL post-training, it behaves as a dynamical quantity governing whether gradients preserve exploratory branching or drive the policy toward premature concentration (Wen et al., 2024, Liu et al., 16 Oct 2025, Chawla et al., 3 Jun 2026, Luo et al., 17 Jun 2026).

4. Selective token updating beyond entropy-only heuristics

A major development after entropy collapse analyses is the move from entropy as a scalar score to entropy-coupled token selection rules. ICT argues that Shannon entropy is too coarse for selecting RL-critical tokens because two token distributions can share the same entropy while inducing different reasoning trajectories. It therefore uses Jensen–Shannon divergence from the group-average token distribution,

$c=(x,o_{<t})$7

and keeps the top $c=(x,o_{<t})$8 most “unique” tokens for gradient updates. Updating only this subset yields average pass@4 gains of $c=(x,o_{<t})$9 and a maximum gain of $H = -\sum_v \pi_v \ln \pi_v,$0 over GRPO, 20-Entropy, and STAPO across seven reasoning benchmarks (Feng et al., 18 Jun 2026).

RSI-S resolves a different tension: entropy-based methods prefer high-uncertainty positions, whereas probability-based methods seek to suppress unstable low-probability samples. Its Relative Surprisal Index explicitly couples predictive entropy with the realized sampled-token probability,

$H = -\sum_v \pi_v \ln \pi_v,$1

Tokens are retained only if $H = -\sum_v \pi_v \ln \pi_v,$2, which removes both overly predictable low-surprisal tokens and unstable high-surprisal tail tokens. The paper reports $H = -\sum_v \pi_v \ln \pi_v,$3–$H = -\sum_v \pi_v \ln \pi_v,$4 percentage-point avg@32 gains over GRPO on AIME and AMC across Qwen2.5-1.5B, 3B, and 7B (Lv et al., 30 Jun 2026).

EGSPO uses per-token entropy as a routing signal in hybrid SFT–RL training. It computes

$H = -\sum_v \pi_v \ln \pi_v,$5

sends the top $H = -\sum_v \pi_v \ln \pi_v,$6 most uncertain tokens to PPO-style RL, and sends low-entropy tokens to SFT, while enforcing a minimum SFT fraction $H = -\sum_v \pi_v \ln \pi_v,$7. The paper reports that about $H = -\sum_v \pi_v \ln \pi_v,$8 of tokens have low entropy $H = -\sum_v \pi_v \ln \pi_v,$9, about $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$0 are high-entropy tokens, and high-entropy tokens exhibit $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$1 higher variance in process reward model rewards, with Pearson correlation $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$2, $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$3. Removing entropy gating reduces AIME from $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$4 to $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$5, and random token selection reaches only $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$6 (Hu et al., 3 Feb 2026).

Probability–entropy calibration appears in supervised fine-tuning as well. RankTuner treats token entropy as intrinsic uncertainty and ground-truth probability as downstream alignment, then compares the realized rank of the ground-truth token with its expected rank under the predictive distribution. Its Relative Rank Indicator and inverse Relative Scale reweight the token-level loss so that tokens are emphasized when they are under-learned relative to the ambiguity of the position, not merely low-probability in absolute terms. The paper reports that removing the entropy term harms performance across all $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$7, and that the calibrated signal is less prone than entropy-dominant or probability-dominant baselines to surfacing injected noise as important (Yu et al., 2 Feb 2026).

Taken together, these methods suggest that entropy is rarely sufficient in isolation for selective optimization. Recent token-selection work systematically combines entropy with sampled-token probability, group-relative deviation, reward variance, or rank-based calibration (Feng et al., 18 Jun 2026, Lv et al., 30 Jun 2026, Hu et al., 3 Feb 2026, Yu et al., 2 Feb 2026).

5. Inference-time scheduling, decoding control, pruning, and compression

Per-token entropy also serves as a runtime control variable during inference. EntMTP uses local generation entropy to adapt multi-token prediction depth in speculative decoding. Rather than committing to a static draft-tree topology, it uses block-start entropy and entropy with exponential smoothing, with windows $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$8, to choose among a bank of Pareto-optimal trees. The core intuition is explicit: low-entropy regions support reliable multi-step drafting, whereas high-entropy regions require smaller, more conservative trees. Evaluated on Humaneval, ShareGPT, GSM8k, and Litbench, EntMTP reports a consistent $$\mathbf{p}_{m,t} = \text{SoftMax}(\mathbf{f}_{m,t}), \qquad H_{m,t} = -\sum_{i=1}^{D} p_{m,t,i}\log(p_{m,t,i}),$$9 speedup against Hydra and a peak $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$0 speedup against Medusa while preserving lossless generation quality and keeping continuation perplexity within about $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$1 nats of the base model or Hydra (Chen, 25 Jun 2026).

GELATO introduces a two-loop device–edge speculative decoding framework in which the inner loop is driven by per-token generative entropy of the draft model:

$$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$2

It posits a monotone decreasing entropy-to-acceptance map $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$3, and in experiments fits acceptance with $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$4. Drafting stops early when the uncertainty backlog

$$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$5

exceeds a safety threshold, which determines an adaptive draft length $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$6. This entropy-driven early exit is nested inside an outer Lyapunov scheduler that manages long-term energy-throughput trade-offs. GELATO reports $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$7 higher token throughput and $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$8 lower energy consumption than prior distributed speculative decoding systems under resource-constrained environments, while preserving decoding quality (Tang et al., 11 May 2026).

Entropy can also rank non-textual tokens for pruning. EntropyPrune identifies an “Entropy Collapse Layer” where matrix entropy of visual representations drops sharply, using that layer as a principled pruning point. At that layer, each visual token is scored by matrix entropy of its head-wise covariance, and low-score tokens are pruned without attention maps. By exploiting the spectral equivalence between dual Gram matrices, the paper reduces entropy computation from $$I(\mathbf{x}_i)= -\mathrm{tr}(\boldsymbol{\Sigma}_i \log \boldsymbol{\Sigma}_i) = -\sum_t \sigma_t \log \sigma_t,$$9 to $M$0, giving a $M$1 theoretical speedup for LLaVA-1.5-7B with $M$2 and $M$3. On LLaVA-1.5-7B, the method reports a $M$4 reduction in FLOPs while preserving $M$5 of original performance (Wang et al., 19 Feb 2026).

Decoding control can target information density rather than throughput. Entropy-UID defines entropy

$M$6

and surprisal

$M$7

then chooses tokens by minimizing

$M$8

subject to entropy and UID-like thresholds. Across WikiText-2, OpenWebText, and WMT, it reports the lowest entropy standard deviation and lowest surprisal standard deviation among compared methods, interpreting these reductions as smoother information density over generated sequences (Shou, 20 Feb 2025).

A separate but related line replaces entropy with an entropy-like proxy. Entropy Gate does not use Shannon per-token entropy in its main mechanism; instead it assigns each token an information energy $M$9 from statistical, structural, and positional factors, and in an idealized second phase equates the optimal score with surprisal $\hat{y}_1,\ldots,\hat{y}_M$0. Tokens survive a quenching schedule according to descending energy or Boltzmann survival, while compression halts when an energy-weighted similarity score $\hat{y}_1,\ldots,\hat{y}_M$1 falls below threshold. Its Phase 1 heuristic reports $\hat{y}_1,\ldots,\hat{y}_M$2–$\hat{y}_1,\ldots,\hat{y}_M$3 compression across prompt categories while maintaining $\hat{y}_1,\ldots,\hat{y}_M$4 (Agyemang et al., 2 Jun 2026). A plausible implication is that some work described as entropy-guided token control is in practice driven by related information measures such as surprisal or energy.

6. Multimodal learning, masking, safety, and measurement caveats

In multimodal pretraining, per-token entropy is often treated as a representation-shaping signal. Token Entropy Regularization defines entropy over channel-wise token features and uses it both intrinsically, via an entropy-augmented residual update, and extrinsically, via a Token Entropy Loss

$\hat{y}_1,\ldots,\hat{y}_M$5

The method encourages low-entropy, compact token encodings and reports improved pretraining accuracy across all tested video encoders. For example, TimeSformer improves from $\hat{y}_1,\ldots,\hat{y}_M$6 to $\hat{y}_1,\ldots,\hat{y}_M$7 Acc@1 and from $\hat{y}_1,\ldots,\hat{y}_M$8 to $\hat{y}_1,\ldots,\hat{y}_M$9 Acc@3, while Video Swin + TER reaches $x$0 Acc@1. The same study reports that the entropy of the first token is modality-dependent and interprets this as evidence that different modalities have different representation complexity (Chen et al., 29 Jan 2026).

Entropy-aware masking brings token uncertainty into masked language modeling. Instead of random masking, tokens are ranked by entropy

$x$1

with high-entropy masking, low-entropy masking, mid-entropy masking, marginal entropy masking, and alternating schemes as alternatives. High-entropy masking is the strongest single strategy; teacher-initialized self-masking slightly surpasses teacher-masking alone; and combining entropy-aware masking with knowledge distillation yields the best overall results. The paper reports an average GLUE improvement of $x$2 over the baseline (Srinivasagan et al., 27 May 2026).

Per-token entropy also exposes safety-relevant instabilities. EGA argues that prior entropy-based attacks on vision-LLMs are too diffuse because they maximize uncertainty at all decoding steps. Its main empirical claim is that only about $x$3 of tokens, the top $x$4 highest-entropy positions, function as critical decision points in autoregressive generation. Concentrating perturbations on these positions yields semantic degradation comparable to global methods while converting $x$5–$x$6 of benign outputs into harmful ones across representative VLMs, with transfer harmful rates of $x$7–$x$8 on unseen targets (He et al., 26 Dec 2025).

Measurement caveats remain substantial. In psycholinguistics, first-token entropy over subwords is often used for convenience, yet Monte Carlo word-entropy estimates that allow variable numbers of tokens show that first-token entropy can underestimate and distort the true contextual entropy of words. Regression results on reading times diverge between first-token and MC word entropy, and permutation tests report significant differences for both Shannon entropy $x$9 and Rényi entropy $$U(\hat{y}_m) = \sum_{t=1}^{L_m} H_t = -\sum_{t=1}^{L_m}\sum_{v\in \mathcal{V}} p_t(v)\,\log p_t(v),$$00 (Clark et al., 29 Jul 2025). More broadly, this suggests that conclusions about “per-token entropy effects” are highly sensitive to the level at which the token is defined.

Across the literature, the main misconception is therefore not that entropy is uninformative, but that all entropy-like token scores are interchangeable. The evidence instead shows a family resemblance among several local uncertainty measures whose operational meaning depends on the tokenization level, the representation space, and the decision problem being optimized (Xu, 30 Aug 2025, Chen et al., 29 Jan 2026, Clark et al., 29 Jul 2025).