Entropy-Aware Score Selection

• Entropy-aware score selection is a family of methods that adjust raw scores using entropy-based measures to account for uncertainty, dispersion, and sample coverage.
• It employs foundational concepts like Shannon entropy and mutual information to guide decisions in feature, token, and benchmark selection across tasks such as language modeling and multimodal fusion.
• Empirical studies show that approaches such as CASMI and aver scoring improve model reliability and efficiency compared to traditional relevance or fidelity scoring methods.

Entropy-aware score selection is an information-theoretic design pattern in which selection, routing, pruning, or fusion is driven by entropy, entropy-derived uncertainty, or entropy-like surrogates rather than by relevance or fidelity scores alone. In the literature surveyed here, it appears as Coverage Adjusted Standardized Mutual Information (CASMI) for small-sample feature selection, entropy reduction via the aver score for association, log-determinant or mutual-information objectives for benchmark subset selection, entropy- and varentropy-gated multimodal fusion for speech emotion recognition, entropy-aware masking for masked language modeling, prompt-aware kernel-entropy guidance in diffusion sampling, and entropy-based token pruning in vision–LLMs (Shi et al., 2019, Gamst et al., 7 Nov 2025, Smola, 4 May 2026, Chua et al., 28 Aug 2025, Srinivasagan et al., 27 May 2026, Jalali et al., 11 Jun 2025, Wang et al., 2 Jul 2026). This suggests that the term is best understood as a family of methods in which a score is adjusted to reflect concentration, dispersion, coverage, redundancy, or uncertainty.

1. Conceptual and mathematical foundations

A recurring starting point is Shannon entropy. For a discrete random variable $X$ with probabilities $p(x)$, entropy is

$H(X) = - \sum_x p(x) \log p(x).$

In feature selection, this is paired with mutual information,

$$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$

and then normalized as

$\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$

which is bounded in $[0,1]$ and interpreted as predictability of $Y$ from $X$ (Shi et al., 2019).

The same family of ideas also appears in alternative score constructions. In document or graph association, the aver score is defined as an entropy reduction,

$\mathrm{aver}(A,D) := E - E',$

where $E$ is the entropy of a fitted rank-one bag-of-words model and $p(x)$0 is the entropy after introducing a collaboration document $p(x)$1 for a set $p(x)$2 of documents (Gamst et al., 7 Nov 2025). In prompt-aware diversity guidance, order-2 Rényi kernel entropy is used through the Renyi Kernel Entropy (RKE) score,

$p(x)$3

together with conditional variants based on Hadamard products of prompt and latent kernels (Jalali et al., 11 Jun 2025). In sequence complexity analysis, the entropy-rank ratio

$p(x)$4

places a sequence within the full entropy distribution at fixed length and alphabet size, producing a distribution-aware measure bounded in $p(x)$5 under fixed settings (Pastore et al., 7 Nov 2025).

Several works emphasize normalization and interpretability. CASMI inherits boundedness from $p(x)$6 and scales it by sample coverage, while the entropy-rank ratio explicitly avoids saturation effects that limit classical Shannon entropy for long uniform DNA segments (Shi et al., 2019, Pastore et al., 7 Nov 2025). In clustering, a parameter-free score balances a within-cluster Gaussian dispersion term with a between-cluster entropy term,

$p(x)$7

so that cluster-size entropy acts as a regularizer against trivial over-partitioning (Noble et al., 2019).

2. Feature, variable, and benchmark selection

The most explicit feature-selection formulation is CASMI, introduced for small-sample healthcare datasets and non-ordinal categorical data. Its central score is

$p(x)$8

where $p(x)$9 is sample coverage, estimated by the Good–Turing/Turing singleton-based estimator

$H(X) = - \sum_x p(x) \log p(x).$0

The intended effect is to retain features with predictive information that is well preserved in the sample and to downweight features with low coverage, including ID-like variables dominated by singletons (Shi et al., 2019).

CASMI also depends on an improved entropy estimator. The plug-in estimator is biased downward under small $H(X) = - \sum_x p(x) \log p(x).$1, whereas the paper adopts Zhang’s $H(X) = - \sum_x p(x) \log p(x).$2, whose bias decays exponentially rather than at rate $H(X) = - \sum_x p(x) \log p(x).$3. Mutual information is then estimated by

$H(X) = - \sum_x p(x) \log p(x).$4

and the operational score becomes

$H(X) = - \sum_x p(x) \log p(x).$5

The selection pipeline first eliminates independent features through an MI-based chi-squared test and then performs greedy joint-distribution selection with an automatic stopping rule. Because the greedy stage evaluates joint CASMI of already selected features with each candidate, redundancy is handled at the level of the joint distribution and is not limited to linear dependence (Shi et al., 2019).

The empirical evaluation in that work uses the Information Recovery Ratio,

$H(X) = - \sum_x p(x) \log p(x).$6

and reports that CASMI consistently achieves higher or equal average IRR than MIM, JMI, CMIM, MRMR, DISR, and NJMIM, especially at small $H(X) = - \sum_x p(x) \log p(x).$7. The same study reports that CASMI reaches average IRR $H(X) = - \sum_x p(x) \log p(x).$8 at the smallest sample size among the compared methods in that simulation, and that average R runtimes ranged from $H(X) = - \sum_x p(x) \log p(x).$9 s to $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$0 s, with maximum $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$1 s (Shi et al., 2019).

A related but distinct entropy-aware subset-selection problem appears in benchmark selection under a multivariate Gaussian model. There, differential entropy of a selected subset $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$2 is

$$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$3

so maximizing entropy at fixed cardinality is equivalent to maximizing $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$4. The resulting entropy objective is submodular, greedy entropy selection coincides with pivoted Cholesky factorization, and spectral residual trace diagnostics arise naturally from the residual covariance. The same framework also studies mutual information between selected and remaining benchmarks, which is non-monotone in general but empirically monotone for small subsets; experiments on matrices derived from ten public leaderboards show that mutual-information selection outperforms entropy for imputation at small subsets (Smola, 4 May 2026).

Entropy-aware selection also appears in representation learning. In Sampled-SAE, the columns of the batch activation matrix are scored via $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$5 norm or entropy, a candidate pool of size $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$6 is formed, and Top-$$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$7 selection is then applied across the batch from that restricted pool of features. Varying $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$8 traces a spectrum between batch-level and token-specific selection: $$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)} = H(X) + H(Y) - H(X,Y),$$9 enforces global consistency, while larger $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$0 moves toward standard BatchTopK and more token-specific behavior (Oozeer et al., 29 Aug 2025).

3. Association, ranking, and decision scores

Entropy-aware score selection is not confined to feature subsets. In "Association via Entropy Reduction" (Gamst et al., 7 Nov 2025), the score aver is derived from a simple statistical model rather than from heuristic weighting. The model assumes independent draws of terms and documents, fits $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$1 and $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$2, computes the entropy of the fitted rank-one model, and then introduces a collaboration transformation for a set $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$3 of documents through

$\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$4

Association is declared when the entropy decreases under that transformation, i.e.

$\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$5

The paper emphasizes that this sign criterion supplies a natural threshold that tf-idf does not provide, and that aver can distinguish document pairs for which tf-idf saturates at $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$6 (Gamst et al., 7 Nov 2025).

The same work also extends aver from pairs to sets $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$7 of arbitrary size without altering the underlying transformation. On the Orkut dataset, candidate pairs with at least 100 common friends yielded $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$8 pairs, of which $\kappa(X,Y) = \frac{I(X;Y)}{H(Y)},$9 were positives and $[0,1]$0 negatives. The paper reports that aver consistently attained higher true positive rates than tf-idf at matched false positive rates among the highest-scoring pairs, and that at the natural threshold $[0,1]$1, about half of the surviving positives were true positives (Gamst et al., 7 Nov 2025).

A different decision-scoring formulation appears in large-language-model option selection. The Grade Score computes entropy over the distribution of selected option positions,

$[0,1]$2

normalizes it by $[0,1]$3, defines a Choice Score as the mode frequency over option identities, and combines the two through the harmonic mean

$[0,1]$4

The entropy term measures order bias, while the mode frequency measures choice stability. Across the reported experiments, adding one unrelated option generally improved Grade Score, largely by raising normalized entropy without markedly degrading choice stability (Iourovitski, 2024).

The entropy-rank ratio provides another example of distribution-aware normalization. Rather than using only the entropy value of a DNA window, it computes the proportion of all sequences of the same length whose entropy is equal to or lower than that of the target. The resulting rank-style score is bounded in $[0,1]$5 at fixed $[0,1]$6, and the paper integrates it into ratio-guided cropping for sequence-classification data augmentation (Pastore et al., 7 Nov 2025). This suggests a broader pattern: entropy-aware score selection often replaces raw entropy by a score calibrated against a reference distribution, a coverage term, or a decision threshold.

4. Gating, routing, and sequential allocation

In multimodal speech emotion recognition, Entropy-Aware Score Selection (EASS) is a late score-fusion rule for deciding when to trust a primary acoustic model and when to defer to a secondary text-based sentiment model. The primary pipeline is wav2vec2.0 with a classifier over $[0,1]$7, producing a softmax vector $[0,1]$8. The secondary pipeline uses Whisper-large-v3 for transcription and RoBERTa-XLM for sentiment over $[0,1]$9. EASS computes entropy

$Y$0

and varentropy

$Y$1

then marks a prediction as unreliable when entropy is high and varentropy is low relative to class-wise thresholds (Chua et al., 28 Aug 2025).

The fusion rule is a hard gate:

• if $Y$2 or $Y$3, retain the primary prediction;
• otherwise defer to the text pipeline and map sentiment to emotion, using either a refer-to-primary disambiguation for Negative or a thresholded simple-or-flip rule.

On IEMOCAP, the combined entropy-plus-varentropy version reported UA $Y$4, WA $Y$5, and F1 $Y$6, compared with acoustic-only UA $Y$7, WA $Y$8, and F1 $Y$9. On MSP-IMPROV, the paper reports average improvements of $X$0 UA, $X$1 WA, and $X$2 F1 percentage points (Chua et al., 28 Aug 2025).

Entropy-aware routing also appears in online model selection. DAK-UCB formulates prompt-conditioned model choice as a contextual bandit problem with a reward that combines fidelity and diversity,

$X$3

The diversity term can be based on joint kernel distance (JKD) or on inverse joint Rényi kernel entropy (I-JRKE), both defined over prompt–output pairs via product kernels. The crucial property is that these diversity primitives admit two-sample expectation decompositions, enabling unbiased prompt-level labels from single sampled outputs. Per-arm kernel ridge regression is then used to build UCB indices that combine fidelity and diversity with confidence bonuses (Jafari et al., 24 Mar 2026).

A further routing-allocation variant appears in Annealed Entropic Allocation for ranking and selection. There, the hard maximin objective over challenger-specific large-deviation rates is replaced by a weighted log-sum-exp soft-min,

$X$4

with Gibbs weights

$X$5

The smoothing mitigates hard switching when several challengers are nearly active, while annealing $X$6 preserves the classical first-order large-deviation target. Numerical experiments on Gaussian and exponential instances show competitive performance, especially when multiple challengers are nearly tied (Fei et al., 9 Jun 2026).

5. Training-time score modulation and generative modeling

Entropy-aware score selection also acts directly on training signals. In masked language modeling, the central choice is which tokens to mask. Entropy-aware masking computes a token-level predictive entropy

$X$7

then selects the top-$X$8 highest-entropy positions subject to the mask ratio. The paper contrasts high-, low-, mid-, marginal, and alternating-entropy policies, and introduces a self-masking mechanism in which the student model computes its own entropies after an initialization phase driven by a teacher model (Srinivasagan et al., 27 May 2026).

The reported empirical pattern is asymmetric. For BERT, high-entropy masking improved average GLUE from $X$9 to $\mathrm{aver}(A,D) := E - E',$0, alternating high/low masking reached $\mathrm{aver}(A,D) := E - E',$1, and mid-entropy masking collapsed to $\mathrm{aver}(A,D) := E - E',$2. For BERTlet, self-masking with teacher initialization reached $\mathrm{aver}(A,D) := E - E',$3, compared with a baseline of $\mathrm{aver}(A,D) := E - E',$4. When knowledge distillation was added after initialization, BERTlet reached $\mathrm{aver}(A,D) := E - E',$5, which the paper reports as the best overall result (Srinivasagan et al., 27 May 2026).

In diffusion modeling, Ding, Quan, and Tu derive a thermodynamic interpretation of score matching by defining time-asymmetry entropy production (TAEP) as a path-space log-ratio between forward and learned reverse diffusion dynamics. Their main identity is that the average TAEP is exactly proportional to the score-matching objective,

$\mathrm{aver}(A,D) := E - E',$6

The same framework shows that TAEP obeys exact fluctuation theorems and that $\mathrm{aver}(A,D) := E - E',$7 quantifies sampling unevenness and provides a thermodynamic measure of data-manifold coverage. On that basis, the paper proposes TAEP-guided timestep weighting, architecture and regularization selection, noise-schedule adaptation, and sample acceptance based on trajectory-level $\mathrm{aver}(A,D) := E - E',$8 (Ding et al., 15 Jun 2026).

A related line studies diversity guidance in prompt-conditioned diffusion. SPARKE uses conditional entropy for diversity guidance through conditional latent RKE score guidance. The conditional score is based on a Hadamard product of prompt and latent kernels, and the special case studied in the paper reduces the computational complexity of entropy computation and gradient-based optimization from $\mathrm{aver}(A,D) := E - E',$9 to $E$0. The reported goal is prompt-aware diversity control across semantically similar prompts without prohibitive computational overhead (Jalali et al., 11 Jun 2025).

These generative-model examples shift entropy-aware score selection from static ranking to dynamic score modulation. The score is not merely evaluated after generation; it alters masking choices, diffusion guidance, timestep weighting, or sample filtering during learning and sampling.

6. Structured compression, redundancy control, and cross-cutting limitations

A distinct strand of work uses entropy-aware scores to combat redundancy under strict compute budgets. In Entropy-Aware Dense Pruning (EADP), dense text–vision similarity maps are converted into per-text-token spatial distributions,

$E$1

and their entropies

$E$2

are used to identify dispersed textual noise. Low-entropy text tokens are retained by a lower-tail quantile filter and reweighted via

$E$3

yielding a dense relevance score that is then fused with global guidance, spatially smoothed, polarized, and finally passed to a facility-location objective for submodular token selection (Wang et al., 2 Jul 2026).

The paper argues that naive Top-$E$4 patch selection causes feature fragmentation and redundancy, whereas facility-location selection enforces coverage:

$E$5

Empirically, EADP improves the accuracy-efficiency trade-off across LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, Qwen3-VL, and LLaVA-Video under tight token budgets, while the runtime breakdown reported for Qwen3-VL-8B indicates that scoring, smoothing, and polarization cost about $E$6 ms and facility-location costs about $E$7–$E$8 ms depending on $E$9 (Wang et al., 2 Jul 2026).

Entropy-aware selection can also operate at the dataset level through compression. The entropy law for LLM data selection links downstream performance to compression ratio and first-epoch training loss. ZIP operationalizes this through a greedy three-stage algorithm that first scores individual samples by compression ratio, then updates candidate scores by the union compression ratio $p(x)$00, and finally performs fine-grained greedy minimization of redundancy within a small candidate set. In the reported experiments, ZIP outperformed Random, Cluster, Perplexity, SuperFiltering, and DEITA on SFT selection, and exceeded Random and Score on RLHF selection, while running CPU-only in about $p(x)$01 h for SFT and $p(x)$02 h for RLHF (Yin et al., 2024).

Across the literature, several recurring limitations appear. CASMI relies on discretization for continuous data, and excessive binning causes many singletons and low coverage; very high-dimensional settings with ultra-sparse categories also increase the computational burden of $p(x)$03 (Shi et al., 2019). Aver is naturally scale-free, which the paper notes makes its scores more complicated to interpret, and writing down and computing aver is more complex than tf-idf (Gamst et al., 7 Nov 2025). In entropy-aware masking, mid-entropy masking can be harmful and self-masking suffers from a cold-start problem without an initialization phase (Srinivasagan et al., 27 May 2026). EADP inherits $p(x)$04 selection cost when similarities are precomputed, while DAK-UCB requires per-arm kernel regressors and archives of past prompt–output pairs, with practical scalability depending on Nyström or random-feature approximations (Wang et al., 2 Jul 2026, Jafari et al., 24 Mar 2026).

Taken together, these works present entropy-aware score selection as a general methodology for making scores sensitive to uncertainty, concentration, redundancy, or sample coverage. In some settings the entropy term regularizes a relevance score; in others it supplies a gating criterion, a routing objective, a pruning prior, or a finite-budget surrogate for a harder combinatorial or sequential decision problem. The common mechanism is not the use of a single entropy formula, but the systematic use of entropy-derived structure to decide which information should be trusted, emphasized, retained, or discarded (Shi et al., 2019, Smola, 4 May 2026, Chua et al., 28 Aug 2025, Srinivasagan et al., 27 May 2026, Wang et al., 2 Jul 2026).