Entropy-Based Rejection

• Entropy-based rejection is a family of techniques that use entropy measures to decide acceptance, abstention, or rejection in various statistical and machine learning tasks.
• It leverages distinct entropy measures—from predictive and CRPS-induced entropy to per-label binary entropy—to balance uncertainty and computational efficiency.
• Applications span independence testing, selective classification, multi-label imaging, and entropy-optimal coding, offering both theoretical guarantees and practical improvements.

Entropy-based rejection denotes a family of procedures in which acceptance, abstention, or hypothesis rejection is driven by an entropy quantity, an entropy-derived uncertainty score, or an entropy-regularized discrepancy. Taken together, recent work suggests that the term does not identify a single algorithmic paradigm, but rather several technically distinct mechanisms: rejecting independence when a centered entropic optimal-transport discrepancy is large; abstaining from predictions whose predictive entropy is high; thresholding per-label binary entropy in multi-label medical imaging; rejecting distributional forecasts with large CRPS-induced entropy; and using rejection sampling schemes whose acceptance rate or codelength is controlled by information measures such as Shannon entropy, total variation, or functional information (Liu et al., 2021, Tornetta, 2021, Aperstein et al., 12 Sep 2025, Zaoui et al., 31 Mar 2025, Hill et al., 25 Apr 2026).

1. Scope and taxonomy

A useful synthesis is to treat entropy-based rejection as a decision layer built on top of an uncertainty or discrepancy functional. In some settings the functional is a property of a predictive distribution, in others it is a property of a pair of distributions, and in still others it is an information-theoretic quantity governing the efficiency of a rejection-sampling code.

Domain	Entropy-related quantity	Rejection criterion
Independence testing	Sinkhorn divergence $T_{n,\varepsilon}(X,Y)$	Reject if $T_n(X,Y)>H_n(\alpha)$
Probabilistic classification	Predictive entropy $H(Y\mid x)$	Reject if $H(Y\mid x)>\tau_H$
Multi-label chest X-ray classification	Per-class binary entropy $H(p_c)$	Reject if all class entropies exceed thresholds
Distributional regression	CRPS entropy $\ent(F_x^*)$	Reject if $\ent(F_x^*)>\lambda_\varepsilon$
Entropy equivalence testing	Entropy gap $|H(p)-H(q)|$	Reject $p=q$ if a gap $\ge \varepsilon$ is detected

This taxonomy also clarifies that “entropy” is not uniform across the literature. In probabilistic classification it is the Shannon entropy of $T_n(X,Y)>H_n(\alpha)$0; in distributional regression it is the CRPS-induced entropy $T_n(X,Y)>H_n(\alpha)$1; in entropic optimal transport it enters through the KL penalty $T_n(X,Y)>H_n(\alpha)$2; and in relative entropy coding it appears as mutual information, channel simulation divergence, or functional information, depending on the coding theorem under consideration (Tornetta, 2021, Zaoui et al., 31 Mar 2025, Liu et al., 2021, Hill et al., 25 Apr 2026).

2. Independence testing via entropic optimal transport

In independence testing, entropy-based rejection appears in the Entropy Regularized Optimal Transport Independence Criterion, which compares the joint law $T_n(X,Y)>H_n(\alpha)$3 to the product of marginals $T_n(X,Y)>H_n(\alpha)$4 through a centered entropy-regularized optimal transport discrepancy, i.e. a Sinkhorn divergence (Liu et al., 2021). The basic construction is

$T_n(X,Y)>H_n(\alpha)$5

and the dependence measure is

$T_n(X,Y)>H_n(\alpha)$6

The centering terms remove self-similarity bias, and under compactness/Lipschitz assumptions with positive universal Gibbs kernels one has

$T_n(X,Y)>H_n(\alpha)$7

This makes large positive values a direct certificate of dependence.

The induced hypothesis test is

$T_n(X,Y)>H_n(\alpha)$8

with empirical statistic

$T_n(X,Y)>H_n(\alpha)$9

and rejection rule

$H(Y\mid x)$0

A subtlety stressed in the paper is that $H(Y\mid x)$1 is built from $H(Y\mid x)$2 dependent pairs $H(Y\mid x)$3, so the analysis requires two-sample U-process tools rather than standard plug-in arguments. The theory gives non-asymptotic guarantees, null fluctuations of order $H(Y\mid x)$4, and consistency under the alternative; the experiments use Monte Carlo permutation resampling with 200 permutations for calibration (Liu et al., 2021).

The role of entropy is simultaneously statistical and computational. The KL penalty smooths the OT problem, yields unique and stable couplings and Schrödinger potentials, and enables Sinkhorn-type iterative solvers. The parameter $H(Y\mid x)$5 controls the OT–kernel interpolation: as $H(Y\mid x)$6, the criterion approaches the unregularized OT dependence measure; as $H(Y\mid x)$7, $H(Y\mid x)$8 for additive costs, so excessive regularization destroys signal. The paper also gives a Tensor Sinkhorn algorithm reducing naive $H(Y\mid x)$9 iterations to $H(Y\mid x)>\tau_H$0 time and $H(Y\mid x)>\tau_H$1 space, a positive-random-feature approximation with $H(Y\mid x)>\tau_H$2 time per iteration, and a differentiable implementation for deep learning applications (Liu et al., 2021).

A further structural point is the relation to kernel methods. For additive quadratic costs, the Gibbs kernels become Gaussian, and in the multiplicative-cost limit $H(Y\mid x)>\tau_H$3, ETIC recovers $H(Y\mid x)>\tau_H$4. This suggests that entropy-based rejection in this setting is not merely “thresholding an entropy,” but rejecting via a centered entropic OT discrepancy whose large deviations from zero encode dependence (Liu et al., 2021).

3. Predictive entropy as an abstention statistic in classification

For probabilistic classifiers, entropy-based rejection takes the more familiar form of abstention based on the entropy of the predictive class distribution. If a classifier outputs

$H(Y\mid x)>\tau_H$5

the standard hard decision

$H(Y\mid x)>\tau_H$6

discards the rest of the distribution. The relevant uncertainty score is the predictive entropy

$H(Y\mid x)>\tau_H$7

which is $H(Y\mid x)>\tau_H$8 for a point mass and $H(Y\mid x)>\tau_H$9 for the uniform distribution (Tornetta, 2021). The paper “Entropy methods for the confidence assessment of probabilistic classification models” treats lower entropy as sharper and therefore more confident prediction, and high entropy as a natural abstention signal. It does not formalize a per-example rejection theorem, but it explicitly motivates decision rules “where decisions need to be taken only when their confidence is relatively high,” and a natural selective classifier follows: $H(p_c)$0 Equivalent normalized forms use $H(p_c)$1 or $H(p_c)$2 as the ranking score (Tornetta, 2021).

The same paper emphasizes an important caveat: confidence and correctness are not identical. A model can be highly confident and wrong. Accordingly, entropy-based rejection complements rather than replaces accuracy and calibration analysis. That distinction is formalized in the contrast between entropy score, which measures average sharpness, and purity, which mixes confidence and correctness through the probabilistic confusion matrix (Tornetta, 2021).

The complement Naive Bayes analysis provides a mathematically explicit degradation mechanism. For the transformation

$H(p_c)$3

the paper proves

$H(p_c)$4

with equality only at fixed points. This yields a precise instance in which a modeling choice can improve raw accuracy in difficult or unbalanced settings while systematically producing flatter, higher-entropy outputs. The operational implication is direct: under an entropy-threshold rejection rule, such a model would accept fewer examples at a fixed confidence level (Tornetta, 2021).

A related but distinct line of work concerns training objectives that produce more useful confidence scores. “A margin-based replacement for cross-entropy loss” argues that cross-entropy keeps pushing logits toward extreme values and thereby induces overconfident predictions on unseen data, while the proposed HEM loss improves unknown-class and adversarial rejection when standard confidence scores such as MSP or MLS are used (Spratling et al., 21 Jan 2025). This suggests that entropy-based rejection may also benefit from training objectives that reduce pathological softmax peaking, although that paper does not evaluate predictive entropy directly.

4. Structured selective prediction: multi-label imaging and distributional regression

In multi-label chest X-ray classification, entropy-based rejection is implemented at the label level and then aggregated into an image-level accept/reject rule. The model outputs one probability $H(p_c)$5 per pathology, and uncertainty is measured by the per-class binary entropy

$H(p_c)$6

The paper is explicit that entropy is not aggregated across classes into a sum or mean. Instead, an image is accepted if at least one class prediction is considered confident: $H(p_c)$7 and rejected only if all class predictions are uncertain: $H(p_c)$8 Thresholds are calibrated by a quantile-based procedure on entropy values from correctly classified samples, using either global or class-specific thresholds, under an explicit rejection-rate budget (Aperstein et al., 12 Sep 2025).

The empirical picture is heterogeneous but informative. Across PadChest, NIH ChestX-ray14, and MIMIC-CXR, entropy-based rejection improves the trade-off between diagnostic quality and coverage, and the aggregate comparison reports average AUC $H(p_c)$9 for the baseline, $\ent(F_x^*)$0 for interval-based rejection, and $\ent(F_x^*)$1 for entropy-based rejection (Aperstein et al., 12 Sep 2025). The gains are largest and cleanest on MIMIC-CXR, while NIH exhibits stronger domain sensitivity. The paper also notes a structural limitation: the “accept-if-any-class-is-confident” rule is an image-level decision, not a strict per-label abstention mechanism.

Distributional regression with reject option yields a more formal entropy-thresholding theory. Under CRPS,

$\ent(F_x^*)$2

the expected score decomposes as

$\ent(F_x^*)$3

with

$\ent(F_x^*)$4

For the true conditional cdf $\ent(F_x^*)$5, $\ent(F_x^*)$6 is exactly the conditional Bayes risk under CRPS. The optimal penalized rejector therefore has the closed form

$\ent(F_x^*)$7

and at a fixed rejection rate $\ent(F_x^*)$8, the optimal threshold is the entropy quantile

$\ent(F_x^*)$9

under the continuity assumption on $\ent(F_x^*)>\lambda_\varepsilon$0 (Zaoui et al., 31 Mar 2025).

The estimation procedure is semi-supervised. A labeled sample is used to estimate the conditional distribution and its CRPS entropy, while an unlabeled sample is used to calibrate the threshold quantile. The paper proves distribution-free rejection-rate control,

$\ent(F_x^*)>\lambda_\varepsilon$1

and asymptotic optimality in both error rate and rejection rate. For distributional $\ent(F_x^*)>\lambda_\varepsilon$2-nearest neighbors, explicit excess-risk rates are also derived (Zaoui et al., 31 Mar 2025). This is one of the clearest examples in which entropy-based rejection is not heuristic but exactly Bayes-optimal for a prescribed scoring rule.

5. Rejection sampling under entropy and information constraints

A different lineage uses rejection not as a predictive abstention device but as the core computational primitive, with entropy or information controlling codelength, optimality, or acceptance. In relative entropy coding, Greedy Rejection Coding generalizes rejection-based channel simulation to arbitrary measurable spaces and partition trees. For continuous $\ent(F_x^*)>\lambda_\varepsilon$3 on $\ent(F_x^*)>\lambda_\varepsilon$4 with unimodal density ratio $\ent(F_x^*)>\lambda_\varepsilon$5, the GRCS variant satisfies

$\ent(F_x^*)>\lambda_\varepsilon$6

while retaining the optimal REC codelength form

$\ent(F_x^*)>\lambda_\varepsilon$7

This reframes rejection as an entropy-efficient coding mechanism rather than a mere Monte Carlo filter (Flamich et al., 2023).

The one-shot optimum is sharpened further in “Rejection Sampling is Optimal for Relative Entropy Coding,” which argues that the correct quantity is not mutual information $\ent(F_x^*)>\lambda_\varepsilon$8 but functional information $\ent(F_x^*)>\lambda_\varepsilon$9. For the ring toss code, a rejection-sampling encoder transmits the accepted index $|H(p)-H(q)|$0 with conditional entropy

$|H(p)-H(q)|$1

This yields the tight sandwich

$|H(p)-H(q)|$2

and implies an $|H(p)-H(q)|$3-bit guarantee for singular channels, where $|H(p)-H(q)|$4 (Hill et al., 25 Apr 2026). The companion BBRS construction for singular channels combines bits-back coding with rejection sampling and achieves the same asymptotic efficiency as the Sriramu–Wagner sampler with a direct one-shot bound

$|H(p)-H(q)|$5

and $|H(p)-H(q)|$6 excess over $|H(p)-H(q)|$7 in the product setting (Flamich et al., 7 Apr 2026).

In exact discrete sampling, “Efficient Rejection Sampling in the Entropy-Optimal Range” shows that the ALDR family can achieve

$|H(p)-H(q)|$8

with linearithmic preprocessing and space complexity, provided the amplification depth satisfies $|H(p)-H(q)|$9, $p=q$0. This places rejection sampling in the same entropy-optimal interval as Knuth–Yao while avoiding the latter’s exponential representation cost (Draper et al., 5 Apr 2025). In randomized quantization, rejection-sampled universal quantization uses dithered lattice proposals but accepts only errors in a target set $p=q$1, yielding exact error law $p=q$2 and entropy overhead bounded by $p=q$3. For $p=q$4, the method gives smaller maximum error than known lattice quantizers at the same entropy for dimensions $p=q$5–$p=q$6, and smaller MSE for dimensions $p=q$7–$p=q$8 in the high-resolution limit (Ling et al., 2024).

Entropy-controlled rejection also appears in speculative decoding during RL for LLMs. “Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling” identifies a negative linear relation between policy-entropy increase and target-only MTP acceptance, while rejection sampling changes the relevant acceptance law to

$p=q$9

With an end-to-end TV loss, the paper reports that the entropy-acceptance slope is reduced by over $\ge \varepsilon$0, acceptance can reach up to $\ge \varepsilon$1, and async RL training is accelerated by up to $\ge \varepsilon$2 end-to-end (Li et al., 10 Jun 2026). In this setting, entropy-based rejection is neither abstention nor testing; it is an acceptance mechanism whose robustness is analyzed through entropy and total-variation geometry.

6. Entropy as the test target, and major conceptual boundaries

A particularly literal form of entropy-based rejection is entropy equivalence testing, where the null is still $\ge \varepsilon$3, but the alternative is relaxed to an entropy gap: $\ge \varepsilon$4 The paper shows that this can be substantially easier than full closeness testing. Its tester combines Hellinger pretesting, heavy/light decomposition, a direct estimator for a cross-entropy-like term, and a bias-detection statistic, achieving sample complexity

$\ge \varepsilon$5

with matching lower bounds up to polylogarithmic factors (Canonne et al., 22 May 2026). This is a useful corrective to the common assumption that entropy-based rejection is always a confidence filter on predictions; here entropy is itself the statistical object being tested.

Several misconceptions recur across the literature. First, entropy is not correctness. Predictive entropy measures sharpness or uncertainty, not reliability, and confident errors remain possible (Tornetta, 2021). Second, entropy-based rejection is not equivalent to full distributional rejection: two distributions can be far in TV yet close in entropy, which is precisely why entropy equivalence testing is easier than closeness testing (Canonne et al., 22 May 2026). Third, more entropy regularization is not always beneficial. In ETIC, very large $\ge \varepsilon$6 drives $\ge \varepsilon$7 for additive costs and washes out dependence, while too small $\ge \varepsilon$8 worsens constants and computation (Liu et al., 2021). Fourth, image-level acceptance rules can obscure label-wise uncertainty: in multi-label chest X-ray classification, accepting when any class is confident is not the same as abstaining on each uncertain label separately (Aperstein et al., 12 Sep 2025).

A final boundary is physical rather than statistical. “Entropy Cost of ‘Erasure’ in Physically Irreversible Processes” argues that logical erasure is not by itself equivalent to erasure of thermodynamic entropy, and that a restricted Landauer principle applies only when ontological uncertainty in a thermal quantum system is physically reduced, with compensating entropy arising in conjugate observables (Kastner et al., 2023). This underscores that the word “entropy” spans distinct technical objects across ML, statistics, OT, coding, and physics. Entropy-based rejection is therefore best understood as a family resemblance: the accept/reject decision is mediated by an entropy-like quantity, but the semantics of that quantity depend entirely on the surrounding theory.