SENECA: Small-Sample Discrete Entropy Estimation via Self-Consistent Missing Mass

Abstract: Discrete entropy estimation is a classic information theory problem, wherein the average information content of a discrete random variable is estimated from samples alone. Naive approaches, such as the plugin method, fail to account for the probability mass associated with members of the random variable's support that are unobserved in a given sample, known as the "missing mass." The resulting systemic underestimation is particularly problematic when data is time-consuming or costly to gather. We propose SENECA, an entropy estimation scheme based on a novel ``self-consistent'' missing mass calculation. Extensive numerical experiments indicate that our approach outperforms many state-of-the-art alternatives overall in the small-sample setting. We then apply SENECA to two practical use cases, namely biodiversity estimation and the detection of incorrect LLM responses, where our method is competitive with domain-specific approaches. Our work advances SENECA as an effective drop-in replacement for small-sample entropy estimation, with broad utility across several domains.

• The paper introduces SENECA, a fixed-point based method that improves small-sample discrete entropy estimation by correcting missing mass bias.
• It employs shrinkage and Horvitz-Thompson style adjustments to balance bias and variance, outperforming existing estimators in simulations and real-world tests.
• Its applicability spans biodiversity quantification and LLM uncertainty, demonstrating robust accuracy and practical utility in under-sampled regimes.

SENECA: Self-Consistent Missing Mass for Small-Sample Discrete Entropy Estimation

Introduction and Motivation

Accurate entropy estimation for discrete distributions is a core problem in information theory, underpinning applications in machine learning, ecology, and NLP. Traditional approaches, such as the plugin estimator, substantially underestimate entropy in the low-sample regime due to their failure to account for unobserved elements—termed the "missing mass" problem. SENECA ("Small-Sample Discrete Entropy Estimation via Self-Consistent Missing Mass") addresses this challenge by proposing a novel estimator that couples isotropic shrinkage with a fixed-point missing mass projection, integrating recent advances in support size estimation. This methodological innovation yields robust entropy estimates in regimes where the number of samples $n$ is much lower than the unknown support size $|\mathcal{X}|$—a critical need in costly or time-intensive data gathering contexts.

Figure 1: Illustration of the missing mass challenge in plugin entropy estimation on a Zipf distribution, and the role of SENECA’s fixed-point iteration in accurately estimating the expected missing mass.

Methodological Framework

Decomposing and Estimating Missing Mass

SENECA begins by decomposing the expected missing mass into observed and unobserved components with respect to the sample $X^n$. The estimator introduces isotropic shrinkage: empirical frequencies for observed labels are scaled by a factor $\alpha$ to preserve total probability after accounting for missing mass. For unobserved labels (the support gap $|U|$), SENECA apportions missing mass equally when $|U|$ is known or estimated, leveraging frequency of frequencies (FoF) statistics.

Given the impossibility of unbiased entropy estimation for discrete distributions in general and the infeasibility of observing the full support for large or heavy-tailed distributions, SENECA proposes a self-consistency fixed-point equation for missing mass $m$:

$m^* = \mu(m^*, \upsilon, X^n)$

where $\upsilon$ is the estimated number of unobserved symbols and $\mu$ is derived from observed label statistics paired with a support estimator.

This fixed-point is solved with Steffensen's method, yielding an estimator $|\mathcal{X}|$0 that robustly interpolates missing mass both in the well-sampled and highly under-sampled regimes.

Support Size Estimation and Sensitivity

A critical innovation is SENECA's modularity regarding the choice of support estimator. While the regularized weighted Chebyshev (RWC) estimator is used by default, ablations demonstrate the algorithm's performance is largely insensitive to the precise support estimator used, as the "unobserved" mass proportion remains bounded in practical settings.

Figure 2: SENECA’s performance across different support size estimation strategies is interchangeable, confirming robustness to this component.

Empirical Evaluation

Synthetic Small-Sample Regimes

Extensive simulations were performed on 72 distributional settings, systematically varying support size, prevalence distribution (including uniform, step, Zipf, Dirichlet, and beta-binomial), and sampling regime. SENECA is compared to seven baselines: Plugin, Grassberger, James-Stein, Bonachela, Chao-Shen, Chao-Wang-Jost, and Valiant-Valiant estimators.

SENECA achieves the lowest or second-lowest RMSE in nearly all under-sampled settings ($|\mathcal{X}|$1), outperforming others as the prevalence distribution becomes heavy-tailed or the unobserved mass dominates. In well-sampled regimes, SENECA remains competitive but is occasionally surpassed by estimators like James-Stein or Chao-Shen which are tuned for high sample fractions.

Figure 3: RMSE of entropy estimation in the small-sample regime for diverse prevalence families; SENECA consistently outperforms baselines as sample coverage decreases.

Figure 4: Empirical bias-variance decomposition shows SENECA’s balanced trade-off, maintaining low total error even as sampling becomes sparse.

Real-World Applications

Biodiversity Quantification

SENECA is applied to 58 real insect species-prevalence datasets, a canonical testbed for entropy estimators due to heavy-tailed distributions and limited sampling opportunities. Borda count aggregation of RMSE ranks across populations and sample sizes demonstrates SENECA’s superior reliability and generality compared to both classical and modern entropy estimators.

Figure 5: Borda count summary for biodiversity entropy estimation across tropical insect collections; SENECA outranks competitors for robustness across sample sizes.

LLM Uncertainty Quantification

The method is further validated in the context of LLM predictive uncertainty, where discrete entropy over semantic clusters of generated responses serves as a proxy for epistemic and aleatoric uncertainty. Across diverse biomedical, reasoning, and general QA datasets, SENECA and its variant SENECA-M obtain the second- and third-highest aggregated AUROCs in detecting incorrect responses, surpassed only by a domain-specific semantic support estimator ($|\mathcal{X}|$2).

Figure 6: AUROC performance for LLM uncertainty quantification on QA tasks; SENECA is competitive with, or superior to, specialized entropy-based baselines.

Analysis and Implications

The study makes several strong, empirically supported claims:

• No existing general-purpose estimator matches SENECA in small-sample, under-sampled regimes. Numerical evidence across 72 synthetic and 58 real distributions supports this.
• SENECA’s performance is robust to the choice of support size estimator, reducing sensitivity to an otherwise challenging component in the entropy estimation pipeline.
• SENECA enhances black-box LLM uncertainty quantification, providing higher AUROC than alternative discrete entropy estimates of semantic variability.

Theoretically, SENECA’s self-consistent missing mass estimation advances the design space for information-theoretic property estimation; it offers a new bias-variance trade-off that lifts performance where SBM asymptotics, classical shrinkage, or singleton-based corrections fail.

Practically, these results extend to biodiversity estimation, rare-event modeling, NLP uncertainty metrics, decision tree construction, and mutual information estimation in computational biology. SENECA can serve as a drop-in replacement without hyperparameter tuning or model-specific calibration.

Limitations and Future Directions

Despite its empirical strengths, SENECA does not achieve unbiasedness—the non-existence thereof is known in the literature. Moreover, some settings (e.g., well-sampled, low-variance distributions) may still favor problem-specific estimators. The provision of theoretical convergence rates, large-sample behavior, and joint estimation protocols for associated metrics (e.g., mutual information, $|\mathcal{X}|$3-divergences) is identified as future work.

Further, SENECA’s structure suggests avenues for extension to hierarchical and structured support sizes, adaptive prior incorporation, or semi-parametric mixture models.

Conclusion

SENECA introduces a principled, robust, and practically effective solution to small-sample entropy estimation for discrete distributions. Its design navigates the statistical limitations posed by unseen support and missing mass, enabling improved estimation accuracy across theoretical and real-world domains. The framework’s modularity, empirical reliability, and domain transferability suggest its adoption as a standard baseline in small-sample information-theoretic estimation tasks.