Simple, Efficient Entropy Estimation using Harmonic Numbers (2505.20153v3)

Abstract: The estimation of entropy, a fundamental measure of uncertainty, is central to diverse data applications. For discrete random variables, however, efficient entropy estimation presents challenges, particularly when the cardinality of the support set is large relative to the available sample size. This is because, without other assumptions, there may be insufficient data to adequately characterize a probability mass function. Further complications stem from the dependence among transformations of empirical frequencies within the sample. This paper demonstrates that a simple entropy estimator based on the harmonic number function achieves asymptotic efficiency on discrete random variables with tail probabilities satisfying $p_j =o(j^{-2})$ as $j\rightarrow\infty$. This result renders statistical inference newly feasible for all but very heavy-tailed probability mass functions. Moreover, its strong mean squared error bounds coupled with simple implementation make this estimator an attractive replacement over others in application.