--- title: Shannon Entropy Estimation url: https://www.emergentmind.com/topics/shannon-entropy-estimation type: topic --- # Shannon Entropy Estimation Shannon entropy estimation is the statistical problem of inferring the quantity $$ H(P)=-\sum_i p_i \log p_i $$ for a discrete distribution, or its differential analogue $$ H(f)=-\int f(x)\log f(x)\,dx, $$ from finite observations when the underlying law is unknown. The problem arises in the statistical analysis of data stemming from dynamical systems, time series, symbolic natural sequences, linguistic distributions, data streams, Quantitative Information Flow, ecology, genetics, and mixed discrete–continuous models [2105.12829] [1805.08929] [2206.00921] [2602.08347] [1804.08741]. Across these settings, the central issues are bias, variance, sample complexity, undersampling, support uncertainty, and the effect of structural assumptions such as memorylessness, Zipfian tails, graphical-model factorization, or oracle access. The literature therefore spans plug-in estimators, approximation-theoretic minimax estimators, nearest-neighbour and kernel methods, Bayesian nonparametrics, private and distributed protocols, and quantum algorithms [1502.00326] [1801.02050] [1302.0328] [2305.07751] [1710.06025]. ## 1. Definitions, models, and core statistical difficulties For a discrete random variable with probability mass function \(P=\{p_1,\dots,p_M\}\), Shannon entropy is $$ H(P)=-\sum_{i=1}^M p_i\log p_i. $$ Given \(N\) i.i.d. samples, the empirical distribution is \(\hat p_i=n_i/N\), and the most widespread estimator is the plug-in, or empirical, estimator $$ \hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i $$ [2105.12829] [1805.08929] [1502.00326]. In the multinomial case, the plug-in estimator is consistent, and for large \(n\) it admits a \(1/n\) expansion for both bias and variance [2105.12829]. For continuous random vectors \(X\in\mathbb R^d\) with density \(f\), the differential entropy is $$ H(f)=\int_{\mathbb R^d}(-\log f(x))f(x)\,dx =E[-\log f(X)], $$ and estimation proceeds nonparametrically from i.i.d. observations without assuming a parametric form for \(f\) [1801.02050]. Conditional entropy also appears in mixed discrete–continuous models: if \(Y\) is finite-valued and \(X\in\mathbb R^d\), then $$ H(Y|X)=-E\log f_{Y|X}(Y|X) $$ [1804.08741]. A persistent obstacle is undersampling. When the support size is large relative to the sample size, many symbols remain unseen and the plug-in estimator can severely underestimate entropy [2602.08347]. In large-alphabet discrete estimation, the empirical estimator is consistent only if $$ n\gg S, $$ whereas minimax-rate optimal procedures are consistent as soon as $$ n\gtrsim \frac{S}{\ln S} $$ [1502.00326]. Linguistic data make this especially acute because they are heavy-tailed, Zipfian, and often have a very large or unknown support size \(K\), so plug-in estimation misses the contribution of rare and unseen items [2204.01469]. The same broad difficulty reappears in specialized forms. In data streams, entropy is often approximated through generalized entropies near \(\alpha=1\), because direct maintenance of the full histogram may be infeasible [0910.1495]. In programming-language security, low-entropy regimes are particularly important, and standard sampling-and-evaluation models incur a \(1/H\) dependence that becomes unacceptable as \(H\to 0\) [2206.00921]. This suggests that Shannon entropy estimation is less a single estimator than a family of inference problems whose tractability depends sharply on sampling model, distribution class, and structural prior information. ## 2. Classical discrete estimation: plug-in, bias, variance, and minimax refinements In finite-support multinomial models, the plug-in estimator remains the baseline because of its simplicity and consistency, but its statistical behavior has been analyzed in considerable detail. A classical result states that, provided the variance parameter $$ \Lambda_0(p)=\sum_{i=1}^k p_i(\ln p_i)^2-\left(\sum_i p_i\ln p_i\right)^2 $$ is nonzero, $$ \mathrm{Var}[\hat H_n]=\Lambda_0(p)/n+O(n^{-3/2}), $$ with the equivalent form $$ \Lambda_0=\sum_{i=1}^k p_i[H(p)+\ln p_i]^2. $$ Hence \(\Lambda_0\ge 0\), with equality if and only if \(p\) is uniform; in that case the leading term of \(\mathrm{Var}[\hat H_n]\) is actually \(O(n^{-2})\) [2105.12829]. To leading order, $$ \mathrm{std}\,\hat H_n \simeq \sqrt{\Lambda_0(p)/n}. $$ The same line of work determines the distribution that maximizes \(\Lambda_0\) over the \(k\)-simplex. By symmetry, the maximizer has one outlier symbol of probability \(p_0\) and \(k-1\) equal probabilities \(q_0=(1-p_0)/(k-1)\). Introducing \(v=2p_0-1\), the stationarity condition becomes $$ f(v)\equiv v\ln\frac{1+v}{1-v}=2-v\ln(k-1). $$ If \(\tilde v\in(0,1)\) is the unique positive solution, then the maximizing distribution satisfies $$ p_i^*= \begin{cases} (1+\tilde v)/2 & \text{for the outlier index},\\ (1-\tilde v)/[2(k-1)] & \text{otherwise}, \end{cases} $$ and $$ \Lambda_{0,\max}=1/\tilde v^2-1. $$ For large \(k\), $$ \tilde v\simeq 2/\ln k \quad\Rightarrow\quad \Lambda_{0,\max}\simeq (\ln^2 k)/4 $$ [2105.12829]. These results yield explicit uncertainty bounds: $$ \mathrm{Var}[\hat H_n]\le \Lambda_{0,\max}(k)/n, \qquad \mathrm{std}\,\hat H_n\le \sqrt{\Lambda_{0,\max}(k)/n}\simeq \frac{\ln k}{2\sqrt n} $$ for large \(k\) [2105.12829]. If the alphabet size \(k\) is known but the true \(p\) is unknown, this gives a worst-case error bar under the memoryless assumption. A practical refinement is to estimate $$ \widehat\Lambda_0=\sum_{i=1}^k \hat p_i(\ln \hat p_i)^2-\left(\sum_{i=1}^k \hat p_i\ln \hat p_i\right)^2, $$ so that, to leading order, $$ \mathrm{Var}[\hat H_n]\simeq \widehat\Lambda_0/n $$ [2105.12829]. Bias correction and minimax optimality constitute a second major line of development. In large-alphabet estimation over \(\mathcal M_S\), the minimax \(L_2\) risk obeys $$ R^*(n,S)\asymp \frac{S^2}{(n\ln n)^2}+\frac{(\ln S)^2}{n}, \qquad n\gtrsim \frac{S}{\ln S}, $$ and no estimator can improve the worst-case MSE order [1502.00326]. The empirical estimator has a corresponding worst-case rate $$ \frac{S^2}{n^2}+\frac{(\ln S)^2}{n}, $$ which explains why replacing the plug-in by approximation-theoretic estimators produces the effective sample-size enlargement phenomenon: the minimax-rate estimator with \(n\) samples performs like the MLE with \(n\ln n\) samples [1502.00326]. The rate-optimal estimator uses Poissonization, sample splitting, best polynomial approximation of \(-x\ln x\) on the small-\(p\) regime, and a bias-corrected empirical plug-in on the large-\(p\) regime [1502.00326]. The same estimator is adaptive over entropy-bounded subclasses $$ \mathcal M_S(H)=\{P\in\mathcal M_S:H(P)\le H\}, $$ without knowing \(S\) or \(H\) in advance [1502.00326]. A plausible implication is that “entropy estimation” in the discrete finite-support setting is now better viewed as an approximation-theoretic functional-estimation problem rather than merely a corrected histogram calculation. ## 3. Undersampling, sample complexity, and heavy-tailed symbolic data Several works address the question of how many samples are needed before entropy estimates become reliable. For ranked symbolic natural events modeled by a Zipf–Mandelbrot law $$ p(r)=\frac{1}{Z}(r+q)^{-s}, \qquad r=1,\dots,M, $$ with strictly decreasing probabilities, one can combine the Dvoretzky–Kiefer–Wolfowitz inequality $$ P\{\sup_x |\hat F_N(x)-F(x)|>\epsilon\}\le 2\exp(-2N\epsilon^2) $$ with the minimal tail gap $$ \Delta_0=p(M-1)-p(M) $$ to derive the effective-sample requirement $$ n_0=\frac{2}{\Delta_0^2}\ln\frac{2}{\delta}, $$ and then the total sample size $$ N_0\approx n_0/p(M) $$ [1805.08929]. Expressed in terms of \(M,s,q,\delta\), this gives an explicit lower bound on the required number of observations to resolve the rarest events that dominate the entropy sum [1805.08929]. A different small-sample strategy for natural sequences assumes a modified Zipf–Mandelbrot–Li rank-frequency model and uses rank-based coincidence counting. For rank \(r\), the waiting time \(N_r\) until the next occurrence is geometric with $$ P[N_r=n]=(1-p_r)^{n-1}p_r, \qquad E[N_r]=1/p_r. $$ The method measures the average waiting time \(D_r\), numerically inverts a precomputed model curve \(D_r(M)\), estimates the alphabet size \(\hat M\), reconstructs a fitted distribution $$ \hat P(r)=\frac{(r+\hat\beta)^{-\hat\alpha}}{\hat Z}, $$ and then computes $$ \hat H=-\sum_{r=1}^{\hat M}\hat P(r)\log_2 \hat P(r) $$ [1805.06630]. In the reported synthetic \(M=27\) setting, the true entropy is approximately \(4.261\) bits, and using only \(25\) waiting-time observations for the rank-1 symbol the method produced \(\hat H\approx 4.266\) bits [1805.06630]. For linguistic distributions, direct empirical comparisons of seven estimators show systematic differences in bias and MSE. The plug-in estimator is biased downward by roughly \((K-1)/(2N)\), Miller–Madow reduces the negative bias to \(O(1/N^2)\), Chao–Shen is empirically strongest in the lowest-data regime, and NSB becomes clearly best once \(N\) exceeds a few times \(K\) [2204.01469]. On natural unigrams, NSB and Chao–Shen reduce root-MSE by \(50\%\)–\(80\%\) versus MLE for \(N\) from \(10^2\) up to \(10^6\) [2204.01469]. The same study reports that re-estimation with NSB in two linguistic replication studies reduced reported mutual-information effect sizes while preserving the substantive conclusions [2204.01469]. More recent work on small-data discrete estimation uses entropy decomposability. If the sample space is partitioned into disjoint subsets \(S_1,\dots,S_K\), then $$ H(P)=H(P_S)+\sum_{k=1}^K P_{S_k}H(P|S_k). $$ Using the partition \(S_1=\{n_i=0\}\), \(S_2=\{1\le n_i\le \lambda\}\), \(S_3=\{n_i>\lambda\}\), together with a missing-mass estimator \(\hat m_0\), unseen-outcome estimation, and a Miller–Madow correction on \(S_3\), the resulting estimator matches Chao–Shen and the Valiant–Valiant LP method in RMSE and bias in undersampled regimes, and becomes consistent as \(N\) grows [2512.10133]. These results converge on a common point: rare symbols and unseen mass dominate the finite-sample pathology of entropy estimation. This suggests that the decisive modelling choice is often not the entropy functional itself, but the surrogate used for the unobserved tail. ## 4. Continuous, differential, and conditional entropy estimation For differential entropy, nearest-neighbour, kernel, and spacing estimators are the dominant nonparametric classes. The Kozachenko–Leonenko \(1\)-NN estimator for \(X_1,\dots,X_n\in\mathbb R^d\) is $$ \hat H_n=d\log \bar p+\log V_d+\gamma+\log(n-1), $$ where \(p_i\) is the nearest-neighbour distance from \(X_i\), \(\bar p=(p_1\cdots p_n)^{1/n}\), \(V_d=\pi^{d/2}/\Gamma(1+d/2)\), and \(\gamma\approx 0.5772\) [1801.02050]. Under conditions formulated through Hardy–Littlewood-type functionals \(K_f,Q_f,T_f\), the estimator is asymptotically unbiased, and under stronger assumptions it is \(L^2\)-consistent [1801.02050]. In particular, the results apply to any nondegenerate Gaussian vector [1801.02050]. Kernel plug-in estimation has also been analyzed under dependence. For the one-sided linear process $$ X_n=\sum_{i=0}^\infty a_i\varepsilon_{n-i} $$ with absolutely summable coefficients, the kernel density estimator $$ f_n(x)=\frac{1}{nh_n}\sum_{i=1}^n K\!\left(\frac{x-X_i}{h_n}\right) $$ feeds the integral entropy estimator $$ S_n(f)=-\int_{A_n} f_n(x)\log f_n(x)\,dx, \qquad A_n=\{x:f_n(x)\ge \gamma_n\}. $$ Under the stated regularity and bandwidth assumptions, \(S_n(f)\to S(f)\) almost surely and in \(L^2\) [2009.03472]. The uniform density-estimation error has the order $$ \sup_x |f_n(x)-f(x)|=O\!\left(\frac{\log n}{nh_n}+h_n^2\right) $$ almost surely, and balancing the two terms yields $$ h_n\asymp (n^{-1}\log n)^{1/3} $$ [2009.03472]. An extensive comparative review of \(27\) differential-entropy estimators selected from spacings, KDE, and kNN classes reports that spacings-based estimators generally performed better than the estimators from the other two classes at univariate level, but suffered from non existence at multivariate level [2406.19432]. kNN estimators were generally inferior to the other two classes considered but showed an advantage of existence for all dimensions [2406.19432]. For \(d>1\), the KDE plug-in with normal-reference bandwidth is recommended for small-to-moderate sample sizes, while kNN becomes a fallback that remains defined in all dimensions [2406.19432]. Conditional entropy in mixed discrete–continuous models is treated by a dedicated k-nearest-neighbour estimator. For i.i.d. copies of \((X,Y)\in\mathbb R^d\times M\), with \(Y\) finite-valued, let \(\rho_{n,k,i}\) be the distance from \(X_i\) to its \(k\)-th nearest neighbour and $$ E_{n,k,i}=\#\{j\ne i:Y_j=Y_i,\ \|X_j-X_i\|\le \rho_{n,k,i}\}. $$ Then $$ \hat H_{n,k}(Y|X)=\frac1n\sum_{i=1}^n \left[-\log\left(\frac{E_{n,k,i}+1}{k}\right)\right]. $$ Under positivity, constriction, and moment assumptions, the estimator is asymptotically unbiased and \(L^2\)-consistent; the Gaussian conditional case is covered as a corollary [1804.08741]. The same estimator can be used for feature selection through $$ \widehat I(Y;X^{(j)})=\widehat H(Y)-\hat H(Y|X^{(j)}), $$ with applications to medical and biological investigations [1804.08741]. Histogram-based estimation forms a bridge between discrete and continuous viewpoints. If \(N\) observations are binned into \(m\) fixed-width bins, with \(p_i=n_i/N\), then the histogram entropy is $$ H_B=-\sum_{i=1}^m p_i\log_2 p_i. $$ Watts and Crow relate this to differential entropy through $$ H_B\approx h-\log_2 A, $$ where \(A\) is bin width, and propose the entropy-matched choice $$ A=\frac{2^{\hat h}}{N^{1/M}} $$ with \(2\le M\le 3\) controlling over- and under-binning [2210.02848]. The same work defines $$ M_B=\frac{\log_2 N}{H_B} $$ and interprets \(M_B<2\) as over-binning and \(M_B>3\) as under-binning [2210.02848]. ## 5. Bayesian nonparametrics and sparse-support uncertainty Bayesian methods target the regime in which the support size is unknown, large, or countably infinite. A foundational approach uses the Pitman–Yor process \(\mathrm{PY}(d,\theta,H_0)\), with stick-breaking weights $$ V_k\sim \mathrm{Beta}(1-d,\theta+kd), \qquad \pi_k=V_k\prod_{j1\), the true entropy is finite, and the resulting estimator is consistent in probability as \(N\to\infty\) [2602.08347]. An earlier Bayesian treatment also uses Dirichlet and Pitman–Yor priors, but emphasizes analytical posterior moments and prior geometry. For a finite symmetric Dirichlet prior, the posterior mean satisfies $$ E[H|\mathbf n,a] =\psi_0(\tilde N+1)-\sum_{i=1}^S \frac{\tilde n_i}{\tilde N}\psi_0(\tilde n_i+1), $$ and analogous closed forms exist for posterior variance [1302.0328]. Under a PYP prior \(\mathrm{PY}(d,\alpha)\), the posterior mean is $$ E[H|\mathbf n,d,\alpha] = \psi_0(\alpha+N+1) -\frac{\alpha+Kd}{\alpha+N}\psi_0(1-d) -\frac1{\alpha+N}\sum_{i=1}^K (n_i-d)\psi_0(n_i-d+1) $$ [1302.0328]. The same work shows that a fixed Dirichlet or Pitman–Yor prior implies a narrow prior distribution over \(H\), so the prior strongly determines the entropy estimate in the undersampled regime; to mitigate this, it introduces a Pitman–Yor mixture prior designed to be approximately flat over \(H\) [1302.0328]. A related hierarchical-Bayesian approach uses a symmetric Dirichlet prior with hyperparameter \(\beta\), estimates \(\beta^\*\) by empirical Bayes through the digamma equation $$ \sum_{k=1}^K [\psi_0(n_k+\beta^\*)-\psi_0(\beta^\*)] = K[\psi_0(N+K\beta^\*)-\psi_0(K\beta^\*)], $$ and then computes $$ E[H|n,\beta] = \psi_0(N+K\beta+1) -\sum_{k=1}^K \frac{n_k+\beta}{N+K\beta}\psi_0(n_k+\beta+1) $$ [2301.13647]. The reported computational complexity is \(O(I\cdot K)\), where \(I\) is the number of iterations in the one-dimensional root finding [2301.13647]. The common Bayesian theme is that unseen support must be treated as a posterior object rather than ignored. A plausible implication is that support uncertainty and entropy uncertainty cannot be cleanly separated in sparse regimes. ## 6. Structured data, alternative access models, and computational complexity Entropy estimation changes qualitatively when the observation model departs from simple i.i.d. multinomial sampling. For memoryless \(k\)-symbol sources, worst-case plug-in uncertainty can be bounded through \(\Lambda_0\), but for systems with memory, block entropies become central [2105.12829] [2205.11931]. If \(H_n\) denotes the entropy of overlapping blocks of length \(n\), then for an \(m\)-th order homogeneous Markov chain, $$ H_n=(H_{m+1}-H_m)(n-m)+H_m, \qquad n\ge m, $$ so \(H_n\) is affine-linear in \(n\) beyond the memory depth [2205.11931]. The proposed correlation-coverage estimator replaces the Chao–Shen coverage correction by a sequential coverage estimate \(\hat C_n\) tailored to correlated blocks and then computes $$ \hat H_n = -\sum_{b_i\in S} \frac{\tilde p_i\log \tilde p_i}{1-(1-\tilde p_i)^{N_n}}, \qquad \tilde p_i=\hat C_n\hat p_i. $$ In binary Markov tests, the aggregated MSE of Chao–Shen was approximately \(4.65\) while the CC estimator gave approximately \(0.90\) [2205.11931]. In data streams, one-pass entropy estimation is achieved by estimating frequency moments through Compressed Counting. With $$ F_{(\alpha)}=\sum_{i=1}^D A_t[i]^\alpha, $$ one forms the Rényi entropy $$ H_\alpha=\frac{1}{1-\alpha}\log\left(\sum_i p_i^\alpha\right) $$ near \(\alpha=1\). Compressed Counting uses maximally-skewed stable projections to obtain an estimator of \(F_{(\alpha)}\), and then the optimal-quantile estimator minimizes the leading asymptotic variance constant for the scale estimate [0910.1495]. The same work proves that Rényi entropy is better than Tsallis entropy for approximating Shannon entropy, in the sense that $$ |\Delta_R(\alpha)|\le |\Delta_T(\alpha)| $$ for all \(\alpha\), where \(\Delta_R(\alpha)=H_\alpha-H\) and \(\Delta_T(\alpha)=T_\alpha-H\) [0910.1495]. In privacy-preserving distributed estimation, each user holds exactly one sample and communicates \(O(1)\) bits under \(\epsilon\)-local differential privacy. If a \(d\)-variate discrete distribution factorizes as a tree graphical model with constant marginal support size \(c\), then Shannon entropy can be estimated to additive error \(\alpha d\) using $$ \tilde O\!\left(\frac{d}{\epsilon^2\alpha^5}\log\frac d\delta\right) $$ users, with total communication \(O(1)\) bits per user [2305.07751]. The algorithm uses the Chow–Liu identity $$ H(X)=\sum_{i=1}^d H(X_i)-w(P), $$ where \(w(P)\) is the maximum spanning tree weight under mutual-information edge weights [2305.07751]. The conditional-sampling model yields another departure from standard sample complexity. In the probability-revealing conditional-sampling model, the oracle \(\mathrm{COND}^\*\) returns a conditional sample together with its true probability. The estimator ENTROPY-EST is a median-of-means estimator built from the self-information variable \(g(x)=\log(1/P(x))\), detects any heavy atom with probability \(>1/2\), and obtains a multiplicative \((1+\epsilon)\)-approximation using $$ O\!\left(\frac{m}{\epsilon^2}\log\frac1\delta\right) $$ queries on a domain of size \(2^m\) [2206.00921]. This improves on the previously cited additive-\(\epsilon\) conditional-sampling complexity $$ O(m^7/\epsilon^8\cdot \log(1/\delta)) $$ and avoids the \(1/H\) dependence that is unavoidable under SAMP+EVAL access [2206.00921]. Quantum access further changes the complexity landscape. In the Bravyi–Harrow–Hassidim oracle model, the first quantum algorithm for additive-\(\epsilon\) Shannon entropy estimation uses amplitude estimation and quantum Monte Carlo to achieve $$ \tilde O(\sqrt n/\epsilon^2) $$ queries, compared with classical complexity \(\Theta(n/\log n)\) for constant additive error [1710.06025]. A later quantum probability-oracle result improves the upper bound to $$ \tilde\Theta(\sqrt n/\epsilon), $$ up to logarithmic factors, and states a matching lower bound up to logarithmic factors [2509.07452]. This suggests that the entropy-estimation problem is unusually sensitive to the information-access primitive: the same functional can move from large-alphabet minimax estimation to near-optimal quantum singular-value transformation depending on what queries are allowed. ## 7. Practical estimation regimes and recurrent methodological themes The literature supports a regime-dependent view of Shannon entropy estimation. | Setting | Representative approach | Reported property | |---|---|---| | Multinomial, known \(k\) | Plug-in with \(\Lambda_0\) analysis | Worst-case \(\mathrm{std}\,\hat H_n\le (\ln k)/(2\sqrt n)+o(1/\sqrt n)\) [2105.12829] | | Large alphabet discrete | Approximation-theoretic minimax estimator | Consistent for \(n\gtrsim S/\ln S\) [1502.00326] | | Very sparse linguistic data | Chao–Shen | Lowest MSE in the lowest-data regime [2204.01469] | | Moderate \(N\) relative to \(K\) | NSB | Clearly best once \(N\) exceeds a few times \(K\) [2204.01469] | | Unknown or infinite support | Pitman–Yor-based Bayes estimators | Stable when observed species are fewer than true species [2602.08347] | | Differential entropy, \(d=1\) | Spacings estimators | Generally performed better at univariate level [2406.19432] | | Differential entropy, \(d>1\) | KDE or kNN | KDE preferable at small \(n\); kNN exists for all \(d\) [2406.19432] | A recurring misconception is that plug-in estimation is generically adequate once it is unbiased to first order. The cited works indicate otherwise. In discrete large-alphabet problems, the decisive failure mode is often unseen mass rather than first-order Taylor bias [1502.00326] [2204.01469] [2602.08347]. In continuous problems, consistency may require nontrivial conditions on local averages, tails, and bandwidth sequences [1801.02050] [2009.03472]. In structured-sequence settings, correlation can invalidate coverage corrections designed for independence [2205.11931]. In private, streaming, conditional-sampling, and quantum settings, computational or communication constraints become part of the statistical model itself [0910.1495] [2305.07751] [2206.00921] [1710.06025]. Another recurrent theme is the decomposition of total error into bias, variance, and modelling mismatch. The variance parameter \(\Lambda_0\) isolates leading uncertainty for plug-in multinomial estimation [2105.12829]. Approximation-theoretic estimators trade polynomial-approximation bias against variance to achieve minimax rates [1502.00326]. Rényi-based data-stream methods explicitly optimize the variance–bias trade-off in choosing \(\alpha\) near \(1\) [0910.1495]. Bayesian nonparametrics relocate the dominant uncertainty into the posterior over unseen support [1302.0328] [2602.08347]. Taken together, these works portray Shannon entropy estimation as a technically heterogeneous domain unified by a single functional but fragmented by sampling regime. The strongest general conclusion is that no estimator is uniformly preferred across all settings: the appropriate method depends on whether the principal difficulty is large support, sparse counts, continuous geometry, dependence, privacy, streaming constraints, or restricted oracle access. Source: https://www.emergentmind.com/topics/shannon-entropy-estimation