Shannon Entropy Estimation

• Shannon entropy estimation is a measure of uncertainty computed from probability distributions, crucial for applications in data analysis across various fields.
• Techniques range from plug-in estimators and approximation-theoretic minimax methods to Bayesian nonparametric and nonparametric density approaches for continuous cases.
• Practical insights involve addressing bias, variance, and sample complexity by using bias correction, adaptive estimation, and specialized methods for undersampled or heavy-tailed regimes.

Shannon entropy estimation is the statistical problem of inferring the quantity

$H(P)=-\sum_i p_i \log p_i$

for a discrete distribution, or its differential analogue

$H(f)=-\int f(x)\log f(x)\,dx,$

from finite observations when the underlying law is unknown. The problem arises in the statistical analysis of data stemming from dynamical systems, time series, symbolic natural sequences, linguistic distributions, data streams, Quantitative Information Flow, ecology, genetics, and mixed discrete–continuous models (Ricci et al., 2021, Back et al., 2018, Golia et al., 2022, Hashino et al., 9 Feb 2026, Bulinski et al., 2018). Across these settings, the central issues are bias, variance, sample complexity, undersampling, support uncertainty, and the effect of structural assumptions such as memorylessness, Zipfian tails, graphical-model factorization, or oracle access. The literature therefore spans plug-in estimators, approximation-theoretic minimax estimators, nearest-neighbour and kernel methods, Bayesian nonparametrics, private and distributed protocols, and quantum algorithms (Han et al., 2015, Bulinski et al., 2018, Archer et al., 2013, Bravo-Hermsdorff et al., 2023, Li et al., 2017).

1. Definitions, models, and core statistical difficulties

For a discrete random variable with probability mass function $P=\{p_1,\dots,p_M\}$, Shannon entropy is

$H(P)=-\sum_{i=1}^M p_i\log p_i.$

Given $N$ i.i.d. samples, the empirical distribution is $\hat p_i=n_i/N$, and the most widespread estimator is the plug-in, or empirical, estimator

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$

(Ricci et al., 2021, Back et al., 2018, Han et al., 2015). In the multinomial case, the plug-in estimator is consistent, and for large $n$ it admits a $1/n$ expansion for both bias and variance (Ricci et al., 2021).

For continuous random vectors $X\in\mathbb R^d$ with density $H(f)=-\int f(x)\log f(x)\,dx,$0, the differential entropy is

$H(f)=-\int f(x)\log f(x)\,dx,$1

and estimation proceeds nonparametrically from i.i.d. observations without assuming a parametric form for $H(f)=-\int f(x)\log f(x)\,dx,$2 (Bulinski et al., 2018). Conditional entropy also appears in mixed discrete–continuous models: if $H(f)=-\int f(x)\log f(x)\,dx,$3 is finite-valued and $H(f)=-\int f(x)\log f(x)\,dx,$4, then

$H(f)=-\int f(x)\log f(x)\,dx,$5

(Bulinski et al., 2018).

A persistent obstacle is undersampling. When the support size is large relative to the sample size, many symbols remain unseen and the plug-in estimator can severely underestimate entropy (Hashino et al., 9 Feb 2026). In large-alphabet discrete estimation, the empirical estimator is consistent only if

$H(f)=-\int f(x)\log f(x)\,dx,$6

whereas minimax-rate optimal procedures are consistent as soon as

$H(f)=-\int f(x)\log f(x)\,dx,$7

(Han et al., 2015). Linguistic data make this especially acute because they are heavy-tailed, Zipfian, and often have a very large or unknown support size $H(f)=-\int f(x)\log f(x)\,dx,$8, so plug-in estimation misses the contribution of rare and unseen items (Arora et al., 2022).

The same broad difficulty reappears in specialized forms. In data streams, entropy is often approximated through generalized entropies near $H(f)=-\int f(x)\log f(x)\,dx,$9, because direct maintenance of the full histogram may be infeasible (0910.1495). In programming-language security, low-entropy regimes are particularly important, and standard sampling-and-evaluation models incur a $P=\{p_1,\dots,p_M\}$0 dependence that becomes unacceptable as $P=\{p_1,\dots,p_M\}$1 (Golia et al., 2022). This suggests that Shannon entropy estimation is less a single estimator than a family of inference problems whose tractability depends sharply on sampling model, distribution class, and structural prior information.

2. Classical discrete estimation: plug-in, bias, variance, and minimax refinements

In finite-support multinomial models, the plug-in estimator remains the baseline because of its simplicity and consistency, but its statistical behavior has been analyzed in considerable detail. A classical result states that, provided the variance parameter

$P=\{p_1,\dots,p_M\}$2

is nonzero,

$P=\{p_1,\dots,p_M\}$3

with the equivalent form

$P=\{p_1,\dots,p_M\}$4

Hence $P=\{p_1,\dots,p_M\}$5, with equality if and only if $P=\{p_1,\dots,p_M\}$6 is uniform; in that case the leading term of $P=\{p_1,\dots,p_M\}$7 is actually $P=\{p_1,\dots,p_M\}$8 (Ricci et al., 2021). To leading order,

$P=\{p_1,\dots,p_M\}$9

The same line of work determines the distribution that maximizes $H(P)=-\sum_{i=1}^M p_i\log p_i.$0 over the $H(P)=-\sum_{i=1}^M p_i\log p_i.$1-simplex. By symmetry, the maximizer has one outlier symbol of probability $H(P)=-\sum_{i=1}^M p_i\log p_i.$2 and $H(P)=-\sum_{i=1}^M p_i\log p_i.$3 equal probabilities $H(P)=-\sum_{i=1}^M p_i\log p_i.$4. Introducing $H(P)=-\sum_{i=1}^M p_i\log p_i.$5, the stationarity condition becomes

$H(P)=-\sum_{i=1}^M p_i\log p_i.$6

If $H(P)=-\sum_{i=1}^M p_i\log p_i.$7 is the unique positive solution, then the maximizing distribution satisfies

$H(P)=-\sum_{i=1}^M p_i\log p_i.$8

and

$H(P)=-\sum_{i=1}^M p_i\log p_i.$9

For large $N$0,

$N$1

(Ricci et al., 2021).

These results yield explicit uncertainty bounds:

$N$2

for large $N$3 (Ricci et al., 2021). If the alphabet size $N$4 is known but the true $N$5 is unknown, this gives a worst-case error bar under the memoryless assumption. A practical refinement is to estimate

$N$6

so that, to leading order,

$N$7

(Ricci et al., 2021).

Bias correction and minimax optimality constitute a second major line of development. In large-alphabet estimation over $N$8, the minimax $N$9 risk obeys

$\hat p_i=n_i/N$0

and no estimator can improve the worst-case MSE order (Han et al., 2015). The empirical estimator has a corresponding worst-case rate

$\hat p_i=n_i/N$1

which explains why replacing the plug-in by approximation-theoretic estimators produces the effective sample-size enlargement phenomenon: the minimax-rate estimator with $\hat p_i=n_i/N$2 samples performs like the MLE with $\hat p_i=n_i/N$3 samples (Han et al., 2015).

The rate-optimal estimator uses Poissonization, sample splitting, best polynomial approximation of $\hat p_i=n_i/N$4 on the small-$\hat p_i=n_i/N$5 regime, and a bias-corrected empirical plug-in on the large-$\hat p_i=n_i/N$6 regime (Han et al., 2015). The same estimator is adaptive over entropy-bounded subclasses

$\hat p_i=n_i/N$7

without knowing $\hat p_i=n_i/N$8 or $\hat p_i=n_i/N$9 in advance (Han et al., 2015). A plausible implication is that “entropy estimation” in the discrete finite-support setting is now better viewed as an approximation-theoretic functional-estimation problem rather than merely a corrected histogram calculation.

3. Undersampling, sample complexity, and heavy-tailed symbolic data

Several works address the question of how many samples are needed before entropy estimates become reliable. For ranked symbolic natural events modeled by a Zipf–Mandelbrot law

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$0

with strictly decreasing probabilities, one can combine the Dvoretzky–Kiefer–Wolfowitz inequality

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$1

with the minimal tail gap

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$2

to derive the effective-sample requirement

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$3

and then the total sample size

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$4

(Back et al., 2018). Expressed in terms of $\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$5, this gives an explicit lower bound on the required number of observations to resolve the rarest events that dominate the entropy sum (Back et al., 2018).

A different small-sample strategy for natural sequences assumes a modified Zipf–Mandelbrot–Li rank-frequency model and uses rank-based coincidence counting. For rank $\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$6, the waiting time $\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$7 until the next occurrence is geometric with

$\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$8

The method measures the average waiting time $\hat H_N=-\sum_{i=1}^M \hat p_i\log \hat p_i$9, numerically inverts a precomputed model curve $n$0, estimates the alphabet size $n$1, reconstructs a fitted distribution

$n$2

and then computes

$n$3

(Back et al., 2018). In the reported synthetic $n$4 setting, the true entropy is approximately $n$5 bits, and using only $n$6 waiting-time observations for the rank-1 symbol the method produced $n$7 bits (Back et al., 2018).

For linguistic distributions, direct empirical comparisons of seven estimators show systematic differences in bias and MSE. The plug-in estimator is biased downward by roughly $n$8, Miller–Madow reduces the negative bias to $n$9, Chao–Shen is empirically strongest in the lowest-data regime, and NSB becomes clearly best once $1/n$0 exceeds a few times $1/n$1 (Arora et al., 2022). On natural unigrams, NSB and Chao–Shen reduce root-MSE by $1/n$2–$1/n$3 versus MLE for $1/n$4 from $1/n$5 up to $1/n$6 (Arora et al., 2022). The same study reports that re-estimation with NSB in two linguistic replication studies reduced reported mutual-information effect sizes while preserving the substantive conclusions (Arora et al., 2022).

More recent work on small-data discrete estimation uses entropy decomposability. If the sample space is partitioned into disjoint subsets $1/n$7, then

$1/n$8

Using the partition $1/n$9, $X\in\mathbb R^d$0, $X\in\mathbb R^d$1, together with a missing-mass estimator $X\in\mathbb R^d$2, unseen-outcome estimation, and a Miller–Madow correction on $X\in\mathbb R^d$3, the resulting estimator matches Chao–Shen and the Valiant–Valiant LP method in RMSE and bias in undersampled regimes, and becomes consistent as $X\in\mathbb R^d$4 grows (Bastos et al., 10 Dec 2025).

These results converge on a common point: rare symbols and unseen mass dominate the finite-sample pathology of entropy estimation. This suggests that the decisive modelling choice is often not the entropy functional itself, but the surrogate used for the unobserved tail.

4. Continuous, differential, and conditional entropy estimation

For differential entropy, nearest-neighbour, kernel, and spacing estimators are the dominant nonparametric classes. The Kozachenko–Leonenko $X\in\mathbb R^d$5-NN estimator for $X\in\mathbb R^d$6 is

$X\in\mathbb R^d$7

where $X\in\mathbb R^d$8 is the nearest-neighbour distance from $X\in\mathbb R^d$9, $H(f)=-\int f(x)\log f(x)\,dx,$00, $H(f)=-\int f(x)\log f(x)\,dx,$01, and $H(f)=-\int f(x)\log f(x)\,dx,$02 (Bulinski et al., 2018). Under conditions formulated through Hardy–Littlewood-type functionals $H(f)=-\int f(x)\log f(x)\,dx,$03, the estimator is asymptotically unbiased, and under stronger assumptions it is $H(f)=-\int f(x)\log f(x)\,dx,$04-consistent (Bulinski et al., 2018). In particular, the results apply to any nondegenerate Gaussian vector (Bulinski et al., 2018).

Kernel plug-in estimation has also been analyzed under dependence. For the one-sided linear process

$H(f)=-\int f(x)\log f(x)\,dx,$05

with absolutely summable coefficients, the kernel density estimator

$H(f)=-\int f(x)\log f(x)\,dx,$06

feeds the integral entropy estimator

$H(f)=-\int f(x)\log f(x)\,dx,$07

Under the stated regularity and bandwidth assumptions, $H(f)=-\int f(x)\log f(x)\,dx,$08 almost surely and in $H(f)=-\int f(x)\log f(x)\,dx,$09 (Fortune et al., 2020). The uniform density-estimation error has the order

$H(f)=-\int f(x)\log f(x)\,dx,$10

almost surely, and balancing the two terms yields

$H(f)=-\int f(x)\log f(x)\,dx,$11

(Fortune et al., 2020).

An extensive comparative review of $H(f)=-\int f(x)\log f(x)\,dx,$12 differential-entropy estimators selected from spacings, KDE, and kNN classes reports that spacings-based estimators generally performed better than the estimators from the other two classes at univariate level, but suffered from non existence at multivariate level (Madukaife et al., 2024). kNN estimators were generally inferior to the other two classes considered but showed an advantage of existence for all dimensions (Madukaife et al., 2024). For $H(f)=-\int f(x)\log f(x)\,dx,$13, the KDE plug-in with normal-reference bandwidth is recommended for small-to-moderate sample sizes, while kNN becomes a fallback that remains defined in all dimensions (Madukaife et al., 2024).

Conditional entropy in mixed discrete–continuous models is treated by a dedicated k-nearest-neighbour estimator. For i.i.d. copies of $H(f)=-\int f(x)\log f(x)\,dx,$14, with $H(f)=-\int f(x)\log f(x)\,dx,$15 finite-valued, let $H(f)=-\int f(x)\log f(x)\,dx,$16 be the distance from $H(f)=-\int f(x)\log f(x)\,dx,$17 to its $H(f)=-\int f(x)\log f(x)\,dx,$18-th nearest neighbour and

$H(f)=-\int f(x)\log f(x)\,dx,$19

Then

$H(f)=-\int f(x)\log f(x)\,dx,$20

Under positivity, constriction, and moment assumptions, the estimator is asymptotically unbiased and $H(f)=-\int f(x)\log f(x)\,dx,$21-consistent; the Gaussian conditional case is covered as a corollary (Bulinski et al., 2018). The same estimator can be used for feature selection through

$H(f)=-\int f(x)\log f(x)\,dx,$22

with applications to medical and biological investigations (Bulinski et al., 2018).

Histogram-based estimation forms a bridge between discrete and continuous viewpoints. If $H(f)=-\int f(x)\log f(x)\,dx,$23 observations are binned into $H(f)=-\int f(x)\log f(x)\,dx,$24 fixed-width bins, with $H(f)=-\int f(x)\log f(x)\,dx,$25, then the histogram entropy is

$H(f)=-\int f(x)\log f(x)\,dx,$26

Watts and Crow relate this to differential entropy through

$H(f)=-\int f(x)\log f(x)\,dx,$27

where $H(f)=-\int f(x)\log f(x)\,dx,$28 is bin width, and propose the entropy-matched choice

$H(f)=-\int f(x)\log f(x)\,dx,$29

with $H(f)=-\int f(x)\log f(x)\,dx,$30 controlling over- and under-binning (Watts et al., 2022). The same work defines

$H(f)=-\int f(x)\log f(x)\,dx,$31

and interprets $H(f)=-\int f(x)\log f(x)\,dx,$32 as over-binning and $H(f)=-\int f(x)\log f(x)\,dx,$33 as under-binning (Watts et al., 2022).

5. Bayesian nonparametrics and sparse-support uncertainty

Bayesian methods target the regime in which the support size is unknown, large, or countably infinite. A foundational approach uses the Pitman–Yor process $H(f)=-\int f(x)\log f(x)\,dx,$34, with stick-breaking weights

$H(f)=-\int f(x)\log f(x)\,dx,$35

to define a random discrete distribution on the infinite simplex $H(f)=-\int f(x)\log f(x)\,dx,$36 (Hashino et al., 9 Feb 2026). Given observed counts $H(f)=-\int f(x)\log f(x)\,dx,$37, the posterior admits a Dirichlet–PYP decomposition:

$H(f)=-\int f(x)\log f(x)\,dx,$38

with an independent residual $H(f)=-\int f(x)\log f(x)\,dx,$39 process for the unseen species (Hashino et al., 9 Feb 2026).

The posterior-mean entropy estimator is then

$H(f)=-\int f(x)\log f(x)\,dx,$40

For $H(f)=-\int f(x)\log f(x)\,dx,$41,

$H(f)=-\int f(x)\log f(x)\,dx,$42

while for $H(f)=-\int f(x)\log f(x)\,dx,$43 the tail term is evaluated by truncation plus a correction of order

$H(f)=-\int f(x)\log f(x)\,dx,$44

(Hashino et al., 9 Feb 2026). Under regularly varying tails $H(f)=-\int f(x)\log f(x)\,dx,$45 with $H(f)=-\int f(x)\log f(x)\,dx,$46, the true entropy is finite, and the resulting estimator is consistent in probability as $H(f)=-\int f(x)\log f(x)\,dx,$47 (Hashino et al., 9 Feb 2026).

An earlier Bayesian treatment also uses Dirichlet and Pitman–Yor priors, but emphasizes analytical posterior moments and prior geometry. For a finite symmetric Dirichlet prior, the posterior mean satisfies

$H(f)=-\int f(x)\log f(x)\,dx,$48

and analogous closed forms exist for posterior variance (Archer et al., 2013). Under a PYP prior $H(f)=-\int f(x)\log f(x)\,dx,$49, the posterior mean is

$H(f)=-\int f(x)\log f(x)\,dx,$50

(Archer et al., 2013). The same work shows that a fixed Dirichlet or Pitman–Yor prior implies a narrow prior distribution over $H(f)=-\int f(x)\log f(x)\,dx,$51, so the prior strongly determines the entropy estimate in the undersampled regime; to mitigate this, it introduces a Pitman–Yor mixture prior designed to be approximately flat over $H(f)=-\int f(x)\log f(x)\,dx,$52 (Archer et al., 2013).

A related hierarchical-Bayesian approach uses a symmetric Dirichlet prior with hyperparameter $H(f)=-\int f(x)\log f(x)\,dx,$53, estimates $H(f)=-\int f(x)\log f(x)\,dx,$54 by empirical Bayes through the digamma equation

$H(f)=-\int f(x)\log f(x)\,dx,$55

and then computes

$H(f)=-\int f(x)\log f(x)\,dx,$56

(Piga et al., 2023). The reported computational complexity is $H(f)=-\int f(x)\log f(x)\,dx,$57, where $H(f)=-\int f(x)\log f(x)\,dx,$58 is the number of iterations in the one-dimensional root finding (Piga et al., 2023).

The common Bayesian theme is that unseen support must be treated as a posterior object rather than ignored. A plausible implication is that support uncertainty and entropy uncertainty cannot be cleanly separated in sparse regimes.

6. Structured data, alternative access models, and computational complexity

Entropy estimation changes qualitatively when the observation model departs from simple i.i.d. multinomial sampling. For memoryless $H(f)=-\int f(x)\log f(x)\,dx,$59-symbol sources, worst-case plug-in uncertainty can be bounded through $H(f)=-\int f(x)\log f(x)\,dx,$60, but for systems with memory, block entropies become central (Ricci et al., 2021, Gregorio et al., 2022). If $H(f)=-\int f(x)\log f(x)\,dx,$61 denotes the entropy of overlapping blocks of length $H(f)=-\int f(x)\log f(x)\,dx,$62, then for an $H(f)=-\int f(x)\log f(x)\,dx,$63-th order homogeneous Markov chain,

$H(f)=-\int f(x)\log f(x)\,dx,$64

so $H(f)=-\int f(x)\log f(x)\,dx,$65 is affine-linear in $H(f)=-\int f(x)\log f(x)\,dx,$66 beyond the memory depth (Gregorio et al., 2022). The proposed correlation-coverage estimator replaces the Chao–Shen coverage correction by a sequential coverage estimate $H(f)=-\int f(x)\log f(x)\,dx,$67 tailored to correlated blocks and then computes

$H(f)=-\int f(x)\log f(x)\,dx,$68

In binary Markov tests, the aggregated MSE of Chao–Shen was approximately $H(f)=-\int f(x)\log f(x)\,dx,$69 while the CC estimator gave approximately $H(f)=-\int f(x)\log f(x)\,dx,$70 (Gregorio et al., 2022).

In data streams, one-pass entropy estimation is achieved by estimating frequency moments through Compressed Counting. With

$H(f)=-\int f(x)\log f(x)\,dx,$71

one forms the Rényi entropy

$H(f)=-\int f(x)\log f(x)\,dx,$72

near $H(f)=-\int f(x)\log f(x)\,dx,$73. Compressed Counting uses maximally-skewed stable projections to obtain an estimator of $H(f)=-\int f(x)\log f(x)\,dx,$74, and then the optimal-quantile estimator minimizes the leading asymptotic variance constant for the scale estimate (0910.1495). The same work proves that Rényi entropy is better than Tsallis entropy for approximating Shannon entropy, in the sense that

$H(f)=-\int f(x)\log f(x)\,dx,$75

for all $H(f)=-\int f(x)\log f(x)\,dx,$76, where $H(f)=-\int f(x)\log f(x)\,dx,$77 and $H(f)=-\int f(x)\log f(x)\,dx,$78 (0910.1495).

In privacy-preserving distributed estimation, each user holds exactly one sample and communicates $H(f)=-\int f(x)\log f(x)\,dx,$79 bits under $H(f)=-\int f(x)\log f(x)\,dx,$80-local differential privacy. If a $H(f)=-\int f(x)\log f(x)\,dx,$81-variate discrete distribution factorizes as a tree graphical model with constant marginal support size $H(f)=-\int f(x)\log f(x)\,dx,$82, then Shannon entropy can be estimated to additive error $H(f)=-\int f(x)\log f(x)\,dx,$83 using

$H(f)=-\int f(x)\log f(x)\,dx,$84

users, with total communication $H(f)=-\int f(x)\log f(x)\,dx,$85 bits per user (Bravo-Hermsdorff et al., 2023). The algorithm uses the Chow–Liu identity

$H(f)=-\int f(x)\log f(x)\,dx,$86

where $H(f)=-\int f(x)\log f(x)\,dx,$87 is the maximum spanning tree weight under mutual-information edge weights (Bravo-Hermsdorff et al., 2023).

The conditional-sampling model yields another departure from standard sample complexity. In the probability-revealing conditional-sampling model, the oracle $H(f)=-\int f(x)\log f(x)\,dx,$88 returns a conditional sample together with its true probability. The estimator ENTROPY-EST is a median-of-means estimator built from the self-information variable $H(f)=-\int f(x)\log f(x)\,dx,$89, detects any heavy atom with probability $H(f)=-\int f(x)\log f(x)\,dx,$90, and obtains a multiplicative $H(f)=-\int f(x)\log f(x)\,dx,$91-approximation using

$H(f)=-\int f(x)\log f(x)\,dx,$92

queries on a domain of size $H(f)=-\int f(x)\log f(x)\,dx,$93 (Golia et al., 2022). This improves on the previously cited additive-$H(f)=-\int f(x)\log f(x)\,dx,$94 conditional-sampling complexity

$H(f)=-\int f(x)\log f(x)\,dx,$95

and avoids the $H(f)=-\int f(x)\log f(x)\,dx,$96 dependence that is unavoidable under SAMP+EVAL access (Golia et al., 2022).

Quantum access further changes the complexity landscape. In the Bravyi–Harrow–Hassidim oracle model, the first quantum algorithm for additive-$H(f)=-\int f(x)\log f(x)\,dx,$97 Shannon entropy estimation uses amplitude estimation and quantum Monte Carlo to achieve

$H(f)=-\int f(x)\log f(x)\,dx,$98

queries, compared with classical complexity $H(f)=-\int f(x)\log f(x)\,dx,$99 for constant additive error (Li et al., 2017). A later quantum probability-oracle result improves the upper bound to

$P=\{p_1,\dots,p_M\}$00

up to logarithmic factors, and states a matching lower bound up to logarithmic factors (Shin et al., 9 Sep 2025). This suggests that the entropy-estimation problem is unusually sensitive to the information-access primitive: the same functional can move from large-alphabet minimax estimation to near-optimal quantum singular-value transformation depending on what queries are allowed.

7. Practical estimation regimes and recurrent methodological themes

The literature supports a regime-dependent view of Shannon entropy estimation.

Setting	Representative approach	Reported property
Multinomial, known $P=\{p_1,\dots,p_M\}$01	Plug-in with $P=\{p_1,\dots,p_M\}$02 analysis	Worst-case $P=\{p_1,\dots,p_M\}$03 (Ricci et al., 2021)
Large alphabet discrete	Approximation-theoretic minimax estimator	Consistent for $P=\{p_1,\dots,p_M\}$04 (Han et al., 2015)
Very sparse linguistic data	Chao–Shen	Lowest MSE in the lowest-data regime (Arora et al., 2022)
Moderate $P=\{p_1,\dots,p_M\}$05 relative to $P=\{p_1,\dots,p_M\}$06	NSB	Clearly best once $P=\{p_1,\dots,p_M\}$07 exceeds a few times $P=\{p_1,\dots,p_M\}$08 (Arora et al., 2022)
Unknown or infinite support	Pitman–Yor-based Bayes estimators	Stable when observed species are fewer than true species (Hashino et al., 9 Feb 2026)
Differential entropy, $P=\{p_1,\dots,p_M\}$09	Spacings estimators	Generally performed better at univariate level (Madukaife et al., 2024)
Differential entropy, $P=\{p_1,\dots,p_M\}$10	KDE or kNN	KDE preferable at small $P=\{p_1,\dots,p_M\}$11; kNN exists for all $P=\{p_1,\dots,p_M\}$12 (Madukaife et al., 2024)

A recurring misconception is that plug-in estimation is generically adequate once it is unbiased to first order. The cited works indicate otherwise. In discrete large-alphabet problems, the decisive failure mode is often unseen mass rather than first-order Taylor bias (Han et al., 2015, Arora et al., 2022, Hashino et al., 9 Feb 2026). In continuous problems, consistency may require nontrivial conditions on local averages, tails, and bandwidth sequences (Bulinski et al., 2018, Fortune et al., 2020). In structured-sequence settings, correlation can invalidate coverage corrections designed for independence (Gregorio et al., 2022). In private, streaming, conditional-sampling, and quantum settings, computational or communication constraints become part of the statistical model itself (0910.1495, Bravo-Hermsdorff et al., 2023, Golia et al., 2022, Li et al., 2017).

Another recurrent theme is the decomposition of total error into bias, variance, and modelling mismatch. The variance parameter $P=\{p_1,\dots,p_M\}$13 isolates leading uncertainty for plug-in multinomial estimation (Ricci et al., 2021). Approximation-theoretic estimators trade polynomial-approximation bias against variance to achieve minimax rates (Han et al., 2015). Rényi-based data-stream methods explicitly optimize the variance–bias trade-off in choosing $P=\{p_1,\dots,p_M\}$14 near $P=\{p_1,\dots,p_M\}$15 (0910.1495). Bayesian nonparametrics relocate the dominant uncertainty into the posterior over unseen support (Archer et al., 2013, Hashino et al., 9 Feb 2026).

Taken together, these works portray Shannon entropy estimation as a technically heterogeneous domain unified by a single functional but fragmented by sampling regime. The strongest general conclusion is that no estimator is uniformly preferred across all settings: the appropriate method depends on whether the principal difficulty is large support, sparse counts, continuous geometry, dependence, privacy, streaming constraints, or restricted oracle access.