Bayesian Diversity Estimators

• Bayesian diversity estimators are probabilistic methods that assess variety measures by computing posterior expectations on random partitions.
• They leverage exchangeable random partitions and Poisson–Dirichlet priors to derive closed-form expressions for indices such as Shannon entropy and the Gini index.
• Their martingale structure ensures unbiased sequential updates and strong convergence to the true diversity functional as sample sizes increase.

Bayesian @@@@1@@@@ estimators are probabilistic functionals designed to assess quantitative measures of variety (such as entropy, Gini index, or other functionals of a population composition) under a prior model for the unknown distribution of types or species. The recent theoretical development established that, when constructed as posterior expectations conditional on observed exchangeable random partitions, these estimators form martingale sequences with strong convergence properties, and their local behavior closely mirrors that of classical plug-in (empirical) estimators for diversity functionals (Martinez, 15 Sep 2025).

1. Diversity Functionals on Random Partitions

Diversity indexes are considered as functions $G(s)$, where $s = (s(1), s(2), \ldots)$ is a (possibly infinite) sequence representing proportions of species or clusters, with $s(i) \ge 0$ and $\sum_{i} s(i) = 1$. Typical examples include:

• Shannon entropy: $H(s) = -\sum_{i} s(i) \log s(i)$
• Gini index: $G(s) = 1 - \sum_{i} s(i)^2$ More generally, any symmetric functional (i.e., invariant under species relabeling), often “of sum-type,” can be considered: $G(s) = \sum_{i} g(s(i))$ for some measurable $g$.

In the Bayesian setting, the unknown sequence $S$ is modeled as a random partition of the unit interval (i.e., as a random mass partition with law $P$ on the simplex). Observationally, $n$ individuals are assigned to species by allocating uniform random variables to the intervals determined by $S$, inducing a random partition of $\{1,\ldots,n\}$ and corresponding observed multiplicities $(n_1,\ldots,n_{k})$ (number of detected classes/species).

2. Bayesian Estimator Construction and Explicit Posterior Means

Given observed data (the partition $\Pi_n$), the Bayesian estimator for the diversity functional $G$ is constructed as the posterior expectation:

$E_{q^n}(G) = E[G(S) \mid \Pi_n],$

where $q^n$ designates the conditional law for $S$ given the current observation history.

Explicit formulas are available for some standard diversity indices under Poisson-Dirichlet priors with parameters $(\alpha, \theta)$. For instance, for the Shannon entropy $H$ and counts $(n_1, \ldots, n_k)$,

$$E_{q^n}(H) = (\theta + n + 1) - \frac{(\theta + k)}{(\theta + n)} (1 - \alpha) - \frac{1}{\theta + n} \sum_{i=1}^{k} n_i (n_i - \alpha + 1)$$

(see Equation (13), (Martinez, 15 Sep 2025)). Similar closed-form expressions are derived for the Gini index.

These derivations utilize the Dirichlet or Beta structure of the posterior partition weights, recognizing the conjugacy and independence among class proportions under the Poisson-Dirichlet prior (or related exchangeable partitioning mechanisms).

3. Exchangeable Random Partitions and Species Sampling Priors

The Bayesian model fundamentally employs exchangeable random partitions, leveraging Kingman’s paintbox representation and species sampling models. The prior on $S$ is typically a Poisson-Dirichlet process (PDP) with parameters $\alpha, \theta$, generating a random partition where the number of clusters and their frequencies are random.

Sampling proceeds by drawing $n$ i.i.d. uniform random variables allocated according to the underlying $S$. The induced partition $\Pi_n$ is exchangeable, as the law of the partition is invariant under permutations. This property ensures tractable posterior updates and calculable predictive distributions for cluster frequencies. The plug-in estimator, by contrast, simply uses $n_i/n$ as empirical frequencies to evaluate $G$.

4. Martingale Property and Convergence

A key result is that the sequence $\{E_{q^n}(G)\}_{n\ge1}$ is a martingale with respect to the natural filtration of observed partitions. That is,

$E(E_{q^{n+1}}(G) | \mathcal{F}_n) = E_{q^n}(G),$

where $\mathcal{F}_n$ is the $\sigma$-algebra corresponding to observed data up to stage $n$. Under $P$-integrability of $G$ (i.e., $E_P(|G(S)|) < \infty$), the martingale is uniformly integrable, so by standard results it converges both almost surely and in mean (i.e., $L^1$):

$$\lim_{n\to\infty} E_{q^n}(G) = G(S) \quad \text{a.s. and in } L^1.$$

This provides a strong theoretical guarantee that, as the data accumulate, the Bayesian estimate converges to the true diversity functional of the underlying species distribution almost surely. Additional $L^p$ convergence may be obtained when higher-order moments are finite.

5. Local Behavior and Relation to Plug-in Estimators

A central insight is that the one-step increments (i.e., the change from $n$ to $n+1$) of the Bayesian estimator for $G$ are locally analogous to those of the plug-in estimator. For both estimators, increments are manifest primarily when a previously unobserved species/class is discovered. In the Bayesian case, the martingale structure ensures unbiased sequential updating; in the plug-in case, weak convergence and consistency results (e.g., [Antos and Kontoyiannis, 2001]) lead to almost sure convergence under mild conditions.

In the Poisson–Dirichlet process, explicit formulas permit detailed comparison of these increments. For example, Corollary 1 of (Martinez, 15 Sep 2025) enumerates conditions under which the one-step increment is zero, characterizing both estimators’ behavior in terms of new species discovery events.

Property	Bayesian Posterior Estimator	Plug-in Estimator
Sequential updates	Martingale (self-correcting)	Empirical frequency updates
Almost sure convergence	Yes (a.s. to $G(S)$)	Yes (subject to conditions)
Response to new species	One-step increments at discovery	Same
Closed-form for common $G$	Available (e.g., for Shannon, Gini)	Trivial by evaluation

6. Practical Implications for Diversity Estimation

Modeling the unknown composition via exchangeable random partitions and using the corresponding Bayesian estimator yields robust, theoretically grounded procedures for diversity assessment, even in the presence of rare or unseen taxa. The martingale property provides strong control over estimation error via classical inequalities (e.g., Doob’s inequality), and ensures that each update is unbiased given the observed data.

Practical applications are diverse:

• Ecological studies: estimation of community entropy, Gini index, or more general functionals, in samples with many rare species.
• Machine learning: tracking diversity in clustering or random partition models.
• Forensic science: robust quantification of genetic or categorical variability.
• Any domain where discovery of new classes is central and uncertainty quantification on diversity is required.

The correspondence between Bayesian and plug-in estimators supports both theoretical and applied analysis, permitting dual use depending on availability of computational resources or prior information. The explicit characterization under Poisson–Dirichlet priors and analogous structures extends to generalized diversity measures (Rényi entropy, generalized Gini), positioning Bayesian diversity estimation as a flexible and generalizable framework for variety quantification (Martinez, 15 Sep 2025).