Unseen Species Models: Inference & Transfer

• USMs are frameworks for inferring the number of unseen species using observed fingerprint data and models like Good–Toulmin and Chao estimators.
• They extend classical statistical methods by incorporating side information such as taxonomy, text, and images for zero-shot and few-shot prediction.
• USMs have diverse applications in ecology, computational musicology, and beyond, emphasizing robust uncertainty quantification under incomplete observation.

Unseen Species Models (USMs) are frameworks for inference under incomplete support: the observed data do not exhaust the species universe. In the classical statistical formulation, one observes samples from an unknown discrete population and predicts either the number of species still unseen or the number of new species that would appear after additional sampling. In more recent machine-learning usage, the same label is also applied to models that generalize to species absent from training by using side information such as taxonomy, text, images, or sparse occurrence records. Across these literatures, the common object is inference when the relevant species set is not closed by the data already in hand (Orlitsky et al., 2015, Stolf et al., 2024, Shinoda et al., 23 Mar 2026).

1. Classical statistical formulation

The canonical unseen-species problem starts from an observed sample of size $n$ from an unknown discrete population and asks how many new species would be observed in an additional sample of size $m$. Writing $m=tn$, a standard target is

$U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$

the number of distinct symbols that appear in the future sample but were absent from the original one. The sufficient statistics used by the main estimators are the fingerprint or prevalence counts

$\Phi_i=\sum_x \mathbf 1_{N_x=i},$

where $N_x$ is the observed count of species $x$. Linear estimators take the form

$\hat U=\sum_{i\ge 1}\Phi_i h_i.$

This formulation is studied under multinomial, Poisson, hypergeometric, and Bernoulli-product sampling models, with the Poisson model serving as the main analytic core because species counts become independent (Orlitsky et al., 2015).

A closely related formulation is support-size estimation. If $P$ is a discrete distribution on $\mathcal X$, then

$m$0

is the total support size, while

$m$1

is the number of support elements not yet observed. Under the standard model

$m$2

support-size estimation is equivalent to estimating the unseen count $m$3, since the total support decomposes into observed distinct species plus unobserved support points (Rajaraman et al., 2020).

Bayesian nonparametric work uses a parallel notation. With $m$4 sampled from an unknown discrete law, the number of hitherto unseen species in $m$5 additional draws is

$m$6

equivalently $m$7. In that literature the main inferential targets are the posterior mean of $m$8 and credible intervals for its posterior law (Contardi et al., 27 Jan 2025).

2. Estimator families and optimality theory

The classical baseline is the Good–Toulmin estimator,

$m$9

Under Poisson sampling it is exactly unbiased for every $m=tn$0, but its coefficients grow exponentially when $m=tn$1. The central structural insight is that Good–Toulmin fails beyond $m=tn$2 because of variance rather than bias. Deterministic truncation reduces variance but creates nonvanishing bias, whereas random truncation produces the Smoothed Good–Toulmin (SGT) class

$m=tn$3

whose coefficients damp the unstable $m=tn$4 term while averaging out alternating truncation bias. With suitable Poisson or binomial smoothing, SGT estimators achieve worst-case normalized MSE decaying like $m=tn$5, and accurate prediction is possible uniformly over all distributions up to the sharp frontier $m=tn$6. The same analysis yields the first finite-sample, distribution-free guarantee for the Efron–Thisted estimator by identifying it as a special case of random truncation smoothing (Orlitsky et al., 2015).

A distinct classical strand centers on the Chao estimator. In its basic form,

$m=tn$7

and in the analyzable regularized form,

$m=tn$8

Under Poisson sampling over $m=tn$9, the modified Chao estimator has worst-case MSE smaller than the plug-in estimator by a factor of order $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$0 when $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$1. This gives explicit theory for a widely used low-order fingerprint estimator, while also showing that Chao remains suboptimal relative to modern polynomial-based minimax procedures (Rajaraman et al., 2020).

Recent theory revisits the horizon dependence more finely. In the near-future regime $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$2, the Good–Toulmin estimator is minimax optimal to within a factor $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$3, and its prediction error admits a Gaussian approximation after normalization by an explicit variance proxy. For intermediate horizons, a new linear estimator is defined by minimizing an explicit surrogate $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$4 for worst-case MSE; the resulting estimator achieves the minimax error for linear estimators up to the explicit multiplicative constant $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$5. For large horizons under regular variation, the estimator

$U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$6

with $U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$7 attains a better large-$U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$8 rate than earlier work, and the same rate carries over to incidence data with bounded set size (Eriksson, 9 Feb 2026).

3. Bayesian nonparametrics and uncertainty quantification

A major Bayesian formulation places a Pitman–Yor prior on the unknown discrete law,

$U=\left|\{X_{n+1}^{n+m}\}\setminus \{X^n\}\right|,$9

with $\Phi_i=\sum_x \mathbf 1_{N_x=i},$0 and $\Phi_i=\sum_x \mathbf 1_{N_x=i},$1. The special case $\Phi_i=\sum_x \mathbf 1_{N_x=i},$2 is the Dirichlet process. Given $\Phi_i=\sum_x \mathbf 1_{N_x=i},$3 observed species, the posterior law of $\Phi_i=\sum_x \mathbf 1_{N_x=i},$4 is available in closed form through generalized factorial coefficients, and the posterior mean is explicit both for $\Phi_i=\sum_x \mathbf 1_{N_x=i},$5 and for the Dirichlet case $\Phi_i=\sum_x \mathbf 1_{N_x=i},$6 (Contardi et al., 27 Jan 2025).

The main recent advance in this line is asymptotic uncertainty quantification for large additional sample size $\Phi_i=\sum_x \mathbf 1_{N_x=i},$7. Earlier asymptotic intervals based on a scaled Mittag–Leffler limit worked only for $\Phi_i=\sum_x \mathbf 1_{N_x=i},$8 and required Monte Carlo sampling. A newer Gaussian central limit theorem studies the regime

$\Phi_i=\sum_x \mathbf 1_{N_x=i},$9

and proves that the posterior distribution of $N_x$0 is asymptotically normal with explicit mean and variance expansions. This yields analytic Gaussian credible intervals for the full Pitman–Yor family, including the Dirichlet-process case, and avoids Monte Carlo entirely. Empirically, these Gaussian intervals track exact posterior intervals more closely than Mittag–Leffler approximations over a broad practical range of $N_x$1 (Contardi et al., 27 Jan 2025).

This Bayesian perspective differs from the distribution-free minimax tradition in two ways. First, the prior itself encodes tail behavior and species-richness regularity. Second, the inferential object is the posterior law of future novelty rather than only a worst-case point estimator. A plausible implication is that the choice between the two traditions is driven less by notation than by whether the application demands robustness over all discrete laws or calibrated prediction under a structured prior.

4. Multi-population, hierarchical, and covariate-dependent USMs

Single-population exchangeability is often too restrictive. For two partially exchangeable samples $N_x$2 and $N_x$3, one can define local future discoveries

$N_x$4

global discoveries unseen in both areas,

$N_x$5

and newly shared species,

$N_x$6

Under the Vec-FDP prior, exact predictive distributions are available for local, global, and shared discoveries for arbitrary future sample sizes $N_x$7, together with shared-species coverage probabilities such as $N_x$8. This extends one-step shared-species estimators to arbitrary look-ahead horizons (Colombi et al., 6 Feb 2025).

A more abstract generalization replaces one exchangeable sequence by multiple partially exchangeable groups. Multivariate species sampling models represent a vector of dependent random probability measures $N_x$9 and are characterized by a partially exchangeable partition probability function (pEPPF). The predictive probability that the next observation in group $x$0 is a previously unseen species is expressed as a ratio of pEPPFs, and the framework distinguishes shared from group-specific latent species through explicit decompositions of the random measures. In the regular subclass, correlation $x$1 corresponds to full exchangeability and correlation $x$2 corresponds to independence (Franzolini et al., 31 Mar 2025).

Hierarchical asymptotics provide another route. In hierarchical species sampling models, the global number of observed species $x$3 and the number $x$4 of species observed exactly $x$5 times can be represented as their non-hierarchical analogues evaluated at a random sample size $x$6. For hierarchical Pitman–Yor models this yields almost sure and $x$7 limits, Gaussian fluctuation theorems for $x$8, and large deviation principles for both $x$9 and $\hat U=\sum_{i\ge 1}\Phi_i h_i.$0. In the fully Pitman–Yor case, both $\hat U=\sum_{i\ge 1}\Phi_i h_i.$1 and $\hat U=\sum_{i\ge 1}\Phi_i h_i.$2 grow on the scale $\hat U=\sum_{i\ge 1}\Phi_i h_i.$3, while the asymptotic frequency-of-frequency profile is governed by the top-level discount parameter through $\hat U=\sum_{i\ge 1}\Phi_i h_i.$4 (Favaro et al., 16 Jan 2025).

A different extension appears in infinite joint species distribution models. TRACE combines open-ended species support, covariate-dependent occurrence modeling, and cross-species dependence in a multivariate probit-like framework,

$\hat U=\sum_{i\ge 1}\Phi_i h_i.$5

with a truncation scheme that converges to an infinite species limit. The key predictive quantity is

$\hat U=\sum_{i\ge 1}\Phi_i h_i.$6

the number of new species discovered in $\hat U=\sum_{i\ge 1}\Phi_i h_i.$7 additional samples. TRACE does not target a finite total species richness; instead it models future discovery and assemblage structure under covariates and residual species correlation (Stolf et al., 2024).

5. Machine-learning reinterpretations

In contemporary machine learning, “unseen species modeling” often refers not to extrapolating future discovery counts but to zero-shot or few-shot generalization to species absent from training. One example is LD-SDM, which reformulates species distribution modeling from $\hat U=\sum_{i\ge 1}\Phi_i h_i.$8 to

$\hat U=\sum_{i\ge 1}\Phi_i h_i.$9

so that a species becomes an input representation rather than a fixed output class. Species are encoded by taxonomy text using frozen LLaMA-2 embeddings, fused with geographic and environmental features through cross-attention, and used for zero-shot range prediction on 84 bird species held out from training. The best reported zero-shot result is $P$0 with environmental features and ASL loss (Sastry et al., 2023).

Few-shot range estimation pushes the same principle further. FS-SINR receives a support set of occurrence locations

$P$1

for a novel species $P$2, together with optional text or image metadata, infers a species embedding in a single forward pass, and predicts range scores by combining that embedding with a shared location encoder. This avoids test-time optimization for each new species and improves low-shot range estimation on IUCN and SNT benchmarks relative to optimization-based baselines (Lange et al., 20 Feb 2025).

Bioacoustic and visual recognition work uses a similar open-support idea with taxonomic or textual side information. AnimalCLAP trains a CLAP-style audio–text model on 4,225 hours of vocalizations from 6,823 species and uses prompt randomization over common names, scientific names, and taxonomic sequences. Its reported unseen-species benchmark is intentionally constrained: the 300 held-out test species are disjoint from training, but their genera and families are represented in the training subset. Under that setting, the best zero-shot species-recognition result is top-1 accuracy $P$3 with Tax+Com prompts (Shinoda et al., 23 Mar 2026).

Open-vocabulary image recognition adopts an even larger external candidate space. VR-RAG builds a knowledge base of 11,202 bird species from Wikipedia, retrieves top $P$4 candidate chunks by image–text similarity, reranks them with DINOv2 using up to three anchor images per species, and asks a large multimodal model to reason over the top $P$5 full summaries. On five bird benchmarks evaluated against the full 11,202-species candidate set, Qwen2.5-VL + VR-RAG reaches average top-1 accuracy $P$6, improving plain Qwen2.5-VL by $P$7 points (Khan et al., 8 May 2025).

USM-style transfer has also been extended from identity to structure. Few-shot keypoint detection for unseen species uses support-defined keypoint prototypes, uncertainty-aware localization, and multivariate Gaussian modeling over neighboring keypoints to detect both base and novel keypoints on species absent from training (Lu et al., 2021). This suggests that, in machine learning, the USM label has broadened from unseen counts to unseen categories, structures, and functions.

6. Applications, evaluation regimes, and limitations

USMs have been exported beyond ecology. In computational musicology, Chao-style unseen-species estimation has been used to ask how many composers are missing from RISM, what fraction of Gregorian chant repertories have been cataloged, how complete an ontology of editorial differences is, and how much of a folk or harmonic vocabulary has been observed. The central move is to reinterpret “species” as cultural units such as composers, chant IDs, tune types, difference categories, or harmonic types, while retaining the fingerprint logic based on $P$8, $P$9, and coverage estimates (Moss et al., 19 Jul 2025).

At the same time, the literature is explicit that not every benchmark labeled “unseen species” tests unrestricted open-world generalization. AnimalCLAP preserves taxonomic connectivity by ensuring that held-out species have seen genera and families, and LD-SDM holds out species rather than genera or families (Shinoda et al., 23 Mar 2026, Sastry et al., 2023). VR-RAG is open-vocabulary only with respect to the external knowledge base: a species absent from the 11,202-species retrieval universe cannot be named by the system (Khan et al., 8 May 2025). This suggests that “unseen species” in modern ML often means held-out species under structured side information, not de novo recognition of taxonomically remote or undescribed species.

Another recurring limitation is robustness under nuisance shift. The BioDCASE 2026 baseline for mosquito audio classification is not a USM method, but it isolates a deployment problem that USMs inherit: severe domain dependence. Its closed-set baseline achieves $\mathcal X$0 yet only $\mathcal X$1, with domain shift gap $\mathcal X$2. The paper’s point is that within-domain success can hide near-collapse under unseen recording conditions (Hou et al., 20 Mar 2026).

Theoretical limitations are equally sharp. In the classical distribution-free setting, accurate extrapolation is possible only up to $\mathcal X$3 in the worst case (Orlitsky et al., 2015). Beyond that, meaningful prediction requires additional structure, such as regular variation in the species frequencies (Eriksson, 9 Feb 2026) or a Bayesian prior such as Pitman–Yor (Contardi et al., 27 Jan 2025). In multi-population settings, exact formulas are available only for specific dependent priors and can become combinatorially heavy (Colombi et al., 6 Feb 2025). In infinite JSDMs, the support is open-ended, but posterior computation relies on truncation and approximate inference rather than exact full-dimensional sampling (Stolf et al., 2024).

Taken together, these results delineate two enduring meanings of USMs. In statistics, USMs are models for future novelty, richness, and frequency-of-frequency structure under incomplete observation. In modern machine learning, they are models for zero-shot or few-shot transfer to species outside the training label set. The overlap is real but not complete: both study open support, yet they differ in targets, assumptions, and evaluation protocols.