The Unseen Species Problem Revisited

Abstract: The unseen species problem is a classical problem in statistics. It asks us to, given n i.i.d. samples from an unknown discrete distribution over an unknown set, predict how many never before seen outcomes would be observed if m additional samples were collected. For small m we show the classical but poorly understood Good-Toulmin estimator to be minimax optimal to within a factor 2 and resolve the open problem of constructing principled prediction intervals for it. For intermediate m we propose a new estimator which achieves the minimax error for linear estimators up to an explicit multiplicative constant. Our estimator vastly outperforms the standard Smoothed Good-Toulmin estimator in the worst case and performs substantially better on several real data sets, namely those with many rare species. For large m we show that a previously mentioned estimator which did not have known rate guarantees actually achieves a marginally better rate than subsequent work. We find that this marginal rate improvement translates to meaningfully better performance in practice. We show in all three regimes that the same methods also achieve the same rate on incidence data, without further independence assumptions, provided that the sets are of bounded size. We establish, by means of bounded size biased couplings, concentration for some natural functionals of sequences of i.i.d. discrete-set-valued random variables which may be of independent interest.