Estimate unresolved retrieval-tail hardness from small samples

Develop a reliable method for estimating the per-query hardness values of retrieval positives with hardness x_q less than or approximately equal to 1/S, and therefore invisible in a distractor sample of size S, so that large-gallery retrieval accuracy can be predicted without indexing the full gallery.

Background

For a distractor sample of size S, queries whose per-query hardness satisfies x_q less than or approximately equal to 1/S are unlikely to have any sampled distractor outrank the positive. A direct count estimator therefore returns zero for these queries, incorrectly treating them as immune to ranking errors at larger gallery sizes.

The paper addresses this unresolved-tail issue using a fitted extreme-value-theory model, specifically a bounded generalized Pareto tail, and validates the resulting estimates empirically against 100M-distractor retrieval benchmarks. The cited prior work is identified as having posed the underlying sample-based tail-estimation gap as open; the present paper proposes and evaluates one solution rather than establishing a general guarantee for all distributions or datasets.

References

The queries that dominate large-$N$ accuracy have $x_q\lesssim1/S$ and are invisible in the sample ($K{=}0$); counting returns $0$, so we estimate their hardness from the fitted upper tail---the gap posed as open by \citet{madigan2003retrieval}.

— Web Scale on a Budget: Tail-Aware Retrieval Evaluation  (2610.17986 - Meivar et al., 13 Oct 2026) in Appendix, Section “Error analysis of the mAP estimator,” paragraph “Estimation term: the unresolved tail needs the model”