ANN Search: Recall What Matters

Abstract: Approximate nearest neighbor (ANN) search has become a core primitive in information retrieval and modern machine learning tasks, from classification to retrieval-augmented generation. The community evaluates and tunes ANN algorithms primarily on their throughput at a given Recall@k, the fraction of true exact neighbors retrieved. We argue that what really matters in ANN search is the quality of the retrieved results and not their overlap with the true kNN set. We show that using Recall@k to assess retrieval quality forces unnecessary computational overhead and investigate replacing it by 1/Ratio@k, the inverse approximation ratio. 1/Ratio@k evaluates the differences between the distances of the retrieved and true neighbors. It is judge-free, hyperparameter-free, and computable from standard ANN benchmark inputs alone. We benchmark state-of-the-art ANN algorithms across diverse datasets spanning a wide range of intrinsic dimensionalities, evaluating the two metrics comprehensively across efficiency, downstream classification, and retrieval-augmented generation. On the efficiency axis, optimizing for 1/Ratio@k reaches operational quality thresholds at a substantially lower computational cost than Recall@k. In downstream tasks, performance indicators (label precision, semantic similarity, BERTScore, and LLM-graded quality) remain highly stable even when Recall@k drops significantly. The inverse approximation ratio, on the other hand, closely mirrors this stability, tracking true utility much better than Recall@k. Ultimately, while Recall@k overstates the true cost of approximation, 1/Ratio@k offers a more accurate, deployable proxy for actual ANN quality.

• The paper introduces 1/Ratio@k as a more meaningful alternative metric that captures geometric quality better than Recall@k.
• It demonstrates through empirical analysis that 1/Ratio@k yields significant QPS speedups and lower computational costs while closely tracking downstream task performance.
• The proposed metric is hyperparameter-free and derived purely from embeddings, offering a robust and standardized evaluation for ANN search.

Rethinking ANN Evaluation: 1/Ratio vs Recall

Introduction and Motivation

Approximate Nearest Neighbor (ANN) search is foundational to high-dimensional retrieval tasks across information retrieval, recommender systems, and retrieval-augmented generation (RAG). For over a decade, community benchmarking and algorithmic design for ANN have fixated on Recall@$k$ as the principal metric, shaping the QPS–Recall@$k$ Pareto frontier as the litmus for index quality. Dimitropoulos and Mamoulis, in "ANN Search: Recall What Matters" (2606.04522), provide an incisive critique of this convention. They demonstrate that Recall@$k$ overstates the cost of approximation and propose a judge-free, hyperparameter-free, embedding-only alternative: $1/\text{Ratio}@k$, the inverse average per-rank distance ratio, as a more operationally meaningful metric.

Limitations of Recall@$k$ as an ANN Metric

Recall@$k$ quantifies the fraction of exact top-$k$ matches returned by an ANN algorithm, essentially asserting identifier-level overlap as the target. This measure is agnostic to the relative geometric proximity of retrieved candidates and ground-truth neighbors. As dataset intrinsic dimensionality rises, equidistant candidates proliferate and true neighbors become less discernible by identifiers alone, causing Recall@$k$ to degrade disproportionately.

This misalignment becomes problematic as real-world downstream utility (classification, RAG, etc.) is much less sensitive to permutation within sets of near-equal vectors than Recall@$k$ implies. The rigor of the QPS–Recall protocol pessimistically biases algorithmic tuning, incurring unnecessary computational and engineering cost. Moreover, Recall@$k$-based evaluation often dismisses configurations that are practically indistinguishable in utility from exact search, particularly at moderate- or low-recall regimes.

The authors further situate recent efforts—Semantic Recall, Tolerant Recall, and task-centric evaluation frameworks like ICEBERG—within this context. These methods require labeled data, external judges (often LLMs), or application-specific hyperparameters, impairing standardization and deployability at scale.

The 1/Ratio@$k$0 Metric: Definition and Properties

$k$1 is defined as the inverse of the average per-rank distance ratio between the retrieved set and the exact neighbor set:

$k$2

Here, $k$3 and $k$4 are the distances from the query $k$5 to the $k$6-th exact and retrieved neighbors, respectively. Both Recall@$k$7 and $k$8 are in $k$9, but only the latter reflects geometric quality. Crucially, $k$0 can be computed from embeddings alone, without the need for application semantics or hyperparameters.

Empirical Analysis: Cost, Efficiency, and Quality

Computational Overhead and Efficiency (RQ1)

Comprehensive cost-accuracy benchmarking across five SOTA algorithms (Annoy, SuCo, HNSW, RaBitQ, SymphonyQG) and six datasets reveals that achieving a given Recall@$k$1 threshold requires vastly more computational resources compared to achieving the same $k$2 value. Efficient configurations for $k$3 routinely yield QPS speedups (often greater than $k$4 and as high as $k$5), with similar trends in the number of distance computations and build times.

Figure 1: Minimum distance computations per query to meet target quality for each algorithm at several thresholds. $k$6-based tuning requires dramatically fewer operations than Recall across all ANN methods.

Notably, the magnitude of this advantage grows with the neighbor set size $k$7 and with the local intrinsic dimensionality (LID) of the dataset, tightly linking the metric divergence to the underlying geometry rather than algorithmic idiosyncrasy.

Figure 2: QPS speedup of $k$8 over Recall as a function of neighborhood size $k$9, showing that the speedup increases monotonically with $1/\text{Ratio}@k$0.

Algorithm Rankings and Metric-Induced Biases (RQ2)

Replacing Recall@$1/\text{Ratio}@k$1 with $1/\text{Ratio}@k$2 for evaluation does not generally change the relative ranking of ANN algorithms in terms of throughput or build cost; however, absolute differences—and Pareto optimality—are dramatically skewed. Multiple algorithms that would be dismissed on Recall (e.g., Annoy on difficult datasets) are competitive when measured with $1/\text{Ratio}@k$3. This exposes the inefficiency of benchmark practices tuned for Recall@$1/\text{Ratio}@k$4.

Intrinsic Dimensionality and the Metric Divergence (RQ3)

The gap between Recall@$1/\text{Ratio}@k$5 and $1/\text{Ratio}@k$6 is highly predictable from dataset LID and grows with $1/\text{Ratio}@k$7. High-LID regimes (e.g., SimpleWiki, Gist) exacerbate the probability that ANN returns center-close but identifier-different items, penalizing Recall@$1/\text{Ratio}@k$8 but leaving actual quality intact.

Downstream Task Correlation: Classification and RAG (RQ4)

The central question is whether Recall@$1/\text{Ratio}@k$9 misspecifies practical performance. The authors evaluate this via two paradigms:

• Classification: On four vision benchmarks, k-NN classification quality (Label Precision@100) remains essentially flat as Recall drops from 1.0 to 0.4. In contrast, $k$0 is a highly accurate predictor, tracking real performance deviations to within a few percent.

Figure 3: LP@100 and 1/Ratio@100 (normalized to exact search) vs synthetic recall. Actual classification quality remains almost unaffected across wide drops in Recall, while $k$1 accurately models this stability.

• Retrieval-Augmented Generation: Even under aggressive recall loss, BERTScore, semantic similarity, and LLM-graded answer quality—as measured by both human and LLM reference answers—degrade minimally. Across multiple QA and BEIR datasets, absolute downstream performance drops by less than $k$2 while Recall drops by $k$3. $k$4 again traces the residual quality loss to within tight bounds.

Quantitative Proxy Analysis

To rigorously compare metrics, the authors compute the mean absolute deviation (MAD) between downstream performance and its proxy—Recall@$k$6 or $k$7. Across all datasets and tasks, $k$8 achieves sub-3\% MAD; Recall@$k$9 exhibits errors an order of magnitude higher, consistently misrepresenting the true utility delivered to downstream prediction or generative tasks.

Figure 4: MAD between Recall or $k$0 and measured downstream quality, demonstrating that $k$1 is a consistently superior proxy.

Implications and Future Directions

The findings of this study have direct practical and theoretical implications:

• Benchmarking and System Design: Adopting $k$2 as the canonical metric unlocks substantial computational economies during both system tuning and large-scale deployment. Costly configurations targeting high Recall can be replaced with much cheaper, faster alternatives that achieve the same operational quality.
• Metric Choice and Algorithm Development: Since $k$3 exhibits lower task-agnostic volatility and aligns with geometric as well as downstream utility, it is better suited for universal benchmarks and facilitates more robust comparison across algorithmic families.
• Standardization: Unlike task-centric or LLM-judged alternatives, $k$4 can be retroactively computed on all standard ANN benchmarks, providing forward and backward compatibility for evaluation, and offers a single interpretable operational axis for both research and production settings.
• Future Progress: More granular investigation of how $k$5 interacts with different retrieval architectures (quantization, graph, hash, hybrid) can reveal further efficiency frontiers. Additionally, as new semantically meaningful embedding spaces are developed, geometric measures such as $k$6 provide robust metrics invariant to semantic label drift or judge bias.

Conclusion

This work provides a methodologically rigorous, quantitatively decisive case against the primacy of Recall@$k$7 for ANN evaluation. The empirical evidence across modalities, algorithm families, and downstream tasks establishes $k$8 as a computationally efficient, operationally meaningful, and robust alternative. Theoretical and empirical alignment with geometric proximity and real-world utility imply that both ANN research and practice should migrate toward $k$9 optimization and benchmarking for more accurate and efficient deployment.

Reference:

"ANN Search: Recall What Matters" (2606.04522).