Identify the best candidate generators for reranking pipelines

Identify the best candidate-generating retrievers for reranking pipelines by evaluating retrieval recall at larger candidate-set cutoffs, such as the top 100 or top 1000 documents, rather than only at the small cutoffs used for standalone ranking metrics.

Background

The evaluation primarily uses small cutoffs, which reward retrievers for placing relevant documents near the top of the ranking. In a multistage retrieval pipeline, however, a reranker typically receives a much larger candidate set, so the relevant criterion is how many relevant documents are retrieved within the top 100 or top 1000. The paper therefore leaves unresolved which retrievers are most suitable as candidate generators for reranking systems.

References

Our results therefore show which retrievers rank well on their own, but identifying the best candidate generators for a reranking pipeline would require evaluating at these larger cutoffs, which we leave to future work.

— A Systematic Multi-Domain Evaluation of Document Retrievers  (2609.29455 - Velev et al., 24 Sep 2026) in Section 5, “Implications”

A first-stage retriever with higher R@5 need not produce a higher agent pass rate, since what matters downstream is whether the top-ranked skill is useful for the task, and our helpfulness-gap diagnostic (§4.4) shows these two can move in opposite directions. Running the strongest first-stage alternatives through the same cross-encoder and the same agent loop would settle this, and we leave it to future work.

— SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale  (2609.38822 - Yang et al., 30 Sep 2026) in Limitations, subsection “First-Stage Alternatives Are Compared by Recall Only”