LTRR: Learning To Rank Retrievers for LLMs (2506.13743v1)

Abstract: Retrieval-Augmented Generation (RAG) systems typically rely on a single fixed retriever, despite growing evidence that no single retriever performs optimally across all query types. In this paper, we explore a query routing approach that dynamically selects from a pool of retrievers based on the query, using both train-free heuristics and learned routing models. We frame routing as a learning-to-rank (LTR) problem and introduce LTRR, a framework that learns to rank retrievers by their expected utility gain to downstream LLM performance. Our experiments, conducted on synthetic QA data with controlled query type variations, show that routing-based RAG systems can outperform the best single-retriever-based systems. Performance gains are especially pronounced in models trained with the Answer Correctness (AC) metric and with pairwise learning approaches, especially with XGBoost. We also observe improvements in generalization to out-of-distribution queries. As part of the SIGIR 2025 LiveRAG challenge, our submitted system demonstrated the practical viability of our approach, achieving competitive performance in both answer correctness and faithfulness. These findings highlight the importance of both training methodology and metric selection in query routing for RAG systems.

Overview of LTRR: Learning To Rank Retrievers for LLMs

This paper investigates the innovative application of query routing within Retrieval-Augmented Generation (RAG) systems, where LLMs are bolstered by external knowledge retrieval methods. Present RAG systems commonly use a single, fixed retriever, despite the recognition that no singular retriever performs optimally for every query type. To address this limitation, the authors introduce a novel framework that advances the field by dynamically selecting retrievers from a varied pool using both heuristic approaches and trained routing models framed within a learning-to-rank (LTR) problem: Learning to Rank Retrievers (LTRR).

Key Contributions and Methodology

The LTRR framework is designed to enhance LLM utility by learning to rank retrievers based on their contribution to downstream performance rather than traditional retrieval metrics. This is framed as a learning-to-rank problem using various LTR algorithms, incorporating both train-free and learning-based models. The approach leverages train-free heuristics, such as query-corpus similarity, alongside learning-to-rank models including XGBoost, SVM, feedforward networks (FFN), and DeBERTa, optimizing for both pointwise and pairwise performance metrics.

An essential aspect of this research is the diverse experimental setup entailing six distinct retrieval strategies combining sparse BM25 and dense E5 retrievers, each paired with reranking strategies for specific goals. By aligning the framework with a query routing function that includes a 'no-retrieval' option, the model presents a comprehensive solution to select retrievers and assess the necessity of retrieval.

Experimental Validation and Results

Experiments were conducted on a synthetic QA dataset that control variations in query types demonstrating the model's capacity to adapt and outperform the best single-retriever systems. The LTRR exhibited pronounced performance improvements especially when trained using the Answer Correctness (AC) metric and pairwise learning with XGBoost. Statistical significance tests confirm these gains, particularly in scenarios involving unseen query types, underscoring the framework's generalization capability.

The paper also highlights the critical nature of utility metrics in determining the efficacy of learning models in this context. While both Answer Correctness (AC) and BEM utility metrics were used, the results favored models trained with AC, indicating its higher alignment with human evaluations of LLM performance.

Implications and Future Directions

This research provides a compelling shift in RAG system design towards more dynamic and contextually adaptive architectures. The inclusion of a 'no-retrieval' option in query routing paves the way for not just selecting the most appropriate retrieval strategy but also discerning when the usage of an external retriever is unnecessary. It opens avenues for more nuanced interactions between LLMs and information retrieval systems, potentially leading to reduced computational costs and increased retrieval efficiency.

Future developments may look into expanding the LTRR's architecture to support multi-retriever selection, thereby promoting enhanced content diversity and broader information coverage in RAG systems. This framework serves as a robust foundation for future explorations, particularly in evolving information environments with a proliferation of retrieval techniques.

In conclusion, the LTRR framework represents a significant advancement in optimizing retrieval strategies within RAG systems, providing empirical evidence of its benefits over traditional, single-retriever setups and establishing a model that is not only adaptable to varied query types but also potentially transformative to the retrieval-augmented generation landscape.