Identifying optimal configurations for RAG pipelines under practical constraints

Determine the optimal configurations of retrieval-augmented generation pipelines—including choices of retrievers, embedding models, language models, chunk sizes, and context limits—when labeled data is scarce and query distributions are diverse and unpredictable, given the practical limitations of current evaluation and tuning tools such as RAGGED, RAGAS, and RAGProbe.

Background

The authors review existing evaluation and tuning tools for RAG pipelines, such as RAGGED, RAGAS, and RagProbe, and note practical limitations in real-world settings: scarce labeled data and wide variability in user queries.

Against this backdrop, they explicitly state that optimal configurations are often unknown, highlighting an unresolved challenge in selecting and tuning RAG components and parameters for specific applications.