Identifying optimal configurations for RAG pipelines under practical constraints

Determine the optimal configurations of retrieval-augmented generation pipelines—including choices of retrievers, embedding models, language models, chunk sizes, and context limits—when labeled data is scarce and query distributions are diverse and unpredictable, given the practical limitations of current evaluation and tuning tools such as RAGGED, RAGAS, and RAGProbe.

Background

The authors review existing evaluation and tuning tools for RAG pipelines, such as RAGGED, RAGAS, and RagProbe, and note practical limitations in real-world settings: scarce labeled data and wide variability in user queries.

Against this backdrop, they explicitly state that optimal configurations are often unknown, highlighting an unresolved challenge in selecting and tuning RAG components and parameters for specific applications.

References

However, these tools have practical limitations: optimal configurations are often unknown, labeled data is scarce, and the range of possible queries is vast and unpredictable.

— RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines  (2504.13587 - Lauro et al., 18 Apr 2025) in Section 2.2 (Related Work: RAG Pipeline Development) — RAG Development Challenges

We distinguish auto-configuration (one-time per-domain setup at ingestion time) from auto-tuning (continuous online adaptation as the corpus and query distribution evolve); both are open engineering challenges whose solution would complete the self-improving knowledge perception loop shown in Figure~\ref{fig:UniK_overview}.

— UniK: Universal Knowledge Perception for Digital and Physical AI  (2609.23971 - Desai et al., 21 Sep 2026) in Section 3, 'The UniK Platform'