Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage (2410.15531v1)

Abstract: Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types -- core, background, and follow-up -- to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

• The paper introduces a novel evaluation framework that decomposes queries into core, background, and follow-up sub-questions to assess RAG responses.
• It demonstrates that core sub-question coverage is crucial, revealing a 50% oversight rate in commercial systems and a 74% win rate when integrated.
• Leveraging GPT-4 for automatic classification, the approach aligns with human ranking preferences at 82% accuracy and suggests clear optimization paths.

Evaluating and Optimizing RAG Systems with Sub-Question Coverage

The paper "Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage" presents an examination of retrieval-augmented generation (RAG) systems, focusing on a novel evaluation framework employing sub-question coverage as a key metric for assessing response quality. The authors propose decomposing complex, open-ended questions into core, background, and follow-up sub-questions to enhance the analytical depth of RAG system evaluations.

Key Contributions

A central innovation of this work lies in its sub-question coverage framework, which introduces a more nuanced perspective on evaluation. By assessing how well a RAG system's responses address these categorized sub-questions, the paper provides finer insights into the efficacy of retrieval and generation processes. This evaluation strategy involves several distinct metrics, including coverage rates for core, background, and follow-up sub-questions in both retrieved content and final answers.

The authors conducted their analysis on commercial answer engines like You.com, Perplexity AI, and Bing Chat, revealing significant gaps in core sub-question coverage, with approximately 50% of core sub-questions overlooked. The measured sub-question coverage also demonstrated a capacity to approximate human preferences, achieving 82% accuracy in ranking responses, which surpassed traditional LLM-as-a-judge models.

Methodological Framework

The paper proposes a robust methodology for sub-question decomposition, leveraging GPT-4 for automatic classification, adhering to types: core, background, and follow-up. This classification informs a range of metrics, such as retrieval coverage and identification capabilities, providing a comprehensive understanding of RAG systems' strengths and limitations.

One of the pivotal innovations introduced is the improvement of RAG responses by integrating core sub-questions into retrieval and generation stages, yielding a 74% win rate over the baseline. This emphasizes the critical role that core sub-question coverage plays in enhancing response quality, suggesting important directions for system optimization.

Implications and Future Directions

The implications of this paper extend beyond mere evaluation, suggesting potential enhancements in RAG systems' architecture, particularly by focusing on retrieval completeness and core sub-question emphasis. The demonstrated correlation between core sub-question coverage and answer quality reveals opportunities for optimizing training processes and response generation models.

The framework also opens avenues for future research in AI and NLP, such as developing datasets and metrics that more accurately reflect the multi-dimensional nature of complex queries. Extending these insights into real-world applications, such as automated customer support and information retrieval systems, may lead to substantial advancements in those fields.

Conclusion

This paper presents a detailed and methodological approach to evaluating RAG systems through sub-question coverage, providing significant insights into both evaluation and potential enhancement strategies. By emphasizing core sub-question prioritization, this work contributes to a more sophisticated understanding of response quality, setting a foundation for future advancements in retrieval-augmented generation and broader AI research.