Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation (2407.01158v2)

Abstract: Interactions with LLMs often yield long and detailed responses, leveraging both parametric knowledge and retrieval-augmented generation (RAG). While these responses can provide rich insights, they often include redundant or less engaging content not aligned with user interests. This issue becomes apparent when users specify particular subtopics to include or exclude -- termed coverage-conditioned ($C^2$) queries -- as LLMs often struggle to provide tailored responses. To address this challenge, we investigate the role of query outlines, sequences of subqueries designed to guide LLMs in generating responses that meet specific user requirements. To systematically create and evaluate these outlines, we introduce QTree, a dataset of 10K hierarchical sets of information-seeking subqueries that define structured boundaries for outline creation and evaluation in $C^2$ scenarios. Additionally, we develop QPlanner, a 7B LLM trained to generate customized outlines within boundaries of QTree. We evaluate the effectiveness of the generated outlines through automatic and human judgements, focusing on their impact within retrieval-augmented generation (RAG) systems. Experimental results demonstrate that QPlanner, especially when trained with alignment techniques like DPO, generates higher-quality outlines that better fulfill diverse user needs.

• The paper introduces QTree, a hierarchical query structure, and QPlanner, a 7B autoregressive model to generate refined query outlines for complex requests.
• It employs a dual training method using supervised fine-tuning and direct preference optimization to bolster retrieval-augmented generation performance.
• Automatic and human evaluations demonstrate that preference-aligned QPlanner significantly improves query relevance and content quality compared to standard RAG setups.

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

"Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation" investigates a novel methodology to enhance the generation of long-form responses from LLMs by leveraging query outlines. The paper introduces the concept of coverage-conditioned (C²) queries and proposes a framework, consisting of QTree and QPlanner, aimed at refining the ability of LLMs to handle complex, user-specific informational requests.

Key Contributions

1. QTree Construction: The authors develop QTree, a hierarchical set of 10,000 decomposable queries derived from various datasets, representing diverse perspectives on specific topics. This hierarchical structure allows for a nuanced exploration of user queries.
2. QPlanner Model: They introduce QPlanner, a 7B autoregressive LLM designed to generate query outlines tailored to C² scenarios. These outlines are expected to increase the efficacy of retrieval-augmented generation (RAG) systems.
3. Evaluation Framework: The effectiveness of QPlanner-generated outlines is rigorously evaluated through both automatic and human evaluations. The authors demonstrate that QPlanner, particularly when fine-tuned with preference alignment, significantly improves the quality of generated outlines and the relevance of the final long-form responses.

Framework Details

Query Outlining and QTree

The primary objective of QTree is to manage the complexity inherent in long-form content generation by creating a structured representation of possible subtopics. This hierarchical approach includes three levels of depth and three branches at each level, ensuring a comprehensive mapping of potential subtopics related to a user's initial query (base query).

Coverage Query (q_cov)

To simulate C² scenarios, the paper makes use of background queries (randomly selected subtopics from QTree) and intent operations (Inclusion and Exclusion operations). This enables the generation of q_cov, which adds specific constraints to the base query, thereby crafting more refined user requests. Five potential q_cov queries are generated per base query, ensuring one that meets the C² requirements.

QPlanner Training

QPlanner aims to generate outlines that not only align with users’ informational needs but also improve subsequent document retrieval and RAG tasks. The authors employ both supervised fine-tuning (SFT) and direct preference optimization (DPO) stages:

• SFT: Trained on 31,488 C² queries and corresponding outlines.
• DPO: Further refined using preference-aligned data to optimize outline quality and model performance.

Experimental Results

Automatic and Human Evaluations

Automatic Evaluation: Performance measured using the five-point Likert scale indicates that DPO-aligned QPlanner provides the best results with a mean score of 3.16 and the lowest standard deviation, indicating robustness.

Human Evaluation:

• Outline Evaluation: Human evaluators confirmed the superior performance of the DPO-QPlanner model, giving it higher average ratings (3.29 out of 5) compared to the SFT model (3.03 out of 5), and exhibiting positive correlations with automatic evaluations.
• Response Evaluation: Two studies highlighted QPlanner’s dual role in improving document retrieval and content drafting.
  • Search Query Performance: QPlanner significantly enhances document relevance compared to a vanilla RAG setup.
  • Content Draft Quality: Preference-aligned outlines generated by QPlanner lead to more preferred long-form responses.

Implications and Future Work

This work has dual practical and theoretical implications:

• Practical: The proposed QPlanner can be integrated into existing LLM frameworks to improve the quality of long-form responses, particularly under complex user-specific scenarios.
• Theoretical: By leveraging hierarchical query structures and aligning model outputs with user preferences, this work advances our understanding of effective user interaction with AI models in information retrieval contexts.

Future developments may involve adapting the number and complexity of queries within an outline to suit more varied and dynamic informational needs. Furthermore, the introduction of advanced fact verification mechanisms could augment the reliability of RAG responses, thereby enhancing their applicability in real-world scenarios where accuracy and specificity are paramount.