Generating Multi-Aspect Queries for Conversational Search (2403.19302v3)

Abstract: Conversational information seeking (CIS) systems aim to model the user's information need within the conversational context and retrieve the relevant information. One major approach to modeling the conversational context aims to rewrite the user utterance in the conversation to represent the information need independently. Recent work has shown the benefit of expanding the rewritten utterance with relevant terms. In this work, we hypothesize that breaking down the information of an utterance into multi-aspect rewritten queries can lead to more effective retrieval performance. This is more evident in more complex utterances that require gathering evidence from various information sources, where a single query rewrite or query representation cannot capture the complexity of the utterance. To test this hypothesis, we conduct extensive experiments on five widely used CIS datasets where we leverage LLMs to generate multi-aspect queries to represent the information need for each utterance in multiple query rewrites. We show that, for most of the utterances, the same retrieval model would perform better with more than one rewritten query by 85% in terms of nDCG@3. We further propose a multi-aspect query generation and retrieval framework, called MQ4CS. Our extensive experiments show that MQ4CS outperforms the state-of-the-art query rewriting methods. We make our code and our new dataset of generated multi-aspect queries publicly available.

• The paper introduces LLM-based AD, QD, and AQD methods that significantly boost conversational response retrieval performance.
• Experiments on the TREC iKAT dataset reveal that AQD, especially with GPT-4, outperforms traditional single-query rewriting techniques.
• The study underscores the potential for integrating user feedback and refined query generation to address complex information needs in conversational search.

Generate then Retrieve: Enhancing Conversational Response Retrieval with LLMs

Methods Overview

The paper introduces novel approaches to improve conversational response retrieval by leveraging LLMs. It identifies the main limitation of existing retrieval systems, which typically employ a single rewritten query for passage retrieval, failing to address complex information needs that require reasoning over multiple facts. To overcome this, the authors propose three methods:

1. Answer-driven Query Generation (AD): Utilizing the LLM's generated answer as a singular long query for retrieval.
2. Query Generation (QD): Prompting the LLM to directly generate multiple queries from the conversational context.
3. Answer and Query Generation (AQD): A two-step method where the LLM first generates an answer and then produces multiple queries to refine this answer.

An additional variant, AQDAnswer, re-ranks results based on predicted relevance to the LLM's generated response, aiming to improve the quality of retrieved passages. The paper compares these methods against standard approaches and evaluates them using LLMs including GPT-4 and Llama-2 in different settings.

Experimental Setup and Results

The experiments are conducted on the TREC Interactive Knowledge Assistance Track (iKAT) dataset, showcasing the complexity of conversational information seeking tasks. The proposed methods are evaluated against baselines that follow either generate-then-retrieval or retrieval-then-generate paradigms, using a variety of LLMs.

Results indicate that AQD and AD methods, particularly when utilizing GPT-4, significantly outperform the baselines. AQD shows superior performance over single-query rewriting approaches (QR) and even outpaces human-rewritten queries in certain metrics. Notably, AQDAnswer's re-ranking strategy based on the initial generated answer leads to further improvements, showcasing the potential of LLMs in enhancing retrieval through a nuanced understanding of the conversational context and the user's information need.

Implications and Future Work

This paper presents a significant shift towards utilizing the generative capabilities of LLMs for improving information retrieval in conversational systems. By demonstrating that multiple queries generated from LLMs' responses can lead to better retrieval outcomes, it opens up new avenues for research in conversational search systems. It also highlights the importance of leveraging LLMs not just for generating responses but as integral components of the information retrieval process.

One promising direction for future work is exploring the optimal number of queries to generate and the impact of query quality on retrieval effectiveness. Additionally, integrating user feedback into the generative process could further personalize and refine the retrieval outcomes, making the conversational system more responsive to the user's specific needs.

Ethical Considerations and Limitations

The reliance on LLMs introduces potential biases and errors inherent in these models, which can affect the quality of generated responses and queries. Moreover, the effectiveness of the proposed methods is contingent upon the quality of the LLM's initial response, highlighting a dependency that could be problematic if the LLM fails to understand the user's request accurately. Future research should address these challenges, ensuring that conversational systems remain reliable, unbiased, and user-centric in their approach to information retrieval.