GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation (2404.03746v1)

Abstract: Query Reformulation(QR) is a set of techniques used to transform a user's original search query to a text that better aligns with the user's intent and improves their search experience. Recently, zero-shot QR has been shown to be a promising approach due to its ability to exploit knowledge inherent in LLMs. By taking inspiration from the success of ensemble prompting strategies which have benefited many tasks, we investigate if they can help improve query reformulation. In this context, we propose an ensemble based prompting technique, GenQREnsemble which leverages paraphrases of a zero-shot instruction to generate multiple sets of keywords ultimately improving retrieval performance. We further introduce its post-retrieval variant, GenQREnsembleRF to incorporate pseudo relevant feedback. On evaluations over four IR benchmarks, we find that GenQREnsemble generates better reformulations with relative nDCG@10 improvements up to 18% and MAP improvements upto 24% over the previous zero-shot state-of-art. On the MSMarco Passage Ranking task, GenQREnsembleRF shows relative gains of 5% MRR using pseudo-relevance feedback, and 9% nDCG@10 using relevant feedback documents.

Ensemble Prompting for Generative Query Reformulation in Zero-Shot LLMs

Introduction

The challenge of transforming a user's initial query into a more effective search string lies at the heart of enhancing search experiences. Traditionally, Query Reformulation (QR) has tackled this by incorporating additional terms or paraphrasing the query. Recent advancements hinge on zero-shot learning within LLMs to leverage inherent knowledge for QR without reliance on labeled examples. This work introduces GenQREnsemble, an ensemble-based approach to QR, utilizing multiple paraphrased instructions to generate varied keyword expansions. The paper further presents GenQREnsembleRF, a post-retrieval variant that incorporates pseudo relevance feedback (PRF), marking significant improvements over existing zero-shot QR techniques.

Ensemble Prompting for Query Reformulation

The principal innovation, GenQREnsemble, employs multiple paraphrased prompting instructions with a user's query to generate a diversified set of keyword expansions. Unlike single-instruction prompting, this ensemble strategy aims to encapsulate various interpretations and reformulations of the query, potentially uncovering a broader and more relevant set of keywords for search enhancement. This method significantly transcends the prevailing zero-shot QR benchmarks by delivering up to 18\% relative improvements in nDCG@10 and up to 24\% in MAP across four Information Retrieval (IR) benchmarks.

Post-retrieval Enhancement with Relevance Feedback

Building on the pre-retrieval ensemble strategy, GenQREnsembleRF introduces the concept of leveraging post-retrieval insights through relevance feedback. By incorporating context from either pseudo-relevance feedback or user-selected documents, this extension offers a nuanced and context-aware reformulation of queries, evidencing relative gains of 5\% in MRR and 9% in nDCG@10 on the MSMarco Passage Ranking task. These findings underscore the potential of employing relevance feedback within an ensemble prompting framework to refine search queries further.

Experimental Setup and Findings

The experiments were meticulously designed to evaluate the efficacy of GenQREnsemble and GenQREnsembleRF across various benchmarks and settings, including both sparse (BM25) and neural rankers (MonoT5). The ensemble approach consistently outperformed single-instruction methodologies, highlighting the advantage of diversifying the prompting instructions to encompass a broader lexical space for query expansion. Moreover, the introduction of relevance feedback in the post-retrieval setting further underscored the ensemble method's capability to adapt and refine query reformulations based on contextual insights.

Implications and Future Directions

This research elucidates the potential of ensemble prompting strategies in enhancing zero-shot QR, extending the utility of LLMs in search query reformulation. The significant performance improvements posited by GenQREnsemble and GenQREnsembleRF suggest a promising area for future explorations, particularly in operationalizing relevance feedback more dynamically and exploring ensemble methods for other IR tasks. Furthermore, the scalability and computational efficiency of such ensemble strategies warrant further scrutiny, especially as they become integrated into real-world search systems. The adaptable nature of ensemble prompting offers a conduit for future work to refine and optimize the balance between computational overhead and search enhancement benefits.

In conclusion, this investigation into ensemble prompting for generative QR delineates a promising avenue for leveraging the latent knowledge within LLMs more effectively. The implications for practical search applications are considerable, opening pathways to more nuanced and contextually aware search experiences. As the field advances, exploring the intersection between zero-shot learning, ensemble methodologies, and relevance feedback will undoubtedly yield novel insights and methodologies for enhancing information retrieval systems.