Generate rather than Retrieve: Large Language Models are Strong Context Generators (2209.10063v3)

Abstract: Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with LLM generators. We call our method generate-then-read (GenRead), which first prompts a LLM to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.

Generate Rather than Retrieve: LLMs as Context Generators

The paper "Generate rather than Retrieve: LLMs are Strong Context Generators" proposes a novel approach for addressing knowledge-intensive tasks like open-domain question answering (QA), fact checking, and dialogue systems. Traditionally, these tasks require a retrieve-then-read pipeline where relevant contextual documents are retrieved from a large corpus before being processed for answers. The authors introduce an alternative method named "Generate-Then-Read" (GenRead) that utilizes LLMs as context generators rather than relying on document retrieval.

Key Contributions

1. Generate-Then-Read Approach: The GenRead method replaces the conventional document retrieval with generation, using LLMs to create contextual documents directly from the input question. The generated documents are then read to produce a final answer. This shift aims to offset the limitations of retrieval-based methods, such as retrieving irrelevant information or shallow interaction in dense retrieval models.
2. Clustering-Based Prompting: To enhance recall performance and generate varied contexts, the paper presents a novel clustering-based prompting technique. By selecting diverse prompts through clustering, the method generates multiple document perspectives for better recall of acceptable answers.
3. Significant Improvement in Open-Domain QA: The experimental results of the GenRead approach are remarkable. Without retrieving any documents from external sources, it achieves exact match scores of 71.6 and 54.4 on the TriviaQA and WebQ datasets, respectively. This performance marks a significant improvement over state-of-the-art retrieve-then-read methods like DPR-FiD.
4. Extensive Evaluation: The method is extensively tested across different tasks demonstrating its flexibility and effectiveness not only in zero-shot settings but also in supervised settings. GenRead shows promising results across various benchmarks.

Implications for Future AI Developments

The implications of using LLMs as generators are profound. This approach could redefine how knowledge-intensive tasks are structured and solved. It capitalizes on the inherent strengths of LLMs, such as their vast stored knowledge and depth of language understanding. The paper indicates that blending retrieval and generation might further enhance model performance, suggesting a hybrid approach as a potential area of exploration.

Practical and Theoretical Implications

Practically, the use of LLMs like InstructGPT for context generation could simplify pipeline architectures and reduce dependency on large-scale external corpora. This reduction in complexity has potential benefits in terms of computational efficiency and model scalability. Theoretically, it opens up new research avenues into understanding and leveraging the generative capabilities of LLMs for diverse NLP tasks.

Conclusion

By proposing the use of LLMs for generating relevant contexts, the paper challenges traditional approaches to knowledge-intensive tasks. Its comprehensive experimentation and promising results underscore the potential of generation over retrieval in certain settings. The hybridization of these methods could be key to unlocking further advancements in AI-driven knowledge tasks. The GenRead approach triumphs in illustrating that the future of intelligent systems may well be characterized by what they can generate, not just what they can retrieve.