PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2404.18424v3)

Abstract: Utilizing LLMs for zero-shot document ranking is done in one of two ways: (1) prompt-based re-ranking methods, which require no further training but are only feasible for re-ranking a handful of candidate documents due to computational costs; and (2) unsupervised contrastive trained dense retrieval methods, which can retrieve relevant documents from the entire corpus but require a large amount of paired text data for contrastive training. In this paper, we propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. Our method only requires prompts to guide an LLM to generate query and document representations for effective document retrieval. Specifically, we prompt the LLMs to represent a given text using a single word, and then use the last token's hidden states and the corresponding logits associated with the prediction of the next token to construct a hybrid document retrieval system. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations given by the LLM. Our experimental evaluation on the MSMARCO, TREC deep learning and BEIR zero-shot document retrieval datasets illustrates that this simple prompt-based LLM retrieval method can achieve a similar or higher retrieval effectiveness than state-of-the-art LLM embedding methods that are trained with large amounts of unsupervised data, especially when using a larger LLM.

Prompting LLMs for Dual Dense and Sparse Representations in Zero-Shot Document Retrieval

Introduction

The efficacious prompting of LLMs for text generation and natural language understanding underscores their potential in zero-shot document retrieval tasks. Prior approaches utilized prompt-based LLMs for re-ranking small document subsets or leveraged unsupervised methods to prepare LLMs for dense retrieval roles. However, these methods either face scalability issues due to high computational costs or necessitate extensive unsupervised training.

Innovation in Document Retrieval Methodology

The work introduced in the paper, known as "PromptReps," represents a significant advancement in the application of LLMs to document retrieval tasks. This method ingeniously combines the generation of both dense and sparse text representations from a single LLM prompt, exploiting the model's ability to generate embeddings directly suitable for both retrieval paradigms.

• Dense Representation: Extracted from the hidden states of the last token in response to the input prompt.
• Sparse Representation: Derived from the logits output predicting the subsequent token after the input prompt.

This dual representation enables the efficient indexing of documents which can then be applied to either or both dense and sparse retrieval strategies. An experiment conducted on the BEIR benchmark showcases that PromptReps, especially when using larger LLMs, achieves favorable results compared to state-of-the-art methods that rely on extensive training.

Core Methodology

PromptReps uses minimal prompts to guide LLMs in generating text representations that are immediately usable for constructing search indices. In practical terms, it involves:

1. Prompting an LLM to encapsulate a text into a single word representation.
2. Extracting the last token’s hidden states as a dense embedding and using the logits for a sparse, bag-of-words representation.

Experimental Evaluation

The empirical evaluation of PromptReps illustrated its robustness across varied datasets included in the BEIR benchmark. Results indicate that:

• The method achieves similar or superior retrieval effectiveness compared to LLM-based methods reliant on heavy contrastive pre-training.
• Larger LLM sizes consistently yield better retrieval results, emphasizing the scalability benefits of PromptReps.

Theoretical and Practical Implications

Theoretically, the success of PromptReps suggests that the inherent capabilities of LLMs can be more fully utilized without the need for additional training, through effective prompt design. Practically, this method offers a viable solution for large-scale information retrieval systems where traditional training approaches are either impractical or too costly.

Future Directions

Given the effectiveness of the zero-shot methodology proposed in PromptReps, future research could explore:

• Adaptation of the technique to other forms of semantic search tasks.
• Optimization of prompt structures to enhance the quality of text representations for specific retrieval tasks.
• Examination of the trade-offs between retrieval quality and computational efficiency, particularly in online search systems.

Conclusion

PromptReps demonstrates a novel use of LLMs in document retrieval, exploiting their intrinsic generation capabilities to produce useful dense and sparse text representations through simple prompting. This method opens avenues for further research into efficient, scalable, and training-free retrieval systems leveraging the raw power of pre-trained LLMs.