Dice Question Streamline Icon: https://streamlinehq.com

Effectiveness of EOL Prompts for Information Retrieval

Determine whether explicit one-word limitation (EOL) prompts used to elicit sentence embeddings from large language models yield effective retrieval performance on information retrieval datasets with large document corpora.

Information Square Streamline Icon: https://streamlinehq.com

Background

Prior work on prompting LLMs to generate sentence embeddings has introduced Explicit One-word Limitation (EOL) prompts, which instruct models to represent a sentence with a single word. These approaches have primarily been evaluated on semantic textual similarity (STS) benchmarks.

It remains uncertain whether these EOL-based sentence embeddings are effective when applied to information retrieval tasks that involve large document corpora, motivating an explicit question about their performance in such retrieval settings.

References

However, these works only evaluate such prompts on STS datasets, and their effectiveness on information retrieval datasets with large document corpora is unknown.

— PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2404.18424 - Zhuang et al., 29 Apr 2024) in Section 2.4 (Prompting LLM for sentence embeddings)