Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (2410.02642v1)

Abstract: Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, LLMs have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more directly leverage such signals, we propose in-context re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Overview of "Attention in LLMs Yields Efficient Zero-shot Re-rankers"

In "Attention in LLMs Yields Efficient Zero-shot Re-rankers," the authors investigate the potential of leveraging attention mechanisms within LLMs to create more efficient zero-shot re-ranking methods for information retrieval (IR) systems. Traditional LLM-based re-ranking approaches have relied heavily on the generative capabilities of these models. Such methods typically demand multiple costly forward passes, making them inefficient for broader application with open-weight models. The authors propose a novel in-context re-ranking (ICR) method aiming to circumvent these inefficiencies.

Key Contributions

1. In-context Re-Ranking (ICR) Methodology: The paper introduces ICR, which uses changes in the attention patterns of LLMs in response to a search query to re-rank documents efficiently. This approach negates the necessity for autoregressive generation by directly leveraging attention signals, allowing for only two forward passes, or $O(1)$, instead of the conventional $O(N)$ passes required by generative methods.
2. Calibration with a Content-free Query: To mitigate biases inherent in LLMs, the authors propose using a content-free query to calibrate re-ranking scores, isolating the relevance signal from undesired biases.
3. Application to Open-weight LLMs: ICR's design ensures its applicability across LLMs without specialized fine-tuning, offering a significant advantage over generative methods that often require proprietary models.

Experimental Evaluation

The methodology is validated through extensive experiments across single-hop and multi-hop retrieval benchmarks using two open-weight LLMs (Mistral 7B and Llama-3.1 8B). The results exhibit the following:

• Performance on Single-hop Tasks: ICR outperforms RankGPT, specifically when utilizing Llama-3.1 8B, improving results on nine datasets from the BEIR benchmark and demonstrating its effectiveness in processing complex re-ranking signals requiring deeper contextual understanding.
• Efficacy in Multi-hop Settings: ICR shows superior performance in multi-hop retrieval tasks, emphasizing its ability to integrate information across multiple documents more effectively than other methods.
• Efficiency Gains: The experiments reveal ICR's substantial reduction in re-ranking latency by over 60%, highlighting its capability to offer competitive performance at a reduced computational cost.

Implications and Future Directions

The ICR method's enhanced ability to size attention patterns repositions LLMs beyond conventional generative roles, unveiling a novel means of exploiting these models' capabilities. The attentional insights within LLMs that ICR leverages could foster new advancements in IR efficiency and effectiveness, particularly important in contexts requiring fine-grained document ranking without incurring significant computational overhead.

Future research can explore further refinements in attention-based ranking methods, extending them across diverse domains and applications in AI. The potential expansion of ICR to incorporate other LLM architectures, including encoder-decoder models, presents another intriguing direction, as does the examination of more sophisticated calibration strategies to further enhance model robustness.

Overall, the paper opens up new avenues for utilizing LLMs in efficient, non-generative applications, providing a compelling alternative to existing methods in information retrieval tasks.