AGRaME: Any-Granularity Ranking with Multi-Vector Embeddings (2405.15028v1)

Abstract: Ranking is a fundamental and popular problem in search. However, existing ranking algorithms usually restrict the granularity of ranking to full passages or require a specific dense index for each desired level of granularity. Such lack of flexibility in granularity negatively affects many applications that can benefit from more granular ranking, such as sentence-level ranking for open-domain question-answering, or proposition-level ranking for attribution. In this work, we introduce the idea of any-granularity ranking, which leverages multi-vector embeddings to rank at varying levels of granularity while maintaining encoding at a single (coarser) level of granularity. We propose a multi-granular contrastive loss for training multi-vector approaches, and validate its utility with both sentences and propositions as ranking units. Finally, we demonstrate the application of proposition-level ranking to post-hoc citation addition in retrieval-augmented generation, surpassing the performance of prompt-driven citation generation.

• The paper introduces AGRaME, a method that leverages multi-vector embeddings to enable ranking at any desired granularity without specialized encoders.
• It employs a multi-granular contrastive loss to significantly enhance both sentence-level and proposition-level ranking performance in various applications.
• The approach supports efficient post-hoc citation addition, outperforming traditional methods in retrieval-augmented generation tasks.

Overview of Any-Granularity Ranking via Multi-Vector Embeddings

The paper presents a novel method, AGRaME (Any-Granularity Ranking with Multi-vector Embeddings), which aims to address the inflexibility in granularity in traditional ranking systems. Standard ranking algorithms often operate at a singular level of granularity, typically at the passage level, which can limit their effectiveness in various applications such as sentence-level ranking in open-domain question-answering and proposition-level ranking for attribution. AGRaME leverages multi-vector embeddings to allow for ranking at varying levels of granularity while maintaining encoding at a single, coarser level.

Key Contributions

1. Introduction of AGRaME: The method enables ranking at different granularities using a common encoding level. This flexibility is achieved without the necessity for specialized encoders for each granularity level, a significant departure from existing methodologies.
2. Multi-Granular Contrastive Loss: The paper proposes a novel training loss that incorporates multi-granular contrastive loss to improve fine-grained ranking capability. This technique enhances the ability of multi-vector approaches to rank sentences even when the encoding is performed at the passage level.
3. Superior Proposition-Level Ranking: Demonstrating significant performance improvements in proposition-level ranking, AGRaME outperforms state-of-the-art methods, showcasing its potential in applications requiring fine-grained attribution.
4. Post-Hoc Citation Addition: The proposed system includes the PropCite method, which uses proposition-level ranking to efficiently add citations to generated text post-hoc. This methodology surpasses traditional prompt-driven citation generation, providing a robust solution for citation in retrieval-augmented generation applications.

Experimental Insights

Motivating Experiments: Initial experiments used ColBERTv2, a popular multi-vector model, to explore the performance of sentence ranking when encoded at the passage level. Results highlighted a notable drop in performance, sparking the development of multi-granular contrastive loss to address this issue.

Improved Sentence-Level Ranking: By incorporating sentence-level relevance supervision during training and using distinct query markers to signal granularity, AGRaME significantly improves sentence-level ranking performance. This improvement is evident even in cross-domain evaluations, indicating the robustness and generalizability of the approach.

Proposition-Level Ranking: Experimental results in the PropSegmEnt dataset demonstrate that AGRaME excels in proposition-level ranking tasks, establishing it as a state-of-the-art method in this domain.

Practical and Theoretical Implications

Practically, AGRaME has significant implications for applications that require fine granularity in ranking, such as question answering systems and content attribution. The flexibility to rank at any granularity level without the need for specialized encoding models simplifies the implementation and potentially reduces computational overhead.

Theoretically, AGRaME's approach to using multi-vector embeddings for varying granularity levels opens new avenues for research in ranking and retrieval systems. The multi-granular contrastive loss introduced offers a new perspective on how contrastive learning can be applied to improve fine-grained ranking capabilities.

Future Directions

Future research can explore extending the multi-granular contrastive loss to other domains and applications that benefit from fine-grained ranking. Additionally, investigating the scalability of AGRaME in more extensive datasets and real-time applications could further enhance its utility. Potential improvements in the PropCite method could involve integrating dynamically adaptive thresholding mechanisms to balance precision and recall more effectively across different domains and use cases.

In summary, AGRaME presents a compelling solution to the challenges of granularity in ranking systems. By leveraging multi-vector embeddings and innovative training techniques, it offers enhanced flexibility and performance in various ranking tasks, setting a new benchmark in the field of information retrieval.