Relevance Filtering for Embedding-based Retrieval (2408.04887v1)

Abstract: In embedding-based retrieval, Approximate Nearest Neighbor (ANN) search enables efficient retrieval of similar items from large-scale datasets. While maximizing recall of relevant items is usually the goal of retrieval systems, a low precision may lead to a poor search experience. Unlike lexical retrieval, which inherently limits the size of the retrieved set through keyword matching, dense retrieval via ANN search has no natural cutoff. Moreover, the cosine similarity scores of embedding vectors are often optimized via contrastive or ranking losses, which make them difficult to interpret. Consequently, relying on top-K or cosine-similarity cutoff is often insufficient to filter out irrelevant results effectively. This issue is prominent in product search, where the number of relevant products is often small. This paper introduces a novel relevance filtering component (called "Cosine Adapter") for embedding-based retrieval to address this challenge. Our approach maps raw cosine similarity scores to interpretable scores using a query-dependent mapping function. We then apply a global threshold on the mapped scores to filter out irrelevant results. We are able to significantly increase the precision of the retrieved set, at the expense of a small loss of recall. The effectiveness of our approach is demonstrated through experiments on both public MS MARCO dataset and internal Walmart product search data. Furthermore, online A/B testing on the Walmart site validates the practical value of our approach in real-world e-commerce settings.

• The paper introduces the Cosine Adapter that transforms raw cosine similarities into interpretable relevance scores to significantly enhance retrieval precision.
• It employs a dual encoder framework with various monotonic transformation functions, validated on MS MARCO and Walmart datasets with improved PR AUC and P@R95 metrics.
• Its real-world integration in Walmart’s system achieved a 5.34% increase in top-5 precision, demonstrating practical benefits in large-scale e-commerce search.

Relevance Filtering for Embedding-based Retrieval

Nicholas Rossi et al. present an insightful paper on enhancing the precision of embedding-based retrieval systems through a relevance filtering mechanism. The proposed approach, termed "Cosine Adapter," addresses the significant challenge of precision faced in dense retrieval scenarios where Approximate Nearest Neighbor (ANN) search forms the backbone of retrieval systems. The inherent challenge in dense retrieval is its deficiency in a natural cutoff mechanism compared to lexical retrieval, which often results in a compromised precision due to the return of irrelevant documents.

Summary of Contributions

The paper introduces a novel component that transforms raw cosine similarity scores from embedding-based retrieval systems into interpretable and query-dependent scores. These transformed scores facilitate the application of a global threshold that effectively filters out irrelevant results, thereby improving precision with minimal loss in recall.

Key contributions include:

1. Cosine Adapter: A neural network component that generates query-dependent parameters for transforming cosine similarity scores into relevance scores.
2. Transformation Functions: Several monotonic transformation functions (linear, square root, quadratic, power) are employed to preserve ranking while facilitating score interpretation.
3. Experimental Validation: Extensive experimental evaluation on both the public MS MARCO dataset and proprietary Walmart product search data. The results demonstrate significant improvements in precision across various metrics, including PR AUC and P@R95.
4. Real-world Application: The integration of the Cosine Adapter into Walmart's production retrieval system, validated through online A/B testing, showcasing practical utility in a large-scale e-commerce environment.

Methodology

The Cosine Adapter model leverages a dual encoder framework consisting of two distinct encoders for queries and candidates, which convert inputs into dense vectors. The relevance of a candidate is gauged by the cosine similarity of its embedding to the query embedding. By adopting transformation functions, the Cosine Adapter converts these similarity scores into more interpretable and uniform relevance scores across queries.

Specifically, the Cosine Adapter is trained using query embeddings and a range of transformation functions:

• Linear
• Square Root
• Quadratic
• Power

These functions are parameterized based on robust neural networks that are able to draw upon the semantic information encapsulated in the dense vector representations of queries.

Experimental Evaluation

Experiments were conducted on the MS MARCO dataset and a proprietary Walmart dataset to assess the impact of the Cosine Adapter on retrieval precision. The MS MARCO experiments indicated significant improvements in PR AUC and P@R95 metrics, particularly when applying the power transformation function. For example, the Cosine Adapter applied to the listwise-loss-trained model achieved an impressive PR AUC of 0.8619 and P@R95 of 0.6225.

Furthermore, real-world implications were demonstrated through integration into Walmart’s search system. A/B testing results highlighted an improvement in the precision of top results (e.g., a 5.34% increase for top 5 results), leading to a more refined user search experience without adversely impacting engagement metrics such as the number of orders and GMV.

Implications and Future Work

The implications of this research are twofold: practical and theoretical. Practically, the Cosine Adapter can be readily integrated into existing large-scale search systems to enhance user satisfaction through improved precision. Theoretically, it opens up new avenues for research into query-dependent relevance mapping and further optimization of dense retrieval systems.

Future research could explore more advanced machine learning techniques for even finer granularity in the query relevance mapping. Further, extending this framework to different models and diverse datasets could offer deeper insights into its efficacy and robustness.

In conclusion, the introduction of the Cosine Adapter for relevance filtering in embedding-based retrieval systems represents a critical advancement in the pursuit of balancing recall and precision in dense retrieval environments. Its demonstrable improvements in both experimental and real-world settings underscore its potential for broad application across various domains within AI and information retrieval.