Improving Pinterest Search Relevance Using Large Language Models (2410.17152v1)

Abstract: To improve relevance scoring on Pinterest Search, we integrate LLMs into our search relevance model, leveraging carefully designed text representations to predict the relevance of Pins effectively. Our approach uses search queries alongside content representations that include captions extracted from a generative visual LLM. These are further enriched with link-based text data, historically high-quality engaged queries, user-curated boards, Pin titles and Pin descriptions, creating robust models for predicting search relevance. We use a semi-supervised learning approach to efficiently scale up the amount of training data, expanding beyond the expensive human labeled data available. By utilizing multilingual LLMs, our system extends training data to include unseen languages and domains, despite initial data and annotator expertise being confined to English. Furthermore, we distill from the LLM-based model into real-time servable model architectures and features. We provide comprehensive offline experimental validation for our proposed techniques and demonstrate the gains achieved through the final deployed system at scale.

• The paper leverages a cross-encoder LLM approach to achieve a 19.7% accuracy boost in query-to-Pin matching.
• The study implements semi-supervised learning and knowledge distillation to create cost-efficient, real-time models handling multilingual and visual data.
• Online A/B testing validates the approach by realizing over a 1% lift in search feed relevance and greater than 1.5% improvement in search fulfillment rates.

Improving Pinterest Search Relevance Using LLMs

The paper "Improving Pinterest Search Relevance Using LLMs" explores enhancing search relevance on Pinterest by integrating LLMs into existing frameworks. The authors address the complexities of visual and multilingual data while scaling up cost-efficiently.

Overview of Methodology

The researchers propose a multi-faceted approach leveraging LLMs to predict the relevance of search results on Pinterest. They utilize LLMs to evaluate query-Pin relationships through a cross-encoder architecture. The model uses both traditional text features and enriched metadata, such as user-curated board titles and high-engagement query tokens. The approach includes the innovative use of synthetic image captions generated by visual LLMs to improve the text data associated with Pins.

For training, a semi-supervised learning technique is implemented to transcend the limitations of human-annotated data, allowing the model to handle new languages and domains. Here, the knowledge distillation process plays a crucial role by transferring insights from a computationally intense LLM to a lightweight, servable student model appropriate for real-time applications.

Results

The paper provides comprehensive offline and online validation, demonstrating significant improvements in relevance prediction. The Llama-3-8B model, for instance, improves relevance prediction accuracy by 19.7% compared to baseline models. Enriched text features contributed substantially to these results by providing comprehensive representations of Pins.

Moreover, training the student model with distilled labels from the teacher model led to enhanced performance, as evidenced by the increase in accuracy with larger training datasets. The online A/B testing further corroborates these findings, showcasing a tangible improvement of over 1% in search feed relevance and a >1.5% increase in search fulfiLLMent rates.

Implications and Future Directions

The paper's approach sets a precedent for applying LLMs in recommendation systems, highlighting the practicality of enhancing search relevance with enriched text data and multilingual capabilities. This integration not only improves the user experience but also supports broader engagement across different languages and cultural contexts.

Looking forward, several promising avenues for further research exist. The exploration of serveable LLMs presents an opportunity to optimize latency and cost without sacrificing accuracy. Additionally, the incorporation of vision-and-language multimodal models could refine relevance predictions by integrating visual data more effectively.

The authors also highlight active learning as a future direction. This could dynamically enrich datasets, allowing models to adapt more rapidly to evolving content trends and user expectations.

In summary, this paper provides a detailed examination of leveraging LLMs for improved search relevance in a large-scale, multilingual, and visual discovery platform like Pinterest. The findings and methodologies posited here could inform the future design of recommendation systems in similar contexts.