Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning (2006.02282v3)

Abstract: Nowadays e-commerce search has become an integral part of many people's shopping routines. Two critical challenges stay in today's e-commerce search: how to retrieve items that are semantically relevant but not exact matching to query terms, and how to retrieve items that are more personalized to different users for the same search query. In this paper, we present a novel approach called DPSR, which stands for Deep Personalized and Semantic Retrieval, to tackle this problem. Explicitly, we share our design decisions on how to architect a retrieval system so as to serve industry-scale traffic efficiently and how to train a model so as to learn query and item semantics accurately. Based on offline evaluations and online A/B test with live traffics, we show that DPSR model outperforms existing models, and DPSR system can retrieve more personalized and semantically relevant items to significantly improve users' search experience by +1.29% conversion rate, especially for long tail queries by +10.03%. As a result, our DPSR system has been successfully deployed into JD.com's search production since 2019.

• The paper introduces DPSR, a deep learning system that improves e-commerce search by effectively modeling user intent and semantic relevance.
• It employs a two-tower neural network with an attention-based loss function and leverages extensive click log data for robust candidate retrieval and ranking.
• Online evaluations at JD.com demonstrate significant improvements in conversion rates for long-tail queries, validating the system’s practical impact.

Overview of DPSR System

The JD.com research team has developed an innovative solution, designated as DPSR (Deep Personalized and Semantic Retrieval), which addresses significant e-commerce search challenges. The main hurdles faced in current search environments are twofold: retrieving items that may not exactly match the search query terms but are semantically relevant and tailoring search results to the unique profiles of individual users.

System Design

DPSR is architected to effectively serve large-scale e-commerce traffic and is trained to learn item and query semantics with high accuracy. The system comprises three fundamental components: a query processing unit that interprets user queries, a candidate retrieval mechanism using sophisticated indexing, and a ranking system sorting these candidates based on various metrics.

The focus of the team's effort lies in enhancing the candidate retrieval stage, with the aim of delivering more personalized and semantically relevant search results. This is particularly crucial for e-commerce platforms like JD.com, which need precise and individualized search functionalities to serve millions of active users and process transactions amounting to billions of dollars.

Methodology and Innovations

The researchers developed a neural network model utilizing a two-tower architecture, multi-head query tower design, attention-based loss function, and a negative sampling approach. This model is trained using significant human supervision and large-scale click log data from the platform. For implementation, the team significantly customized TensorFlow, a popular machine learning framework, to ensure consistency in online and offline scenarios, optimize input data storage, and facilitate scalable distributed training.

Results and Deployment

The DPSR system has been methodically evaluated both offline and online. In offline experiments, the system demonstrated superior capabilities to retrieve semantically relevant items. It particularly excelled in handling long-tail queries, which pose a challenge for traditional retrieval systems. Online A/B tests conducted in live environments showcased that DPSR could improve the e-commerce platform's search experience, leading to a marked increase in conversion rates for long-tail queries.

Given the successful deployment of DPSR within JD.com’s search functionality since 2019, the solution presents an advanced example of neural network-based retrieval in action. It stands as an embodiment of cutting-edge e-commerce search technology, capable of nuanced semantic understanding and individual user catering.