Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search (2404.15675v2)

Published 24 Apr 2024 in cs.IR

Abstract: Leveraging generative retrieval (GR) techniques to enhance search systems is an emerging methodology that has shown promising results in recent years. In GR, a text-to-text model maps string queries directly to relevant document identifiers (docIDs), dramatically simplifying the retrieval process. However, when applying most GR models in large-scale E-commerce for personalized item search, we must face two key problems in encoding and decoding. (1) Existing docID generation methods ignore the encoding of efficiency information, which is critical in E-commerce. (2) The positional information is important in decoding docIDs, while prior studies have not adequately discriminated the significance of positional information or well exploited the inherent interrelation among these positions. To overcome these problems, we introduce an efficient Hierarchical encoding-decoding Generative retrieval method (Hi-Gen) for large-scale personalized E-commerce search systems. Specifically, we first design a representation learning model using metric learning to learn discriminative feature representations of items to capture semantic relevance and efficiency information. Then, we propose a category-guided hierarchical clustering scheme that makes full use of the semantic and efficiency information of items to facilitate docID generation. Finally, we design a position-aware loss to discriminate the importance of positions and mine the inherent interrelation between different tokens at the same position. This loss boosts the performance of the LLM used in the decoding stage. Besides, we propose two variants of Hi-Gen (Hi-Gen-I2I and Hi-Gen-Cluster) to support online real-time large-scale recall in the online serving process. Hi-Gen gets 3.30% and 4.62% improvements over SOTA for Recall@1 on the public and industry datasets, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Information Processing Systems 35 (2022), 31668–31683.
  2. Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual Web search engine. In Proceedings of the Seventh International Conference on World Wide Web 7. Elsevier Science Publishers B. V., NLD, 107–117.
  3. Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.10687 (2019).
  4. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  5. Challenges and Advances in Generative Information Retrieval. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum). 57–66.
  6. AOL4PS: A Large-Scale Dataset for Personalized Search. Data Intelligence 3 (08 2021), 1–17. https://doi.org/10.1162/dint_a_00104
  7. Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3. Springer, 84–92.
  8. How to Index Item IDs for Recommendation Foundation Models. arXiv preprint arXiv:2305.06569 (2023).
  9. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020).
  10. Mahmut Kaya and Hasan Şakir Bilge. 2019. Deep metric learning: A survey. Symmetry 11, 9 (2019), 1066.
  11. IncDSI: incrementally updatable document retrieval. In International Conference on Machine Learning. PMLR, 17122–17134.
  12. Brian Kulis et al. 2013. Metric learning: A survey. Foundations and Trends® in Machine Learning 5, 4 (2013), 287–364.
  13. Generative retrieval for long sequences. arXiv preprint arXiv:2204.13596 (2022).
  14. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019).
  15. PersonalTM: Transformer memory for personalized retrieval. (2023).
  16. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing 7, 1 (2003), 76–80. https://doi.org/10.1109/MIC.2003.1167344
  17. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing 7 (2003), 76–80.
  18. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs.CL]
  19. DSI++: Updating Transformer Memory with New Documents. arXiv preprint arXiv:2212.09744 (2022).
  20. A Fast and Flexible Algorithm for Solving the Lasso in Large-scale and Ultrahigh-dimensional Problems. bioRxiv (2019). https://doi.org/10.1101/630079
  21. Language Models are Unsupervised Multitask Learners. (2019).
  22. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  23. Recommender Systems with Generative Retrieval. arXiv preprint arXiv:2305.05065 (2023).
  24. LaMP: When Large Language Models Meet Personalization. arXiv preprint arXiv:2304.11406 (2023).
  25. Learning to Tokenize for Generative Retrieval. arXiv preprint arXiv:2304.04171 (2023).
  26. Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (, Long Beach, CA, USA,) (KDD ’23). Association for Computing Machinery, 4904–4913.
  27. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems 35 (2022), 21831–21843.
  28. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 839–848.
  29. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems 35 (2022), 25600–25614.
  30. Liu Yang and Rong Jin. 2006. Distance metric learning: A comprehensive survey. Michigan State Universiy 2, 2 (2006), 4.
  31. Large Scale Product Graph Construction for Recommendation in E-commerce. ArXiv abs/2010.05525 (2020). https://api.semanticscholar.org/CorpusID:222291351
  32. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320 (2023).
  33. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063 (2022).
  34. A dual augmented two-tower model for online large-scale recommendation. DLP-KDD (2021).
  35. A Personalized Dense Retrieval Framework for Unified Information Access.
  36. Towards personalized and semantic retrieval: An end-to-end solution for e-commerce search via embedding learning. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2407–2416.
  37. Dense text retrieval based on pretrained language models: A survey. arXiv preprint arXiv:2211.14876 (2022).
  38. Guoqing Zheng and Jamie Callan. 2015. Learning to reweight terms with distributed representations. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 575–584.
  39. Encoding history with context-aware representation learning for personalized search. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 1111–1120.
  40. Ultron: An ultimate retriever on corpus with a model-based indexer. arXiv preprint arXiv:2208.09257 (2022).
  41. DynamicRetriever: A Pre-trained Model-based IR System Without an Explicit Index. Machine Intelligence Research 20, 2 (2023), 276–288.
  42. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206.10128 (2022).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yanjing Wu (1 paper)
  2. Yinfu Feng (2 papers)
  3. Jian Wang (967 papers)
  4. Wenji Zhou (1 paper)
  5. Yunan Ye (3 papers)
  6. Rong Xiao (44 papers)
  7. Jun Xiao (134 papers)

Summary

Enhancing Large-Scale Personalized E-commerce Search with Hierarchical Generative Retrieval

Introduction

Recent advancements in Generative Retrieval (GR) techniques have shown promising directions for enhancing search systems, but applying these methods to large-scale E-commerce search systems introduces unique challenges in encoding and decoding document identifiers (docIDs). The paper presents a novel method, Hierarchical encoding-decoding Generative retrieval (Hi-Gen), which addresses several shortcomings of existing generative retrieval models in the context of large-scale personalized E-commerce searches.

Key Methodological Innovations

Representation Learning and Metric Learning:

Hi-Gen introduces a dual-component model to capture both semantic relevance and efficiency information of items:

  • Representation Learning: This component is responsible for generating discriminative embeddings that integrate semantic, common, and efficiency-based features of items using a multi-task learning framework.
  • Metric Learning: Post the initial embeddings, a metric learning module refines these embeddings further to ensure that similar items cluster closely in the embedded space, by employing a triplet margin loss that prioritizes both user interactions and item efficiency.

Category-Guided Hierarchical Clustering for docID Generation:

Rather than relying on traditional methods, Hi-Gen uses a unique category-guided hierarchical clustering to generate docIDs. This method leverages the hierarchical category structure inherent to E-commerce platforms to generate meaningful and structured identifiers encompassing both item category and efficiency metrics.

Decoding with Position-Aware Loss:

A position-aware loss function is employed to fine-tune the LLM that decodes user queries into docIDs. This loss function emphasizes the positional significance of tokens in the docIDs, aiming to capture the hierarchical importance and inter-token relationships more accurately.

Experimental Validation

Quantitative Improvements:

Hi-Gen demonstrates significant improvements over state-of-the-art models in well-known metrics. Specifically, the paper reports a 3.30% and 4.62% increase over existing best models in Recall@1 in public and industry-specific datasets, respectively. Larger gains are noted when exploring metrics like Recall@10, showcasing Hi-Gen's robustness in retrieving relevant documents.

Ablation Study:

Removing any single component from Hi-Gen (such as position-aware loss or metric learning) leads to notable performance dips, underlining the significance of each component in the integrated model. The category-guided clustering emerges as a particularly critical feature, its removal resulting in the most substantial performance decrease.

Zero-Shot and Scalability Tests:

Hi-Gen's generalization capabilities are compelling, particularly in zero-shot learning scenarios where it significantly outperforms other models. The adaptability of Hi-Gen is further validated through scalable online A/B testing on a large-scale E-commerce platform, demonstrating improvements in key business metrics such as Recall, GMV, and conversion rates without incurring additional latency overheads.

Implications and Future Work

The innovation of Hi-Gen lies in its holistic approach, covering feature representation, structured docID generation, and fine-tuning retrieval through position-aware mechanisms. This methodology could set a new standard for personalized search in massive, dynamic inventories typical of modern E-commerce platforms. Future explorations could refine the encoding strategies or adapt the model for other types of personalized recommendation systems, potentially exploring the integration with other forms of machine learning models to enhance decision-making processes further. Additionally, further elaboration on the impacts of different features and embedding layers could provide deeper insights into the contributions of individual model components.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets