Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Contrastive Pre-training for Deep Session Data Understanding (2403.02825v1)

Published 5 Mar 2024 in cs.IR

Abstract: Session data has been widely used for understanding user's behavior in e-commerce. Researchers are trying to leverage session data for different tasks, such as purchase intention prediction, remaining length prediction, recommendation, etc., as it provides context clues about the user's dynamic interests. However, online shopping session data is semi-structured and complex in nature, which contains both unstructured textual data about the products, search queries, and structured user action sequences. Most existing works focus on leveraging the coarse-grained item sequences for specific tasks, while largely ignore the fine-grained information from text and user action details. In this work, we delve into deep session data understanding via scrutinizing the various clues inside the rich information in user sessions. Specifically, we propose to pre-train a general-purpose User Behavior Model (UBM) over large-scale session data with rich details, such as product title, attributes and various kinds of user actions. A two-stage pre-training scheme is introduced to encourage the model to self-learn from various augmentations with contrastive learning objectives, which spans different granularity levels of session data. Then the well-trained session understanding model can be easily fine-tuned for various downstream tasks. Extensive experiments show that UBM better captures the complex intra-item semantic relations, inter-item connections and inter-interaction dependencies, leading to large performance gains as compared to the baselines on several downstream tasks. And it also demonstrates strong robustness when data is sparse.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. In Proceedings of the Second Workshop on Insights from Negative Results in NLP. ACL, 125–135.
  2. BERT Goes Shopping: Comparing Distributional Models for Product Representations. In Proceedings of The 4th Workshop on e-Commerce and NLP. ACL.
  3. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems. 1877–1901.
  4. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 1597–1607.
  5. AIR: Attentional Intention-Aware Recommender Systems. In 2019 IEEE 35th International Conference on Data Engineering. 304–315.
  6. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In Proceedings of the 2020 International Conference on Learning Representations.
  7. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv (2019).
  8. Predicting Shopping Intent of e-Commerce Users using LSTM Recurrent Neural Networks. In 10th International Conference on Data Science, Technology and Applications.
  9. Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks. In IJCAI’2022 SURVEY TRACK.
  10. A Survey of Vision-Language Pre-Trained Models.
  11. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research (2010), 625–660.
  12. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. ACL, 6894–6910.
  13. NISER: Normalized Item and Session Representations with Graph Neural Networks. CoRR (2019).
  14. Shashank Gupta and Subhadeep Maji. 2020. Predicting Session Length for Product Search on E-Commerce Platform. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM.
  15. Dimensionality reduction by learning an invariant mapping. In CVPR, Vol. 2. 1735–1742.
  16. Tobias Hatt and Stefan Feuerriegel. 2020. Early Detection of User Exits from Clickstream Data: A Markov Modulated Marked Point Process Model. In Proceedings of The Web Conference 2020. ACM, 1671–1681.
  17. Momentum contrast for unsupervised visual representation learning. In CVPR. 9729–9738.
  18. Deep Residual Learning for Image Recognition.
  19. Session-based Recommendations with Recurrent Neural Networks. CoRR (2016).
  20. A Survey on Contrastive Self-Supervised Learning. Technologies (2021).
  21. Hard Negative Mixing for Contrastive Learning. In Neural Information Processing Systems.
  22. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations.
  23. Learning to Ask Critical Questions for Assisting Product Search. In Proceedings of ACM SIGIR Workshop on eCommerce (SIGIR eCom’22).
  24. STAMP: Short-Term Attention/Memory Priority Model for Session-Based Recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1831–1839.
  25. RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  26. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL, 2227–2237.
  27. Factorizing Personalized Markov Chains for Next-Basket Recommendation. In Proceedings of the 19th International Conference on World Wide Web. ACM, 811–820.
  28. Contrastive Learning with Hard Negative Samples. In International Conference on Learning Representations.
  29. Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks. Neural Computing and Applications (2019).
  30. An MDP-Based Recommender System. In Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers Inc.
  31. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations.
  32. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. ACM, 1441–1450.
  33. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. ACL, 5100–5111.
  34. Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 565–573.
  35. What Makes for Good Views for Contrastive Learning?. In Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates Inc.
  36. Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation. CoRR (2019).
  37. A Survey on Session-Based Recommender Systems. Comput. Surveys (2021).
  38. Sequential Recommender Systems: Challenges, Progress and Prospects. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. IJCAI, 6332–6338.
  39. Tongzhou Wang and Phillip Isola. 2020. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings of the 37th International Conference on Machine Learning. JMLR.
  40. Session-Based Recommendation with Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence (2019), 346–353.
  41. ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In ACL. 5065–5075.
  42. ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL, 5065–5075.
  43. KG-BERT: BERT for knowledge graph completion. arXiv (2019).
  44. Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking. Proceedings of the 30th ACM International Conference on Information and Knowledge Management (2021).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zixuan Li (63 papers)
  2. Lizi Liao (44 papers)
  3. Yunshan Ma (42 papers)
  4. Tat-Seng Chua (359 papers)