Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language (2310.13540v3)

Published 20 Oct 2023 in cs.IR

Abstract: With the thriving of pre-trained LLM (PLM) widely verified in various of NLP tasks, pioneer efforts attempt to explore the possible cooperation of the general textual information in PLM with the personalized behavioral information in user historical behavior sequences to enhance sequential recommendation (SR). However, despite the commonalities of input format and task goal, there are huge gaps between the behavioral and textual information, which obstruct thoroughly modeling SR as LLMing via PLM. To bridge the gap, we propose a novel Unified pre-trained LLM enhanced sequential recommendation (UPSR), aiming to build a unified pre-trained recommendation model for multi-domain recommendation tasks. We formally design five key indicators, namely naturalness, domain consistency, informativeness, noise & ambiguity, and text length, to guide the text-item adaptation and behavior sequence-text sequence adaptation differently for pre-training and fine-tuning stages, which are essential but under-explored by previous works. In experiments, we conduct extensive evaluations on seven datasets with both tuning and zero-shot settings and achieve the overall best performance. Comprehensive model analyses also provide valuable insights for behavior modeling via PLM, shedding light on large pre-trained recommendation models. The source codes will be released in the future.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. “TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation” In arXiv preprint arXiv:2305.00447, 2023
  2. Iz Beltagy, Matthew E Peters and Arman Cohan “Longformer: The long-document transformer” In arXiv preprint arXiv:2004.05150, 2020
  3. “Language models are few-shot learners” In Advances in neural information processing systems 33, 2020, pp. 1877–1901
  4. “Contrastive Cross-Domain Sequential Recommendation” In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 138–147
  5. “Intent contrastive learning for sequential recommendation” In Proceedings of the ACM Web Conference 2022, 2022, pp. 2172–2182
  6. “Learning phrase representations using RNN encoder-decoder for statistical machine translation” In arXiv preprint arXiv:1406.1078, 2014
  7. “Scaling instruction-finetuned language models” In arXiv preprint arXiv:2210.11416, 2022
  8. “M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems” In arXiv preprint arXiv:2205.08084, 2022
  9. “Bert: Pre-training of deep bidirectional transformers for language understanding” In arXiv preprint arXiv:1810.04805, 2018
  10. “Longnet: Scaling transformers to 1,000,000,000 tokens” In arXiv preprint arXiv:2307.02486, 2023
  11. Tim Donkers, Benedikt Loepp and Jürgen Ziegler “Sequential user-based recurrent neural network recommendations” In Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 152–160
  12. “An image is worth 16x16 words: Transformers for image recognition at scale” In arXiv preprint arXiv:2010.11929, 2020
  13. “Chat-rec: Towards interactive and explainable llms-augmented recommender system” In arXiv preprint arXiv:2303.14524, 2023
  14. “Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5)” In Proceedings of the 16th ACM Conference on Recommender Systems, 2022, pp. 299–315
  15. “VIP5: Towards Multimodal Foundation Models for Recommendation” In arXiv preprint arXiv:2305.14302, 2023
  16. “Hashtag recommendation using attention-based convolutional neural network.” In IJCAI, 2016, pp. 2782–2788
  17. “Adversarial feature translation for multi-domain recommendation” In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 2964–2973
  18. Ruining He, Wang-Cheng Kang and Julian McAuley “Translation-based recommendation” In Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 161–169
  19. “Fusing similarity models with markov chains for sparse sequential recommendation” In 2016 IEEE 16th international conference on data mining (ICDM), 2016, pp. 191–200 IEEE
  20. “Session-based recommendations with recurrent neural networks” In arXiv preprint arXiv:1511.06939, 2015
  21. “Large language models are zero-shot rankers for recommender systems” In arXiv preprint arXiv:2305.08845, 2023
  22. “Towards Universal Sequence Representation Learning for Recommender Systems” In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 585–593
  23. “Improving sequential recommendation with knowledge-enhanced memory networks” In The 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 505–514
  24. “Self-attentive sequential recommendation” In 2018 IEEE international conference on data mining (ICDM), 2018, pp. 197–206 IEEE
  25. “Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction” In arXiv preprint arXiv:2305.06474, 2023
  26. “Text Is All You Need: Learning Language Representations for Sequential Recommendation” In arXiv preprint arXiv:2305.13731, 2023
  27. “Is ChatGPT a Good Recommender? A Preliminary Study” In arXiv preprint arXiv:2304.10149, 2023
  28. “Swin transformer: Hierarchical vision transformer using shifted windows” In Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022
  29. “π𝜋\piitalic_π-net: A parallel information-sharing network for shared-account cross-domain sequential recommendations” In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, 2019, pp. 685–694
  30. “Recurrent neural network based language model.” In Interspeech 2.3, 2010, pp. 1045–1048 Makuhari
  31. “ID-Agnostic User Behavior Pre-training for Sequential Recommendation” In Information Retrieval: 28th China Conference, CCIR 2022, Chongqing, China, September 16–18, 2022, Revised Selected Papers, 2023, pp. 16–27 Springer
  32. Jianmo Ni, Jiacheng Li and Julian McAuley “Justifying recommendations using distantly-labeled reviews and fine-grained aspects” In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 188–197
  33. “Contrastive learning for representation degeneration problem in sequential recommendation” In Proceedings of the fifteenth ACM international conference on web search and data mining, 2022, pp. 813–823
  34. “Exploring the limits of transfer learning with a unified text-to-text transformer” In The Journal of Machine Learning Research 21.1 JMLRORG, 2020, pp. 5485–5551
  35. Steffen Rendle, Christoph Freudenthaler and Lars Schmidt-Thieme “Factorizing personalized markov chains for next-basket recommendation” In Proceedings of the 19th international conference on World wide web, 2010, pp. 811–820
  36. Rico Sennrich, Barry Haddow and Alexandra Birch “Neural machine translation of rare words with subword units” In arXiv preprint arXiv:1508.07909, 2015
  37. “BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer” In Proceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1441–1450
  38. “Personalized top-n sequential recommendation via convolutional sequence embedding” In Proceedings of the eleventh ACM international conference on web search and data mining, 2018, pp. 565–573
  39. “Attention is all you need” In Advances in neural information processing systems 30, 2017
  40. “TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback” In arXiv preprint, 2022
  41. “Zero-Shot Next-Item Recommendation using Large Pretrained Language Models” In arXiv preprint arXiv:2304.03153, 2023
  42. “Transformers: State-of-the-art natural language processing” In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45
  43. “Empowering news recommendation with pre-trained language models” In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 1652–1656
  44. “Session-based recommendation with graph neural networks” In Proceedings of the AAAI conference on artificial intelligence 33.01, 2019, pp. 346–353
  45. “Attacking Pre-trained Recommendation” In Preceedings of SIGIR, 2023
  46. “Personalized prompts for sequential recommendation” In arXiv preprint arXiv:2205.09666, 2022
  47. “Selective fairness in recommendation via prompts” In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 2657–2662
  48. “Uprec: User-aware pre-training for recommender systems” In arXiv preprint arXiv:2102.10989, 2021
  49. “UPRec: User-aware Pre-training for sequential Recommendation” In AI Open, 2023
  50. “Contrastive learning for sequential recommendation” In 2022 IEEE 38th international conference on data engineering (ICDE), 2022, pp. 1259–1273 IEEE
  51. “Long-and short-term self-attention network for sequential recommendation” In Neurocomputing 423 Elsevier, 2021, pp. 580–589
  52. “One person, one model, one world: Learning continual user representation without forgetting” In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 696–705
  53. “Parameter-efficient transfer from sequential behaviors for user modeling and recommendation” In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, 2020, pp. 1469–1478
  54. “Knowledge transfer via pre-training for recommendation: A review and prospect” In Frontiers in big Data 4 Frontiers Media SA, 2021, pp. 602071
  55. “Collaborative knowledge base embedding for recommender systems” In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 353–362
  56. “AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems” In arXiv preprint arXiv:2310.09233, 2023
  57. “Recommendation as instruction following: A large language model empowered recommendation approach” In arXiv preprint arXiv:2305.07001, 2023
  58. “Language models as recommender systems: Evaluations and limitations”, 2021
  59. Lei Zheng, Vahid Noroozi and Philip S Yu “Joint deep modeling of users and items using reviews for recommendation” In Proceedings of the tenth ACM international conference on web search and data mining, 2017, pp. 425–434
  60. “S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization” In Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 1893–1902
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Zekai Qu (3 papers)
  2. Ruobing Xie (97 papers)
  3. Chaojun Xiao (39 papers)
  4. Yuan Yao (292 papers)
  5. Zhiyuan Liu (433 papers)
  6. Fengzong Lian (10 papers)
  7. Zhanhui Kang (45 papers)
  8. Jie Zhou (687 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now