Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Elephant in the Room: Rethinking the Usage of Pre-trained Language Model in Sequential Recommendation (2404.08796v3)

Published 12 Apr 2024 in cs.IR

Abstract: Sequential recommendation (SR) has seen significant advancements with the help of Pre-trained LLMs (PLMs). Some PLM-based SR models directly use PLM to encode user historical behavior's text sequences to learn user representations, while there is seldom an in-depth exploration of the capability and suitability of PLM in behavior sequence modeling. In this work, we first conduct extensive model analyses between PLMs and PLM-based SR models, discovering great underutilization and parameter redundancy of PLMs in behavior sequence modeling. Inspired by this, we explore different lightweight usages of PLMs in SR, aiming to maximally stimulate the ability of PLMs for SR while satisfying the efficiency and usability demands of practical systems. We discover that adopting behavior-tuned PLMs for item initializations of conventional ID-based SR models is the most economical framework of PLM-based SR, which would not bring in any additional inference cost but could achieve a dramatic performance boost compared with the original version. Extensive experiments on five datasets show that our simple and universal framework leads to significant improvement compared to classical SR and SOTA PLM-based SR models without additional inference costs. Our code can be found in https://github.com/777pomingzi/Rethinking-PLM-in-RS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. “Gpt-4 technical report” In arXiv preprint arXiv:2303.08774, 2023
  2. “A bi-step grounding paradigm for large language models in recommendation systems” In arXiv preprint arXiv:2308.08434, 2023
  3. “TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation” In arXiv preprint arXiv:2305.00447, 2023
  4. Iz Beltagy, Matthew E Peters and Arman Cohan “Longformer: The long-document transformer” In arXiv preprint arXiv:2004.05150, 2020
  5. “Language models are few-shot learners” In Advances in neural information processing systems 33, 2020, pp. 1877–1901
  6. “Contrastive Cross-Domain Sequential Recommendation” In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022, pp. 138–147
  7. “Sequential recommendation with graph neural networks” In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, 2021, pp. 378–387
  8. “Intent contrastive learning for sequential recommendation” In Proceedings of the ACM Web Conference 2022, 2022, pp. 2172–2182
  9. “Learning phrase representations using RNN encoder-decoder for statistical machine translation” In arXiv preprint arXiv:1406.1078, 2014
  10. “M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems” In arXiv preprint arXiv:2205.08084, 2022
  11. Alexander Dallmann, Daniel Zoller and Andreas Hotho “A case study on sampling strategies for evaluating neural sequential item recommendation models” In Proceedings of the 15th ACM Conference on Recommender Systems, 2021, pp. 505–514
  12. Tim Donkers, Benedikt Loepp and Jürgen Ziegler “Sequential user-based recurrent neural network recommendations” In Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 152–160
  13. “Chat-rec: Towards interactive and explainable llms-augmented recommender system” In arXiv preprint arXiv:2303.14524, 2023
  14. “Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5)” In Proceedings of the 16th ACM Conference on Recommender Systems, 2022, pp. 299–315
  15. “Evolutionary preference learning via graph nested gru ode for session-based recommendation” In Proceedings of the 31st ACM international conference on information & knowledge management, 2022, pp. 624–634
  16. “Adversarial feature translation for multi-domain recommendation” In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 2964–2973
  17. “Leveraging large language models for sequential recommendation” In Proceedings of the 17th ACM Conference on Recommender Systems, 2023, pp. 1096–1102
  18. Ruining He, Wang-Cheng Kang and Julian McAuley “Translation-based recommendation” In Proceedings of the eleventh ACM conference on recommender systems, 2017, pp. 161–169
  19. “Fusing similarity models with markov chains for sparse sequential recommendation” In 2016 IEEE 16th international conference on data mining (ICDM), 2016, pp. 191–200 IEEE
  20. “Session-based recommendations with recurrent neural networks” In arXiv preprint arXiv:1511.06939, 2015
  21. “Towards Universal Sequence Representation Learning for Recommender Systems” In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 585–593
  22. “Large language models are zero-shot rankers for recommender systems” In arXiv preprint arXiv:2305.08845, 2023
  23. “Self-attentive sequential recommendation” In 2018 IEEE international conference on data mining (ICDM), 2018, pp. 197–206 IEEE
  24. “Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction” In arXiv preprint arXiv:2305.06474, 2023
  25. “On sampled metrics for item recommendation” In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2020, pp. 1748–1757
  26. “Text Is All You Need: Learning Language Representations for Sequential Recommendation” In arXiv preprint arXiv:2305.13731, 2023
  27. “Is ChatGPT a Good Recommender? A Preliminary Study” In arXiv preprint arXiv:2304.10149, 2023
  28. “π𝜋\piitalic_π-net: A parallel information-sharing network for shared-account cross-domain sequential recommendations” In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, 2019, pp. 685–694
  29. “ID-Agnostic User Behavior Pre-training for Sequential Recommendation” In Information Retrieval: 28th China Conference, CCIR 2022, Chongqing, China, September 16–18, 2022, Revised Selected Papers, 2023, pp. 16–27 Springer
  30. Jianmo Ni, Jiacheng Li and Julian McAuley “Justifying recommendations using distantly-labeled reviews and fine-grained aspects” In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 188–197
  31. “Training language models to follow instructions with human feedback” In Advances in neural information processing systems 35, 2022, pp. 27730–27744
  32. “Contrastive learning for representation degeneration problem in sequential recommendation” In Proceedings of the fifteenth ACM international conference on web search and data mining, 2022, pp. 813–823
  33. “Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language” In arXiv preprint arXiv:2310.13540, 2023
  34. “Representation learning with large language models for recommendation” In arXiv preprint arXiv:2310.15950, 2023
  35. Steffen Rendle, Christoph Freudenthaler and Lars Schmidt-Thieme “Factorizing personalized markov chains for next-basket recommendation” In Proceedings of the 19th international conference on World wide web, 2010, pp. 811–820
  36. “BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer” In Proceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1441–1450
  37. “Personalized top-n sequential recommendation via convolutional sequence embedding” In Proceedings of the eleventh ACM international conference on web search and data mining, 2018, pp. 565–573
  38. “Attention is all you need” In Advances in neural information processing systems 30, 2017
  39. “TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback” In arXiv preprint, 2022
  40. “Zero-Shot Next-Item Recommendation using Large Pretrained Language Models” In arXiv preprint arXiv:2304.03153, 2023
  41. “Exploiting intra-and inter-session dependencies for session-based recommendations” In World Wide Web 25.1 Springer, 2022, pp. 425–443
  42. “Llmrec: Large language models with graph augmentation for recommendation” In arXiv preprint arXiv:2311.00423, 2023
  43. “Session-based recommendation with graph neural networks” In Proceedings of the AAAI conference on artificial intelligence 33.01, 2019, pp. 346–353
  44. “Selective fairness in recommendation via prompts” In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 2657–2662
  45. “Contrastive learning for sequential recommendation” In 2022 IEEE 38th international conference on data engineering (ICDE), 2022, pp. 1259–1273 IEEE
  46. “Long-and short-term self-attention network for sequential recommendation” In Neurocomputing 423 Elsevier, 2021, pp. 580–589
  47. “Where to go next for recommender systems? id-vs. modality-based recommender models revisited” In arXiv preprint arXiv:2303.13835, 2023
  48. “Recommendation as instruction following: A large language model empowered recommendation approach” In arXiv preprint arXiv:2305.07001, 2023
  49. “S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization” In Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 1893–1902
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Zekai Qu (3 papers)
  2. Ruobing Xie (97 papers)
  3. Chaojun Xiao (39 papers)
  4. Xingwu Sun (32 papers)
  5. Zhanhui Kang (45 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now