Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Harnessing Large Language Models for Text-Rich Sequential Recommendation (2403.13325v1)

Published 20 Mar 2024 in cs.IR

Abstract: Recent advances in LLMs have been changing the paradigm of Recommender Systems (RS). However, when items in the recommendation scenarios contain rich textual information, such as product descriptions in online shopping or news headlines on social media, LLMs require longer texts to comprehensively depict the historical user behavior sequence. This poses significant challenges to LLM-based recommenders, such as over-length limitations, extensive time and space overheads, and suboptimal model performance. To this end, in this paper, we design a novel framework for harnessing LLMs for Text-Rich Sequential Recommendation (LLM-TRSR). Specifically, we first propose to segment the user historical behaviors and subsequently employ an LLM-based summarizer for summarizing these user behavior blocks. Particularly, drawing inspiration from the successful application of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) models in user modeling, we introduce two unique summarization techniques in this paper, respectively hierarchical summarization and recurrent summarization. Then, we construct a prompt text encompassing the user preference summary, recent user interactions, and candidate item information into an LLM-based recommender, which is subsequently fine-tuned using Supervised Fine-Tuning (SFT) techniques to yield our final recommendation model. We also use Low-Rank Adaptation (LoRA) for Parameter-Efficient Fine-Tuning (PEFT). We conduct experiments on two public datasets, and the results clearly demonstrate the effectiveness of our approach.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Neural news recommendation with long-and short-term user representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 336–345.
  2. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. arXiv preprint arXiv:2305.00447 (2023).
  3. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. PMLR, 2397–2430.
  4. Predicting the popularity of online serials with autoregressive models. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. 1339–1348.
  5. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  6. Recruitpro: A pretrained language model with skill-aware prompt learning for intelligent recruitment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3991–4002.
  7. GUESR: A global unsupervised data-enhancement with bucket-cluster sampling for sequential recommendation. In International Conference on Database Systems for Advanced Applications. Springer, 286–296.
  8. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. 173–182.
  9. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015).
  10. Core: simple and effective session-based recommendation within consistent representation space. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 1796–1801.
  11. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  12. BOSS: A Bilateral Occupational-Suitability-Aware Recommender System for Online Recruitment. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4146–4155.
  13. Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation. arXiv preprint arXiv:2307.09688 (2023).
  14. Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM). IEEE, 197–206.
  15. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Vol. 1. 2.
  16. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1419–1428.
  17. Is ChatGPT a Good Recommender? A Preliminary Study. arXiv preprint arXiv:2304.10149 (2023).
  18. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023).
  19. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
  20. Recent advances in natural language processing via large pre-trained language models: A survey. arXiv preprint arXiv:2111.01243 (2021).
  21. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  22. Towards Efficient and Effective Adaptation of Large Language Models for Sequential Recommendation. (2024).
  23. Summarization is (Almost) Dead. arXiv preprint arXiv:2309.09558 (2023).
  24. A Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics. arXiv preprint arXiv:2307.03195 (2023).
  25. U-BERT: Pre-training user representations for improved recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4320–4327.
  26. Improving language understanding by generative pre-training. (2018).
  27. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  28. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506.
  29. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management. 1441–1450.
  30. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  31. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  32. Attention is all you need. Advances in neural information processing systems 30 (2017).
  33. Variable interval time sequence modeling for career trajectory prediction: Deep collaborative perspective. In Proceedings of the Web Conference 2021. 612–623.
  34. Setrank: A setwise bayesian approach for collaborative ranking from implicit feedback. In Proceedings of the aaai conference on artificial intelligence, Vol. 34. 6127–6136.
  35. Is News Recommendation a Sequential Recommendation Task?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2382–2386.
  36. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3597–3606.
  37. Exploring large language model for graph data understanding in online job recommendations. arXiv preprint arXiv:2307.05722 (2023).
  38. A Survey on Large Language Models for Recommendation. arXiv preprint arXiv:2305.19860 (2023).
  39. Effective Long-Context Scaling of Foundation Models. arXiv preprint arXiv:2309.16039 (2023).
  40. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019).
  41. Talent demand forecasting with attentive neural sequential model. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 3906–3916.
  42. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
  43. Generative job recommendations with large language model. arXiv preprint arXiv:2307.02157 (2023).
  44. DDR: Dialogue Based Doctor Recommendation for Online Medical Service. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4592–4600.
  45. CBR: context bias aware recommendation for debiasing user modeling and click prediction. In Proceedings of the ACM Web Conference 2022. 2268–2276.
  46. Drug package recommendation via interaction-aware graph induction. In Proceedings of the Web Conference 2021. 1284–1295.
  47. Interaction-aware drug package recommendation via policy gradient. ACM Transactions on Information Systems 41, 1 (2023), 1–32.
  48. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941–5948.
  49. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1059–1068.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Zhi Zheng (46 papers)
  2. Wenshuo Chao (5 papers)
  3. Zhaopeng Qiu (13 papers)
  4. Hengshu Zhu (66 papers)
  5. Hui Xiong (244 papers)
Citations (22)
View on Semantic Scholar