Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model (2405.00338v3)

Published 1 May 2024 in cs.IR

Abstract: Owing to their powerful semantic reasoning capabilities, LLMs have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher's knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher's knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings. To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
  2. Fitnets: Hints for thin deep nets. Proc. ICLR 2, 3 (2015), 1.
  3. A bi-step grounding paradigm for large language models in recommendation systems. arXiv preprint arXiv:2308.08434 (2023).
  4. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems. 1007–1014.
  5. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  6. Aligning Large Language Models with Recommendation Knowledge. arXiv preprint arXiv:2404.00245 (2024).
  7. Sequential recommendation with graph neural networks. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 378–387.
  8. Unbiased knowledge distillation for recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 976–984.
  9. Sequential recommendation with user memory networks. In Proceedings of the eleventh ACM international conference on web search and data mining. 108–116.
  10. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision. 4794–4802.
  11. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  12. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems 33 (2020), 12345–12355.
  13. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Transactions on Information Systems (TOIS) 39, 1 (2020), 1–42.
  14. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524 (2023).
  15. Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors. arXiv preprint arXiv:2403.19347 (2024).
  16. Knowledge distillation: A survey. International Journal of Computer Vision 129, 6 (2021), 1789–1819.
  17. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 639–648.
  18. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015).
  19. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
  20. Learning vector-quantized item representation for transferable sequential recommenders. In Proceedings of the ACM Web Conference 2023. 1162–1171.
  21. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 585–593.
  22. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval. Springer, 364–381.
  23. Exact and Efficient Unlearning for Large Language Model-based Recommendation. arXiv:2404.10327 [cs.IR]
  24. Knowledge distillation from a stronger teacher. Advances in Neural Information Processing Systems 35 (2022), 33716–33727.
  25. Item-side Fairness of Large Language Model-based Recommendation System. arXiv preprint arXiv:2402.15215 (2024).
  26. DE-RRD: A knowledge distillation framework for recommender system. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 605–614.
  27. Topology distillation for recommender system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 829–839.
  28. Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM). IEEE, 197–206.
  29. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  30. Collaborative distillation for top-N recommendation. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 369–378.
  31. Youngjune Lee and Kee-Eung Kim. 2021. Dual correction strategy for ranking distillation in top-n recommender system. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 3186–3190.
  32. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th international conference on web search and data mining. 322–330.
  33. Prompt distillation for efficient llm-based recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 1348–1357.
  34. Citation-Enhanced Generation for LLM-based Chatbots. arXiv:2402.16063 [cs.CL]
  35. Exploring fine-tuning chatgpt for news recommendation. arXiv preprint arXiv:2311.05850 (2023).
  36. PAP-REC: Personalized Automatic Prompt for Recommendation Language Model. arXiv:2402.00284 [cs.IR]
  37. Llara: Aligning large language models with sequential recommenders. arXiv preprint arXiv:2312.02445 (2023).
  38. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. arXiv preprint arXiv:2308.11131 (2023).
  39. A multi-facet paradigm to bridge large language model and recommendation. arXiv preprint arXiv:2310.06491 (2023).
  40. Data-efficient Fine-tuning for LLM-based Recommendation. arXiv preprint arXiv:2401.17197 (2024).
  41. A first look at llm-powered generative news recommendation. arXiv preprint arXiv:2305.06566 (2023).
  42. Accurate sum and dot product. SIAM Journal on Scientific Computing 26, 6 (2005), 1955–1988.
  43. Allan Pinkus. 1999. Approximation theory of the MLP model in neural networks. Acta numerica 8 (1999), 143–195.
  44. D2K: Turning Historical Data into Retrievable Knowledge for Recommender Systems. arXiv preprint arXiv:2401.11478 (2024).
  45. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36 (2024).
  46. Representation learning with large language models for recommendation. arXiv preprint arXiv:2310.15950 (2023).
  47. Usha Ruby and Vamsidhar Yendapalli. 2020. Binary cross entropy with deep learning technique for image classification. Int. J. Adv. Trends Comput. Sci. Eng 9, 10 (2020).
  48. Enhancing Long-Term Recommendation with Bi-level Learnable Large Language Model Planning. arXiv preprint arXiv:2403.00843 (2024).
  49. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management. 1441–1450.
  50. Large Language Models Enhanced Collaborative Filtering. arXiv preprint arXiv:2403.17688 (2024).
  51. Towards LLM-RecSys Alignment with Textual ID Learning. arXiv:2403.19021 [cs.IR]
  52. Jiaxi Tang and Ke Wang. 2018a. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining. 565–573.
  53. Jiaxi Tang and Ke Wang. 2018b. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2289–2298.
  54. Revisiting graph based social recommendation: A distillation enhanced social graph network. In Proceedings of the ACM Web Conference 2022. 2830–2838.
  55. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  56. Towards Efficient and Effective Unlearning of Large Language Models for Recommendation. arXiv:2403.03536 [cs.IR]
  57. ALT: Towards Fine-grained Alignment between Language and CTR Models for Click-Through Rate Prediction. arXiv preprint arXiv:2310.19453 (2023).
  58. Lei Wang and Ee-Peng Lim. 2023. Zero-shot next-item recommendation using large pretrained language models. arXiv preprint arXiv:2304.03153 (2023).
  59. Sequential recommender systems: challenges, progress and prospects. arXiv preprint arXiv:2001.04830 (2019).
  60. Enhanced Generative Recommendation via Content and Collaboration Integration. arXiv preprint arXiv:2403.18480 (2024).
  61. Can Small Language Models be Good Reasoners for Sequential Recommendation? arXiv preprint arXiv:2403.04260 (2024).
  62. To Recommend or Not: Recommendability Identification in Conversations with Pre-trained Language Models. arXiv:2403.18628 [cs.IR]
  63. Llmrec: Large language models with graph augmentation for recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 806–815.
  64. Towards open-world recommendation with knowledge augmentation from large language models. arXiv preprint arXiv:2306.10933 (2023).
  65. Contrastive learning for sequential recommendation. In 2022 IEEE 38th international conference on data engineering (ICDE). IEEE, 1259–1273.
  66. Enhancing Content-based Recommendation via Large Language Model. arXiv preprint arXiv:2404.00236 (2024).
  67. Ukd: Debiasing conversion rate estimation via uncertainty-regularized knowledge distillation. In Proceedings of the ACM Web Conference 2022. 2078–2087.
  68. Common Sense Enhanced Knowledge-based Recommendation with Large Language Model. arXiv preprint arXiv:2403.18325 (2024).
  69. A generic learning framework for sequential recommendation with distribution shifts. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 331–340.
  70. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 1285–1294.
  71. Prospect Personalized Recommendation on Large Language Model-based Agent Platform. arXiv:2402.18240 [cs.IR]
  72. Recommendation as instruction following: A large language model empowered recommendation approach. arXiv preprint arXiv:2305.07001 (2023).
  73. Tired of Plugins? Large Language Models Can Be End-To-End Recommenders. arXiv preprint arXiv:2404.00702 (2024).
  74. Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311.09049 (2023).
  75. Harnessing Large Language Models for Text-Rich Sequential Recommendation. arXiv preprint arXiv:2403.13325 (2024).
  76. Ensembled CTR prediction via knowledge distillation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2941–2958.
  77. Collaborative large language model for recommender systems. arXiv preprint arXiv:2311.01343 (2023).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Yu Cui (29 papers)
  2. Feng Liu (1212 papers)
  3. Pengbo Wang (8 papers)
  4. Bohao Wang (11 papers)
  5. Heng Tang (7 papers)
  6. Yi Wan (35 papers)
  7. Jun Wang (990 papers)
  8. Jiawei Chen (160 papers)
Citations (4)
View on Semantic Scholar