Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 82 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 40 tok/s Pro
GPT-5 High 38 tok/s Pro
GPT-4o 96 tok/s Pro
Kimi K2 185 tok/s Pro
GPT OSS 120B 465 tok/s Pro
Claude Sonnet 4 30 tok/s Pro
2000 character limit reached

RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation (2402.04527v2)

Published 7 Feb 2024 in cs.IR and cs.AI

Abstract: LLMs (LLM) have recently emerged as a powerful tool for a variety of natural language processing tasks, bringing a new surge of combining LLM with recommendation systems, termed as LLM-based RS. Current approaches generally fall into two main paradigms, the ID direct usage paradigm and the ID translation paradigm, noting their core weakness stems from lacking recommendation knowledge and uniqueness. To address this limitation, we propose a new paradigm, ID representation, which incorporates pre-trained ID embeddings into LLMs in a complementary manner. In this work, we present RA-Rec, an efficient ID representation alignment framework for LLM-based recommendation, which is compatible with multiple ID-based methods and LLM architectures. Specifically, we treat ID embeddings as soft prompts and design an innovative alignment module and an efficient tuning method with tailored data construction for alignment. Extensive experiments demonstrate RA-Rec substantially outperforms current state-of-the-art methods, achieving up to 3.0% absolute HitRate@100 improvements while utilizing less than 10x training data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 (2020).
  2. Flamingo: a visual language model for few-shot learning. NeurIPS 35 (2022), 23716–23736.
  3. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).
  4. Recommender systems survey. Knowledge-based systems (2013).
  5. Language Models are Few-Shot Learners. In NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.).
  6. Robin Burke. 2007. Hybrid web recommender systems. The adaptive web: methods and strategies of web personalization (2007), 377–408.
  7. Controllable multi-interest framework for recommendation. In SIGKDD.
  8. Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs. arXiv preprint arXiv:2305.09858 (2023).
  9. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
  10. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. New York, NY, USA.
  11. M6-rec: Generative pretrained language models are open-ended recommender systems. arXiv preprint arXiv:2205.08084 (2022).
  12. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  13. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).
  14. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650 (2022).
  15. Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046 (2023).
  16. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524 (2023).
  17. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems. 299–315.
  18. VIP5: Towards Multimodal Foundation Models for Recommendation. arXiv preprint arXiv:2305.14302 (2023).
  19. Humor knowledge enriched transformer for understanding multimodal humor. In Proceedings of the AAAI conference on artificial intelligence.
  20. Balázs Hidasi and Alexandros Karatzoglou. 2017. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. CoRR abs/1706.03847 (2017). arXiv:1706.03847 http://arxiv.org/abs/1706.03847
  21. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015).
  22. Large Language Models are Zero-Shot Rankers for Recommender Systems. arXiv:2305.08845 [cs.IR]
  23. Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845 (2023).
  24. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799.
  25. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  26. UP5: Unbiased Foundation Model for Fairness-aware Recommendation. arXiv preprint arXiv:2305.12090 (2023).
  27. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023).
  28. A review on methods and applications in multimodal deep learning. ACM Transactions on Multimedia Computing, Communications and Applications (2023).
  29. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 422–446.
  30. What Does BERT Learn about the Structure of Language?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 3651–3657. https://doi.org/10.18653/v1/P19-1356
  31. Wang-Cheng Kang and Julian J. McAuley. 2018. Self-Attentive Sequential Recommendation. CoRR abs/1808.09781 (2018). arXiv:1808.09781 http://arxiv.org/abs/1808.09781
  32. Mohadeseh Kaviani and Hossein Rahmani. 2020. Emhash: Hashtag recommendation using neural network based on bert embedding. In 2020 6th International Conference on Web Research (ICWR). IEEE, 113–118.
  33. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
  34. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 (2023).
  35. GPT4Rec: A generative framework for personalized recommendation and user interests interpretation. arXiv preprint arXiv:2304.03879 (2023).
  36. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019).
  37. MLP4Rec: A Pure MLP Architecture for Sequential Recommendations. arXiv:2204.11510 [cs.IR]
  38. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13401–13412.
  39. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021).
  40. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. CoRR (2021).
  41. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. CoRR abs/2110.07602 (2021). https://arxiv.org/abs/2110.07602
  42. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  43. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32 (2019).
  44. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2645–2652.
  45. Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration. arXiv preprint arXiv:2306.09093 (2023).
  46. Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 188–197. https://doi.org/10.18653/v1/D19-1018
  47. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).
  48. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv preprint arXiv:2306.14824 (2023).
  49. Guanghui Qin and Jas’ Eisner. 2021. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In ACL.
  50. U-BERT: Pre-training user representations for improved recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4320–4327.
  51. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  52. Language models are unsupervised multitask learners. OpenAI blog (2019).
  53. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. CoRR abs/1910.10683 (2019). arXiv:1910.10683 http://arxiv.org/abs/1910.10683
  54. A systematic review of ontology use in E-Learning recommender system. Computers and Education: Artificial Intelligence (2022).
  55. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012).
  56. Paul Resnick and Hal R Varian. 1997. Recommender systems. Commun. ACM (1997).
  57. Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences. arXiv:2307.14225 [cs.IR]
  58. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020).
  59. Recommender systems in e-commerce. In 2014 World Automation Congress (WAC). IEEE, 179–184.
  60. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision.
  61. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. CIKM (2019).
  62. Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining. 565–573.
  63. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950 (2019).
  64. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  65. Multimodal few-shot learning with frozen language models. NeurIPS 34 (2021), 200–212.
  66. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research (2008).
  67. Lei Wang and Ee-Peng Lim. 2023. Zero-Shot Next-Item Recommendation using Large Pretrained Language Models. arXiv preprint arXiv:2304.03153 (2023).
  68. Unsupervised Deep Structured Semantic Models for Commonsense Reasoning. arXiv:1904.01938 [cs.CL]
  69. Generative recommendation: Towards next-generation recommender paradigm. arXiv preprint arXiv:2304.03516 (2023).
  70. Finetuned language models are zero-shot learners. In ICLR.
  71. Empowering news recommendation with pre-trained language models. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1652–1656.
  72. A Survey on Large Language Models for Recommendation. arXiv preprint arXiv:2305.19860 (2023).
  73. Graph neural networks in recommender systems: a survey. Comput. Surveys (2022).
  74. Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models. arXiv preprint arXiv:2306.10933 (2023).
  75. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  76. What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? arXiv preprint arXiv:2307.02469 (2023).
  77. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 (2023).
  78. UNBERT: User-News Matching BERT for News Recommendation.. In IJCAI. 3356–3362.
  79. Instruction Tuning for Large Language Models: A Survey. arXiv preprint arXiv:2308.10792 (2023).
  80. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR) (2019).
  81. Language models as recommender systems: Evaluations and limitations. In NeurIPS 2021 Workshop on I (Still) Can’t Believe It’s Not Better.
Citations (6)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now