Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models (2306.10933v4)

Published 19 Jun 2023 in cs.IR

Abstract: Recommender systems play a vital role in various online services. However, the insulated nature of training and deploying separately within a specific domain limits their access to open-world knowledge. Recently, the emergence of LLMs has shown promise in bridging this gap by encoding extensive world knowledge and demonstrating reasoning capability. Nevertheless, previous attempts to directly use LLMs as recommenders have not achieved satisfactory results. In this work, we propose an Open-World Knowledge Augmented Recommendation Framework with LLMs, dubbed KAR, to acquire two types of external knowledge from LLMs -- the reasoning knowledge on user preferences and the factual knowledge on items. We introduce factorization prompting to elicit accurate reasoning on user preferences. The generated reasoning and factual knowledge are effectively transformed and condensed into augmented vectors by a hybrid-expert adaptor in order to be compatible with the recommendation task. The obtained vectors can then be directly used to enhance the performance of any recommendation model. We also ensure efficient inference by preprocessing and prestoring the knowledge from the LLM. Extensive experiments show that KAR significantly outperforms the state-of-the-art baselines and is compatible with a wide range of recommendation algorithms. We deploy KAR to Huawei's news and music recommendation platforms and gain a 7\% and 1.7\% improvement in the online A/B test, respectively.

PDF HTML Abstract

Towards Open-World Recommendation with Knowledge Augmentation from LLMs

The paper "Towards Open-World Recommendation with Knowledge Augmentation from LLMs" presents a novel framework, KAR (Knowledge-Augmented Recommendation), which addresses the limitations of traditional recommender systems (RSs) by integrating open-world knowledge from LLMs. This work proposes an innovative approach to bridge the gap between LLMs and RSs, offering a method for embedding rich, external knowledge into recommendation algorithms to improve prediction accuracy and generalizability.

Framework Overview

KAR is designed as a model-agnostic framework, emphasizing three primary stages for efficiently incorporating external knowledge from LLMs:

  1. Knowledge Reasoning and Generation: This stage leverages LLMs to extract relevant knowledge by executing a novel technique called "factorization prompting." This approach decomposes user preferences into fundamental factors, extracting both reasoning knowledge related to user preferences and factual knowledge about items. This factorization method helps mitigate the issue of the compositional gap in LLMs, allowing the model to effectively recall pertinent world knowledge aligned with user and item data.
  2. Knowledge Adaptation: The adaptation stage interprets and transforms the text-based knowledge generated by LLMs into dense vectors, compatible with recommendation models. This transformation involves encoding knowledge through a knowledge encoder and refining these representations via a hybrid-expert adaptor. The adaptor reduces dimensionality and tunes the embedding space to conform to the requirements of RSs, enhancing the reliability of the input data.
  3. Knowledge Utilization: Once the knowledge is adapted into suitable vectors, these are incorporated as additional features in existing recommendation models. By linking the reasoning and factual knowledge with traditional domain features, the framework allows RSs to leverage both collaborative filtering signals and extensive world knowledge.

Experimental Results

Empirical evaluations demonstrate that KAR significantly enhances various baselines across critical recommendation tasks, such as Click-Through Rate (CTR) prediction and re-ranking. Noteworthy numerical improvements were observed across multiple datasets, including MovieLens-1M and Amazon-Books, with AUC improvements of 1-2% over state-of-the-art methods. Moreover, KAR's deployment in Huawei’s news and music recommendation platforms substantiates its practical viability, showcasing a 7% and 1.7% improvement in online A/B tests, respectively.

Comparative Analysis and Advantages

KAR shows notable advancements over traditional enhancements, such as those relying solely on knowledge graphs or smaller PLMs. The dual extraction of reasoning and factual knowledge provides comprehensive insights that these other methods lack, encompassing both static item-related information and dynamic, inferred user preferences.

Implications and Future Directions

The framework's ability to prestore and preprocess knowledge ensures that it meets low-latency requirements, particularly crucial for large-scale systems where inference time must remain optimal. The paper encourages further exploration into richer interactions between LLMs and RSs, suggesting potential developments in adaptive LLM-based RS architectures that could dynamically update as new information becomes available. Despite the solid groundwork laid by KAR, future research could focus on directly addressing the privacy considerations and mitigating hallucination issues inherent in LLM deployments.

This research contributes significantly to the evolving domain of recommendation systems by integrating the reasoning capabilities of LLMs, pushing the boundaries of what is feasible in personalized content delivery and user satisfaction enhancement.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Learning a deep listwise context model for ranking refinement. In The 41st international ACM SIGIR conference on research & development in information retrieval. 135–144.
  2. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. arXiv preprint arXiv:2305.00447 (2023).
  3. What Does It Mean for a Language Model to Preserve Privacy?. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). 2280–2292.
  4. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023).
  5. Unifying knowledge graph learning and recommendation: Towards a better understanding of user preferences. In The world wide web conference. 151–161.
  6. M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems. arXiv preprint arXiv:2205.08084 (2022).
  7. Uncovering ChatGPT’s Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023).
  8. SPRank: Semantic Path-based Ranking for Top-N Recommendations using Linked Open Data. ACM Transactions on Intelligent Systems and Technology (TIST) (2016).
  9. Zero-shot recommender systems. arXiv preprint arXiv:2105.08318 (2021).
  10. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 320–335.
  11. Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046 (2023).
  12. Leveraging Large Language Models in Conversational Recommender Systems. arXiv preprint arXiv:2305.07961 (2023).
  13. Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. arXiv preprint arXiv:2303.14524 (2023).
  14. Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt and Predict Paradigm (P5). In Proceedings of the 16th ACM Conference on Recommender Systems. 299–315.
  15. An Unified Search and Recommendation Foundation Model for Cold-Start Scenario. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23). 4595–4601.
  16. DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia). 1725–1731.
  17. A survey on knowledge graph-based recommender systems. IEEE Transactions on Knowledge and Data Engineering 34, 8 (2020), 3549–3568.
  18. Disentangled Representations Learning for Multi-target Cross-domain Recommendation. ACM Transactions on Information Systems 41, 4 (2023), 1–27.
  19. Leveraging Large Language Models for Sequential Recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems. 1096–1102.
  20. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web. 507–517.
  21. Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders. In Proceedings of the ACM Web Conference 2023. 1162–1171.
  22. Towards Universal Sequence Representation Learning for Recommender Systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 585–593.
  23. Large Language Models are Zero-Shot Rankers for Recommender Systems. arXiv preprint arXiv:2305.08845 (2023).
  24. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.
  25. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards Reasoning in Large Language Models: A Survey. arXiv:2212.10403 [cs.CL]
  26. FiBiNET: Combining Feature Importance and Bilinear Feature Interaction for Click-through Rate Prediction. In Proceedings of the 13th ACM Conference on Recommender Systems. 169–177.
  27. Adaptive Mixtures of Local Experts. Neural Computation 3, 1 (03 1991), 79–87.
  28. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM Trans. Inf. Syst. (2002), 422–446.
  29. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38.
  30. Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction. arXiv preprint arXiv:2305.06474 (2023).
  31. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171–4186.
  32. Matrix factorization techniques for recommender systems. Computer 42, 8 (2009), 30–37.
  33. TagGPT: Large Language Models are Zero-shot Multimodal Taggers. arXiv preprint arXiv:2304.03022 (2023).
  34. Text Is All You Need: Learning Language Representations for Sequential Recommendation. arXiv preprint arXiv:2305.13731 (2023).
  35. Large Language Models for Generative Recommendation: A Survey and Visionary Discussions. arXiv preprint arXiv:2309.01157 (2023).
  36. Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 539–548.
  37. XDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1754–1763.
  38. Guo Lin and Yongfeng Zhang. 2023. Sparks of Artificial General Recommender (AGR): Early Experiments with ChatGPT. arXiv preprint arXiv:2305.04518 (2023).
  39. How Can Recommender Systems Benefit from Large Language Models: A Survey. arXiv preprint arXiv:2306.05817 (2023).
  40. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823 (2021).
  41. Is ChatGPT a Good Recommender? A Preliminary Study. arXiv preprint arXiv:2304.10149 (2023).
  42. Pre-train, prompt and recommendation: A comprehensive survey of language modelling paradigm adaptations in recommender systems. arXiv preprint arXiv:2302.03735 (2023).
  43. Neural Re-ranking in Multi-stage Recommender Systems: A Review. arXiv preprint arXiv:2202.06602 (2022).
  44. Llm-rec: Personalized recommendation via prompting large language models. arXiv preprint arXiv:2307.15780 (2023).
  45. Large Language Model Augmented Narrative Driven Recommendations. arXiv preprint arXiv:2306.02250 (2023).
  46. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 188–197.
  47. OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). https://doi.org/10.48550/arXiv.2303.08774
  48. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  49. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 1314–1331.
  50. Setrank: Learning a permutation-invariant ranking model for information retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 499–508.
  51. Personalized re-ranking for recommendation. In Proceedings of the 13th ACM conference on recommender systems. 3–11.
  52. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv:2210.03350 [cs.CL]
  53. Reasoning with Language Model Prompting: A Survey. arXiv:2212.09597 [cs.CL]
  54. U-BERT: Pre-training user representations for improved recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4320–4327.
  55. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  56. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  57. Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International conference on data mining. IEEE, 995–1000.
  58. One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4104–4113.
  59. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1161–1170.
  60. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  61. Deep content-based music recommendation. Advances in neural information processing systems 26 (2013).
  62. Multi-task feature learning for knowledge graph enhanced recommendation. In The world wide web conference. 2000–2010.
  63. Knowledge graph convolutional networks for recommender systems. In The world wide web conference. 3307–3313.
  64. Lei Wang and Ee-Peng Lim. 2023. Zero-Shot Next-Item Recommendation using Large Pretrained Language Models. arXiv preprint arXiv:2304.03153 (2023).
  65. Deep & Cross Network for Ad Click Predictions. In Proceedings of the ADKDD’17. Article 12, 7 pages.
  66. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems. In Proceedings of the Web Conference 2021. 1785–1797.
  67. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022).
  68. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021).
  69. A Survey on Large Language Models for Recommendation. arXiv preprint arXiv:2305.19860 (2023).
  70. Multi-Level Interaction Reranking with User Behavior History. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.
  71. Self-supervised learning for recommender systems: A survey. IEEE Transactions on Knowledge and Data Engineering (2023).
  72. A support vector method for optimizing average precision. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 271–278.
  73. Pangu-α𝛼\alphaitalic_α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369 (2021).
  74. Addressing Cold Start in Recommender Systems: A Semi-Supervised Co-Training Algorithm. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR ’14). 73–82.
  75. Language Models as Recommender Systems: Evaluations and Limitations. In I (Still) Can’t Believe It’s Not Better! NeurIPS 2021 Workshop.
  76. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
  77. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on Learning Representations.
  78. Deep Interest Evolution Network for Click-through Rate Prediction (AAAI’19). Article 729, 8 pages.
  79. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1059–1068.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Yunjia Xi (21 papers)
  2. Weiwen Liu (59 papers)
  3. Jianghao Lin (47 papers)
  4. Xiaoling Cai (4 papers)
  5. Hong Zhu (52 papers)
  6. Jieming Zhu (68 papers)
  7. Bo Chen (309 papers)
  8. Ruiming Tang (171 papers)
  9. Weinan Zhang (322 papers)
  10. Rui Zhang (1138 papers)
  11. Yong Yu (219 papers)
Citations (34)
View on Semantic Scholar