Towards Graph Foundation Models for Personalization (2403.07478v1)
Abstract: In the realm of personalization, integrating diverse information sources such as consumption signals and content-based representations is becoming increasingly critical to build state-of-the-art solutions. In this regard, two of the biggest trends in research around this subject are Graph Neural Networks (GNNs) and Foundation Models (FMs). While GNNs emerged as a popular solution in industry for powering personalization at scale, FMs have only recently caught attention for their promising performance in personalization tasks like ranking and retrieval. In this paper, we present a graph-based foundation modeling approach tailored to personalization. Central to this approach is a Heterogeneous GNN (HGNN) designed to capture multi-hop content and consumption relationships across a range of recommendable item types. To ensure the generality required from a Foundation Model, we employ a LLM text-based featurization of nodes that accommodates all item types, and construct the graph using co-interaction signals, which inherently transcend content specificity. To facilitate practical generalization, we further couple the HGNN with an adaptation mechanism based on a two-tower (2T) architecture, which also operates agnostically to content type. This multi-stage approach ensures high scalability; while the HGNN produces general purpose embeddings, the 2T component models in a continuous space the sheer size of user-item interaction data. Our comprehensive approach has been rigorously tested and proven effective in delivering recommendations across a diverse array of products within a real-world, industrial audio streaming platform.
- On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Personalized audiobook recommendations at spotify through graph neural networks. In Proceedings of the ACM Web Conference 2024, Applied Track, 2024.
- Twhin: Embedding the twitter heterogeneous information network for personalized recommendation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 2842–2850, 2022.
- Relational deep learning: Graph representation learning on relational databases. arXiv preprint arXiv:2312.04615, 2023.
- Towards foundation models for knowledge graph reasoning. arXiv preprint arXiv:2310.04562, 2023.
- Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07160, 2023.
- Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pages 299–315, 2022.
- Grale: Designing networks for graph learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2523–2532, 2020.
- Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
- Prodigy: Enabling in-context learning over graphs, 2023.
- Efficient and effective training of language and graph neural network models. arXiv preprint arXiv:2206.10781, 2022.
- Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310.11829, 2023.
- Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023.
- Llm-rec: Personalized recommendation via prompting large language models. arXiv preprint arXiv:2307.15780, 2023.
- Graph foundation models, 2024.
- Graph learning for exploratory query suggestions in an instant search system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4780–4786, 2023.
- Florence-2: Advancing a unified representation for a variety of vision tasks. arXiv preprint arXiv:2311.06242, 2023.
- Graph-aware language model pre-training on a large graph corpus can help multiple graph applications. arXiv preprint arXiv:2306.02592, 2023.
- Deep bidirectional language-knowledge graph pretraining. Advances in Neural Information Processing Systems, 35:37309–37323, 2022.
- Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems, pages 269–277, 2019.
- Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 974–983, 2018.
- Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
- Greaselm: Graph reasoning enhanced language models. In International conference on learning representations, 2021.
- Collm: Integrating collaborative embeddings into large language models for recommendation. arXiv preprint arXiv:2310.19488, 2023.
- Andreas Damianou (28 papers)
- Francesco Fabbri (22 papers)
- Paul Gigioli (2 papers)
- Marco De Nadai (26 papers)
- Alice Wang (9 papers)
- Enrico Palumbo (8 papers)
- Mounia Lalmas (25 papers)