Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks (2405.13085v1)

Published 21 May 2024 in cs.CL and cs.AI

Abstract: Knowledge graphs (KGs) provide reliable external knowledge for a wide variety of AI tasks in the form of structured triples. Knowledge graph pre-training (KGP) aims to pre-train neural networks on large-scale KGs and provide unified interfaces to enhance different downstream tasks, which is a key direction for KG management, maintenance, and applications. Existing works often focus on purely research questions in open domains, or they are not open source due to data security and privacy in real scenarios. Meanwhile, existing studies have not explored the training efficiency and transferability of KGP models in depth. To address these problems, We propose a framework MuDoK to achieve multi-domain collaborative pre-training and efficient prefix prompt tuning to serve diverse downstream tasks like recommendation and text understanding. Our design is a plug-and-play prompt learning approach that can be flexibly adapted to different downstream task backbones. In response to the lack of open-source benchmarks, we constructed a new multi-domain KGP benchmark called KPI with two large-scale KGs and six different sub-domain tasks to evaluate our method and open-sourced it for subsequent research. We evaluated our approach based on constructed KPI benchmarks using diverse backbone models in heterogeneous downstream tasks. The experimental results show that our framework brings significant performance gains, along with its generality, efficiency, and transferability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Healthcare knowledge graph construction: A systematic review of the state-of-the-art, open issues, and opportunities. Journal of Big Data 10, 1 (2023), 81.
  2. Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences. In WWW. ACM, 151–161.
  3. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. In AAAI, Vol. 34. 27–34.
  4. KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction. In WWW. ACM, 2778–2788.
  5. Tele-Knowledge Pre-training for Fault Analysis. In ICDE. IEEE, 3453–3466.
  6. Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey. CoRR abs/2402.05391 (2024).
  7. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). Association for Computational Linguistics, 4171–4186.
  8. Towards foundation models for knowledge graph reasoning. arXiv preprint arXiv:2310.04562 (2023).
  9. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In EMNLP (1). Association for Computational Linguistics, 6894–6910.
  10. Deep Sparse Rectifier Neural Networks. In AISTATS (JMLR Proceedings, Vol. 15). JMLR.org, 315–323.
  11. A Survey on Vision Transformer. IEEE Trans. Pattern Anal. Mach. Intell. 45, 1 (2023), 87–110.
  12. Towards a Unified View of Parameter-Efficient Transfer Learning. In ICLR. OpenReview.net.
  13. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR. ACM, 639–648.
  14. Neural collaborative filtering. In WWW. 173–182.
  15. Bridging Language and Items for Retrieval and Recommendation. arXiv preprint arXiv:2403.03952 (2024).
  16. Leveraging Meta-path based Context for Top- N Recommendation with A Neural Co-Attention Model. In KDD. ACM, 1531–1540.
  17. A Survey of Knowledge Enhanced Pre-Trained Language Models. IEEE Trans. Knowl. Data Eng. 36, 4 (2024), 1413–1430.
  18. Collaborative filtering for implicit feedback datasets. In ICDM. Ieee, 263–272.
  19. DiffKG: Knowledge Graph Diffusion Model for Recommendation. In WSDM. ACM, 313–321.
  20. K-BERT: Enabling Language Representation with Knowledge Graph. In AAAI. AAAI Press, 2901–2908.
  21. Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries. In KDD. ACM, 1120–1130.
  22. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019).
  23. Unifying Large Language Models and Knowledge Graphs: A Roadmap. CoRR abs/2306.08302 (2023).
  24. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  25. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  26. SSLRec: A Self-Supervised Learning Framework for Recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 567–575.
  27. Disentangled Contrastive Collaborative Filtering. In SIGIR. 1137–1146.
  28. Graph Prompt Learning: A Comprehensive Survey and Beyond. CoRR abs/2311.16534 (2023).
  29. Graph Neural Prompting with Large Language Models. In AAAI. AAAI Press, 19080–19088.
  30. Attention is All you Need. In NIPS. 5998–6008.
  31. DKN: Deep Knowledge-Aware Network for News Recommendation. In WWW. ACM, 1835–1844.
  32. Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation. In WWW. ACM, 2000–2010.
  33. Knowledge Graph Convolutional Networks for Recommender Systems. In WWW. ACM, 3307–3313.
  34. Knowledge Graph Embedding: A Survey of Approaches and Applications. IEEE Trans. Knowl. Data Eng. 29, 12 (2017), 2724–2743. https://doi.org/10.1109/TKDE.2017.2754499
  35. KGAT: Knowledge Graph Attention Network for Recommendation. In KDD. ACM, 950–958.
  36. Learning Intents behind Interactions with Knowledge Graph for Recommendation. In WWW. ACM / IW3C2, 878–887.
  37. Explainable Reasoning over Knowledge Graphs for Recommendation. In AAAI. AAAI Press, 5329–5336.
  38. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38–45. https://www.aclweb.org/anthology/2020.emnlp-demos.6
  39. Improving Conversational Recommender System by Pretraining Billion-scale Knowledge Graph. In ICDE. IEEE, 2607–2612.
  40. Knowledge Graph Self-Supervised Rationalization for Recommendation. In KDD. ACM, 3046–3056.
  41. Knowledge Graph Contrastive Learning for Recommendation. In SIGIR. ACM, 1434–1443.
  42. Deep Bidirectional Language-Knowledge Graph Pretraining. In NeurIPS.
  43. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In NAACL-HLT. Association for Computational Linguistics, 535–546.
  44. Improving knowledge graph representation learning by structure contextual pre-training. In Proceedings of the 10th International Joint Conference on Knowledge Graphs. 151–155.
  45. A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models. CoRR abs/2202.08772 (2022).
  46. JAKET: Joint Pre-training of Knowledge Graph and Language Understanding. In AAAI. AAAI Press, 11630–11638.
  47. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In SIGIR. 1294–1303.
  48. Personalized entity recommendation: a heterogeneous information network approach. In WSDM. ACM, 283–292.
  49. Collaborative Knowledge Base Embedding for Recommender Systems. In KDD. ACM, 353–362.
  50. PKGM: A Pre-trained Knowledge Graph Model for E-commerce Application. CoRR abs/2203.00964 (2022).
  51. Billion-scale Pre-trained E-commerce Product Knowledge Graph Model. In ICDE. IEEE, 2476–2487.
  52. Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer. In WWW. ACM, 2581–2590.
  53. GreaseLM: Graph REASoning Enhanced Language Models for Question Answering. CoRR abs/2201.08860 (2022).
  54. Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering. CoRR abs/2311.06503 (2023).
  55. A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation. In IJCAI. ijcai.org, 3001–3008.
  56. A Unified Framework for Cross-Domain and Cross-System Recommendations. IEEE Trans. Knowl. Data Eng. 35, 2 (2023), 1171–1184.
  57. Knowledge Perceived Multi-modal Pretraining in E-commerce. In ACM Multimedia. ACM, 2744–2752.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets