Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning (2310.07365v3)

Published 11 Oct 2023 in cs.LG

Abstract: Graph-structured data is ubiquitous in the world which models complex relationships between objects, enabling various Web applications. Daily influxes of unlabeled graph data on the Web offer immense potential for these applications. Graph self-supervised algorithms have achieved significant success in acquiring generic knowledge from abundant unlabeled graph data. These pre-trained models can be applied to various downstream Web applications, saving training time and improving downstream (target) performance. However, different graphs, even across seemingly similar domains, can differ significantly in terms of attribute semantics, posing difficulties, if not infeasibility, for transferring the pre-trained models to downstream tasks. Concretely speaking, for example, the additional task-specific node information in downstream tasks (specificity) is usually deliberately omitted so that the pre-trained representation (transferability) can be leveraged. The trade-off as such is termed as "transferability-specificity dilemma" in this work. To address this challenge, we introduce an innovative deployment module coined as GraphControl, motivated by ControlNet, to realize better graph domain transfer learning. Specifically, by leveraging universal structural pre-trained models and GraphControl, we align the input space across various graphs and incorporate unique characteristics of target data as conditional inputs. These conditions will be progressively integrated into the model during fine-tuning or prompt tuning through ControlNet, facilitating personalized deployment. Extensive experiments show that our method significantly enhances the adaptability of pre-trained models on target attributed datasets, achieving 1.4-3x performance gain. Furthermore, it outperforms training-from-scratch methods on target data with a comparable margin and exhibits faster convergence.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Group formation in large social networks: membership, growth, and evolution. In Proc. of KDD.
  2. Vladimir Batagelj. 2003. Efficient algorithms for citation network analysis. arXiv preprint cs/0309023 (2003).
  3. Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking. In Proc. of ICLR.
  4. Chen Cai and Yusu Wang. 2020. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 (2020).
  5. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  6. Adaptive graph encoder for attributed graph embedding. In Proc. of KDD.
  7. Prompt tuning for graph neural networks. arXiv preprint arXiv:2209.15240 (2022).
  8. Victor Garcia and Joan Bruna. 2017. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043 (2017).
  9. Bootstrap your own latent-a new approach to self-supervised learning. Proc. of NeurIPS (2020).
  10. Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proc. of KDD.
  11. Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In Proc. of ICML.
  12. Graphmae: Self-supervised masked graph autoencoders. In Proc. of KDD.
  13. Variational diffusion models. Proc. of NeurIPS (2021).
  14. Thomas N Kipf and Max Welling. 2016. Variational Graph Auto-Encoders. NIPS Workshop on Bayesian Deep Learning (2016).
  15. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences (2017).
  16. Learning the kernel matrix with semidefinite programming. Journal of Machine learning research (2004).
  17. Transfer learning for deep learning on graph-structured data. In Proc. of AAAI.
  18. Overcoming catastrophic forgetting in graph neural networks. In Proc. of AAAI.
  19. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023.
  20. Image-based recommendations on styles and substitutes. In Proc. of SIGIR.
  21. Mark EJ Newman and Michelle Girvan. 2004. Finding and evaluating community structure in networks. Physical review E (2004).
  22. Alleviate Representation Overlapping in Class Incremental Learning by Contrastive Class Concentration. CoRR (2021).
  23. Continual Vision-Language Representation Learning with Off-Diagonal Information. In Proc. of ICML.
  24. Recommendation quality, transparency, and website quality for trust-building in recommendation agents. Electronic Commerce Research and Applications (2016).
  25. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).
  26. Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering (2009).
  27. Gcc: Graph contrastive coding for graph neural network pre-training. In Proc. of KDD.
  28. struc2vec: Learning node representations from structural identity. In Proc. of KDD.
  29. Martin Riedmiller. 1994. Advanced supervised learning in multi-layer perceptrons—from backpropagation to adaptive learning algorithms. Computer Standards & Interfaces (1994).
  30. A scalable permutation approach reveals replication and preservation patterns of network modules in large datasets. Cell systems (2016).
  31. High-resolution image synthesis with latent diffusion models. In Proc. of CVPR.
  32. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 (2017).
  33. Collective classification in network data. AI magazine (2008).
  34. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018).
  35. How to fine-tune bert for text classification?. In Proc. of CCL.
  36. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 (2019).
  37. GPPT: Graph Pre-Training and Prompt Tuning to Generalize Graph Neural Networks. In Proc. of KDD.
  38. All in One: Multi-Task Prompting for Graph Neural Networks. In Proc. of KDD.
  39. Deep Graph Infomax. In Proc. of ICLR.
  40. Unsupervised domain adaptive graph convolutional networks. In Proc. of WWW.
  41. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022.
  42. How Powerful are Graph Neural Networks?. In Proc. of ICLR.
  43. Design space for graph neural networks. Proc. of NeurIPS (2020).
  44. Universal domain adaptation. In Proc. of CVPR.
  45. Graph contrastive learning automated. In Proc. of ICML.
  46. Graph contrastive learning with augmentations. Proc. of NeurIPS (2020).
  47. Oag: Toward linking large-scale heterogeneous entity graphs. In Proc. of KDD.
  48. Lvmin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 (2023).
  49. COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning. In Proc. of KDD.
  50. Transfer learning of graph neural networks with ego-graph information maximization. Proc. of NeurIPS (2021).
  51. SGL-PT: A Strong Graph Learner with Graph Prompt Tuning. arXiv preprint arXiv:2302.12449 (2023).
  52. RoSA: A Robust Self-Aligned Framework for Node-Node Graph Contrastive Learning. In Proc. of IJCAI.
  53. MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning. arXiv preprint arXiv:2307.13055 (2023).
  54. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond.
  55. A comprehensive survey on transfer learning. Proc. IEEE (2020).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yun Zhu (52 papers)
  2. Yaoke Wang (8 papers)
  3. Haizhou Shi (25 papers)
  4. Zhenshuo Zhang (3 papers)
  5. Dian Jiao (10 papers)
  6. Siliang Tang (116 papers)
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now