Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unifying Graph Contrastive Learning via Graph Message Augmentation (2401.03638v1)

Published 8 Jan 2024 in cs.LG and cs.CV

Abstract: Graph contrastive learning is usually performed by first conducting Graph Data Augmentation (GDA) and then employing a contrastive learning pipeline to train GNNs. As we know that GDA is an important issue for graph contrastive learning. Various GDAs have been developed recently which mainly involve dropping or perturbing edges, nodes, node attributes and edge attributes. However, to our knowledge, it still lacks a universal and effective augmentor that is suitable for different types of graph data. To address this issue, in this paper, we first introduce the graph message representation of graph data. Based on it, we then propose a novel Graph Message Augmentation (GMA), a universal scheme for reformulating many existing GDAs. The proposed unified GMA not only gives a new perspective to understand many existing GDAs but also provides a universal and more effective graph data augmentation for graph self-supervised learning tasks. Moreover, GMA introduces an easy way to implement the mixup augmentor which is natural for images but usually challengeable for graphs. Based on the proposed GMA, we then propose a unified graph contrastive learning, termed Graph Message Contrastive Learning (GMCL), that employs attribution-guided universal GMA for graph contrastive learning. Experiments on many graph learning tasks demonstrate the effectiveness and benefits of the proposed GMA and GMCL approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. H. Sun, J. Liu, and J. Zhang, “A survey of contrastive learning in nlp,” in International Symposium on Advances in Electrical, Electronics, and Computer Engineering, vol. 12294, pp. 1073–1078, SPIE, 2022.
  2. P. H. Le-Khac, G. Healy, and A. F. Smeaton, “Contrastive representation learning: A framework and review,” IEEE Access, vol. 8, pp. 193907–193934, 2020.
  3. X. Wang and G.-J. Qi, “Contrastive learning with stronger augmentations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5549–5560, 2023.
  4. Y. Wang, J. Li, H. Wang, Y. Qian, C. Wang, and Y. Wu, “Wav2vec-switch: Contrastive learning from original-noisy speech pairs for robust speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7097–7101, 2022.
  5. J. Giorgi, O. Nitski, B. Wang, and G. Bader, “Declutr: Deep contrastive learning for unsupervised textual representations,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 879–895, 2021.
  6. Y. Zhu, Y. Xu, Q. Liu, and S. Wu, “An empirical study of graph contrastive learning,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  7. Y. Xie, Z. Xu, J. Zhang, Z. Wang, and S. Ji, “Self-supervised learning of graph neural networks: A unified review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2412–2429, 2023.
  8. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017.
  9. B. Wang, B. Jiang, J. Tang, and B. Luo, “Generalizing aggregation functions in gnns: Building high capacity and robust gnns via nonlinear aggregation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13454–13466, 2023.
  10. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
  11. B. Jiang, B. Wang, S. Chen, J. Tang, and B. Luo, “Graph neural network meets sparse representation: Graph sparse neural networks via exclusive group lasso,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12692–12698, 2023.
  12. D. Chen, L. O’Bray, and K. Borgwardt, “Structure-aware transformer for graph representation learning,” in International Conference on Machine Learning, pp. 3469–3489, PMLR, 2022.
  13. Z. Ma, P. Zhu, S. Lv, X. Li, C. Zhao, and J. Wang, “Dual self-attention transformer for graph node classification,” in International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–6, 2023.
  14. H. Li, X. Wang, Z. Zhang, Z. Yuan, H. Li, and W. Zhu, “Disentangled contrastive learning on graphs,” in Advances in Neural Information Processing Systems, vol. 34, pp. 21872–21884, 2021.
  15. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” in Advances in Neural Information Processing Systems, vol. 33, pp. 5812–5823, 2020.
  16. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Deep graph contrastive representation learning,” arXiv preprint arXiv:2006.04131, 2020.
  17. H. Zhang, Q. Wu, J. Yan, D. Wipf, and P. S. Yu, “From canonical correlation analysis to self-supervised graph neural networks,” in Advances in Neural Information Processing Systems, vol. 34, pp. 76–89, 2021.
  18. W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for pre-training graph neural networks,” in International Conference on Learning Representations (ICLR), 2020.
  19. Z. Zhang, Q. Liu, H. Wang, C. Lu, and C.-K. Lee, “Motif-based graph self-supervised learning for molecular property prediction,” in Advances in Neural Information Processing Systems, vol. 34, pp. 15870–15882, 2021.
  20. F. Manessi and A. Rozza, “Graph-based neural network models with multiple self-supervised auxiliary tasks,” Pattern Recognition Letters, vol. 148, pp. 15–21, 2021.
  21. X. Han, Z. Jiang, N. Liu, and X. Hu, “G-mixup: Graph data augmentation for graph classification,” in International Conference on Machine Learning, vol. 162, pp. 8230–8248, 17–23 Jul 2022.
  22. Y. Wang, W. Wang, Y. Liang, Y. Cai, and B. Hooi, “Mixup for node and graph classification,” in Proceedings of the Web Conference 2021, pp. 3663–3674, 2021.
  23. Y. Yin, Q. Wang, S. Huang, H. Xiong, and X. Zhang, “Autogcl: Automated graph contrastive learning via learnable view generators,” in the AAAI Conference on Artificial Intelligence, vol. 36, pp. 8892–8900, 2022.
  24. S. Suresh, P. Li, C. Hao, and J. Neville, “Adversarial graph augmentation to improve graph contrastive learning,” in Advances in Neural Information Processing Systems, vol. 34, pp. 15920–15933, 2021.
  25. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Graph contrastive learning with adaptive augmentation,” in Proceedings of the Web Conference 2021, pp. 2069–2080, 2021.
  26. Y. You, T. Chen, Z. Wang, and Y. Shen, “Bringing your own view: Graph contrastive learning without prefabricated data augmentations,” in International Conference on Web Search and Data Mining, pp. 1300–1309, 2022.
  27. Y. Jiao, Y. Xiong, J. Zhang, Y. Zhang, T. Zhang, and Y. Zhu, “Sub-graph contrast for scalable self-supervised graph representation learning,” in 2020 IEEE international conference on data mining (ICDM), pp. 222–231, IEEE, 2020.
  28. Y. You, T. Chen, Y. Shen, and Z. Wang, “Graph contrastive learning automated,” in International Conference on Machine Learning, vol. 139, pp. 12121–12132, 2021.
  29. Y. Luo, M. C. McThrow, W. Y. Au, T. Komikado, K. Uchino, K. Maruhashi, and S. Ji, “Automated data augmentations for graph classification,” in The Eleventh International Conference on Learning Representations, 2023.
  30. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International conference on machine learning, pp. 1263–1272, 2017.
  31. T. Fang, Z. Xiao, C. Wang, J. Xu, X. Yang, and Y. Yang, “Dropmessage: Unifying random dropping for graph neural networks,” arXiv preprint arXiv:2204.10037, 2022.
  32. W. Feng, J. Zhang, Y. Dong, Y. Han, H. Luan, Q. Xu, Q. Yang, E. Kharlamov, and J. Tang, “Graph random neural networks for semi-supervised learning on graphs,” in Advances in Neural Information Processing Systems, vol. 33, pp. 22092–22103, 2020.
  33. F.-Y. Sun, J. Hoffman, V. Verma, and J. Tang, “Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization,” in International Conference on Learning Representations, 2020.
  34. P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, “Deep graph infomax,” in International Conference on Learning Representations, 2018.
  35. Y. Ren and B. Liu, “Heterogeneous deep graph infomax,” in Workshop of Deep Learning on Graphs: Methodologies and Applications co-located with the Thirty-Fourth AAAI Confer-ence on Artificial Intelligence, 2020.
  36. F. L. Opolka, A. Solomon, C. Cangea, P. Veličković, P. Liò, and R. D. Hjelm, “Spatio-temporal deep graph infomax,” in Proc. Int. Conf. Learn. Representations Workshop, 2019.
  37. M. Xu, H. Wang, B. Ni, H. Guo, and J. Tang, “Self-supervised graph-level representation learning with local and global structure,” in International Conference on Machine Learning, pp. 11548–11558, PMLR, 2021.
  38. S. Lin, C. Liu, P. Zhou, Z.-Y. Hu, S. Wang, R. Zhao, Y. Zheng, L. Lin, E. Xing, and X. Liang, “Prototypical graph contrastive learning,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2022.
  39. M. Peng, X. Juan, and Z. Li, “Graph prototypical contrastive learning,” Information Sciences, vol. 612, pp. 816–834, 2022.
  40. C. Wang, S. Pan, G. Long, X. Zhu, and J. Jiang, “Mgae: Marginalized graph autoencoder for graph clustering,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, p. 889–898, 2017.
  41. K. Hassani and A. H. Khasahmadi, “Contrastive multi-view representation learning on graphs,” in International conference on machine learning, pp. 4116–4126, PMLR, 2020.
  42. X. Xia, H. Yin, J. Yu, Q. Wang, L. Cui, and X. Zhang, “Self-supervised hypergraph convolutional networks for session-based recommendation,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, pp. 4503–4511, 2021.
  43. X. Xu, C. Deng, Y. Xie, and S. Ji, “Group contrastive self-supervised learning on graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3169–3180, 2023.
  44. Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning?,” Advances in neural information processing systems, vol. 33, pp. 6827–6839, 2020.
  45. A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, “Deep variational information bottleneck,” in International Conference on Learning Representations, 2016.
  46. T. Zhao, Y. Liu, L. Neves, O. Woodford, M. Jiang, and N. Shah, “Data augmentation for graph neural networks,” in Proceedings of the aaai conference on artificial intelligence, vol. 35, pp. 11015–11023, 2021.
  47. T. Zhao, X. Tang, D. Zhang, H. Jiang, N. Rao, Y. Song, P. Agrawal, K. Subbian, B. Yin, and M. Jiang, “Autogda: Automated graph data augmentation for node classification,” in Learning on Graphs Conference, vol. 198, pp. 1–17, 2022.
  48. K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?,” in The International Conference on Learning Representations, 2019.
  49. D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, “Convolutional networks on graphs for learning molecular fingerprints,” Advances in neural information processing systems, vol. 28, 2015.
  50. L. Wu, J. Xia, Z. Gao, H. Lin, C. Tan, and S. Z. Li, “Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 519–535, Springer, 2022.
  51. S. Mo, H. Kang, K. Sohn, C.-L. Li, and J. Shin, “Object-aware contrastive learning for debiased scene representation,” Advances in Neural Information Processing Systems, vol. 34, pp. 12251–12264, 2021.
  52. Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li, “Efficient attention: Attention with linear complexities,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3531–3539, 2021.
  53. Y. Hao, L. Dong, F. Wei, and K. Xu, “Self-attention attribution: Interpreting information interactions inside transformer,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 12963–12971, 2021.
  54. T. Yang, J. Deng, X. Quan, Q. Wang, and S. Nie, “Ad-drop: Attribution-driven dropout for robust language model fine-tuning,” in Advances in Neural Information Processing Systems, vol. 35, pp. 12310–12324, 2022.
  55. E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in International Conference on Learning Representations, 2016.
  56. I. A. M. Huijben, W. Kool, M. B. Paulus, and R. J. G. van Sloun, “A review of the gumbel-max trick and its extensions for discrete stochasticity in machine learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 1353–1371, 2023.
  57. K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” Advances in neural information processing systems, vol. 29, 2016.
  58. C. Morris, N. M. Kriege, F. Bause, K. Kersting, P. Mutzel, and M. Neumann, “Tudataset: A collection of benchmark datasets for learning with graphs,” in ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020.
  59. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations, 2015.
  60. N. Shervashidze, S. Vishwanathan, T. Petri, K. Mehlhorn, and K. Borgwardt, “Efficient graphlet kernels for large graph comparison,” in Artificial intelligence and statistics, pp. 488–495, 2009.
  61. N. Shervashidze, P. Schweitzer, E. J. Van Leeuwen, K. Mehlhorn, and K. M. Borgwardt, “Weisfeiler-lehman graph kernels,” Journal of Machine Learning Research, vol. 12, no. 9, pp. 2539–2561, 2011.
  62. P. Yanardag and S. Vishwanathan, “Deep graph kernels,” in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 1365–1374, 2015.
  63. A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855–864, 2016.
  64. B. Adhikari, Y. Zhang, N. Ramakrishnan, and B. A. Prakash, “Sub2vec: Feature learning for subgraphs,” in Advances in Knowledge Discovery and Data Mining, pp. 170–182, 2018.
  65. A. Narayanan, M. Chandramohan, R. Venkatesan, L. Chen, Y. Liu, and S. Jaiswal, “graph2vec: Learning distributed representations of graphs,” arXiv preprint arXiv:1707.05005, 2017.
  66. G. Li, M. Muller, A. Thabet, and B. Ghanem, “Deepgcns: Can gcns go as deep as cnns?,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  67. A. Gaulton, L. J. Bellis, A. P. Bento, J. Chambers, M. Davies, A. Hersey, Y. Light, S. McGlinchey, D. Michalovich, B. Al-Lazikani, et al., “Chembl: a large-scale bioactivity database for drug discovery,” Nucleic acids research, vol. 40, no. D1, pp. D1100–D1107, 2012.
  68. P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, “Collective classification in network data,” AI magazine, vol. 29, no. 3, p. 93, 2008.
  69. Y. Li, W. Jin, H. Xu, and J. Tang, “Deeprobust: a platform for adversarial attacks and defenses,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 16078–16080, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now