Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Perfect Alignment May be Poisonous to Graph Contrastive Learning (2310.03977v2)

Published 6 Oct 2023 in cs.LG and cs.AI

Abstract: Graph Contrastive Learning (GCL) aims to learn node representations by aligning positive pairs and separating negative ones. However, few of researchers have focused on the inner law behind specific augmentations used in graph-based learning. What kind of augmentation will help downstream performance, how does contrastive learning actually influence downstream tasks, and why the magnitude of augmentation matters so much? This paper seeks to address these questions by establishing a connection between augmentation and downstream performance. Our findings reveal that GCL contributes to downstream tasks mainly by separating different classes rather than gathering nodes of the same class. So perfect alignment and augmentation overlap which draw all intra-class samples the same can not fully explain the success of contrastive learning. Therefore, in order to understand how augmentation aids the contrastive learning process, we conduct further investigations into the generalization, finding that perfect alignment that draw positive pair the same could help contrastive loss but is poisonous to generalization, as a result, perfect alignment may not lead to best downstream performance, so specifically designed augmentation is needed to achieve appropriate alignment performance and improve downstream accuracy. We further analyse the result by information theory and graph spectrum theory and propose two simple but effective methods to verify the theories. The two methods could be easily applied to various GCL algorithms and extensive experiments are conducted to prove its effectiveness. The code is available at https://github.com/somebodyhh1/GRACEIS

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
  2. Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106.09943, 2021.
  3. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815, 2017.
  4. Further reverse results for jensen’s discrete inequality and applications in information theory. RGMIA research report collection, 3(1), 2000.
  5. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020a.
  6. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243–22255, 2020b.
  7. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.  1263–1272. PMLR, 06–11 Aug 2017.
  8. Contrastive multi-view representation learning on graphs. In International conference on machine learning, pp. 4116–4126. PMLR, 2020.
  9. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020a.
  10. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp.  639–648, 2020b.
  11. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.00743, 2021.
  12. Semi-supervised learning with graph learning-convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11313–11320, 2019.
  13. Advanced data augmentation approaches: A comprehensive survey and future directions. arXiv preprint arXiv:2301.02830, 2023.
  14. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems, 34:309–323, 2021.
  15. Let invariant rationale discovery inspire graph contrastive learning. In International conference on machine learning, pp. 13052–13065. PMLR, 2022.
  16. Spectral augmentation for self-supervised learning on graphs. arXiv preprint arXiv:2210.00643, 2022.
  17. Revisiting graph contrastive learning from the perspective of graph spectrum. Advances in Neural Information Processing Systems, 35:2972–2983, 2022.
  18. Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
  19. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  20. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pp. 19250–19286. PMLR, 2022.
  21. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  22. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.
  23. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  1711–1719, 2020.
  24. Gilbert W Stewart. Matrix perturbation theory. 1990.
  25. Adversarial graph augmentation to improve graph contrastive learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  15920–15933. Curran Associates, Inc., 2021.
  26. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827–6839, 2020.
  27. Understanding self-supervised graph representation learning from a data-centric perspective. 2022.
  28. Deep graph infomax. arXiv preprint arXiv:1809.10341, 2018.
  29. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16041–16050, 2022a.
  30. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929–9939. PMLR, 2020.
  31. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457, 2022b.
  32. Boosting graph contrastive learning via graph contrastive saliency. In International Conference on Machine Learning, pp. 36839–36855. PMLR, 2023.
  33. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121, 2019.
  34. Infogcl: Information-aware graph contrastive learning, 2021.
  35. A new perspective on the effects of spectrum in graph neural networks. In International Conference on Machine Learning, pp. 25261–25279. PMLR, 2022.
  36. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pp.  40–48. PMLR, 2016.
  37. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812–5823, 2020.
  38. Towards generalizable graph contrastive learning: An information theory perspective. arXiv preprint arXiv:2211.10929, 2022.
  39. Graph data augmentation for graph machine learning: A survey. arXiv preprint arXiv:2202.08871, 2022a.
  40. Graph data augmentation for graph machine learning: A survey. arXiv preprint arXiv:2202.08871, 2022b.
  41. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131, 2020.
  42. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pp.  2069–2080, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Jingyu Liu (53 papers)
  2. Huayi Tang (12 papers)
  3. Yong Liu (721 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now