Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Smoothed Graph Contrastive Learning via Seamless Proximity Integration (2402.15270v2)

Published 23 Feb 2024 in cs.LG and cs.AI

Abstract: Graph contrastive learning (GCL) aligns node representations by classifying node pairs into positives and negatives using a selection process that typically relies on establishing correspondences within two augmented graphs. The conventional GCL approaches incorporate negative samples uniformly in the contrastive loss, resulting in the equal treatment of negative nodes, regardless of their proximity to the true positive. In this paper, we present a Smoothed Graph Contrastive Learning model (SGCL), which leverages the geometric structure of augmented graphs to inject proximity information associated with positive/negative pairs in the contrastive loss, thus significantly regularizing the learning process. The proposed SGCL adjusts the penalties associated with node pairs in contrastive loss by incorporating three distinct smoothing techniques that result in proximity-aware positives and negatives. To enhance scalability for large-scale graphs, the proposed framework incorporates a graph batch-generating strategy that partitions the given graphs into multiple subgraphs, facilitating efficient training in separate batches. Through extensive experimentation in the unsupervised setting on various benchmarks, particularly those of large scale, we demonstrate the superiority of our proposed framework against recent baselines.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Geometric multimodal deep learning with multiscaled graph wavelet convolutional network. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2022. doi: 10.1109/TNNLS.2022.3213589.
  2. TIDE: Time derivative diffusion for deep learning on graphs. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  2015–2030. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/behmanesh23a.html.
  3. Graph neural networks with convolutional arma filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  4. Graph barlow twins: A self-supervised representation learning framework for graphs. Knowledge-Based Systems, 256:109631, 2022. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2022.109631. URL https://www.sciencedirect.com/science/article/pii/S095070512200822X.
  5. Protein function prediction via graph kernels. volume 21, pp.  i47–i56, 06 2005. doi: 10.1093/bioinformatics/bti1007. URL https://doi.org/10.1093/bioinformatics/bti1007.
  6. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017. doi: 10.1109/MSP.2017.2693418.
  7. GRAND: graph neural diffusion. In International Conference on Machine Learning, pp.  1407–1418. PMLR, 2021.
  8. Self-supervised contrastive graph representation with node and graph augmentation. Neural Networks, 167:223–232, 2023. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2023.08.039. URL https://www.sciencedirect.com/science/article/pii/S0893608023004598.
  9. Nonlinear anisotropic filtering of mri data. IEEE Transactions on Medical Imaging, 11(2):221–232, 1992. doi: 10.1109/42.141646.
  10. Neural message passing for quantum chemistry. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.  1263–1272. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/gilmer17a.html.
  11. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
  12. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
  13. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, 2017.
  14. Contrastive multi-view representation learning on graphs. In Proceedings of International Conference on Machine Learning, pp.  3451–3461. 2020.
  15. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  9726–9735, 2020. doi: 10.1109/CVPR42600.2020.00975.
  16. Open graph benchmark: Datasets for machine learning on graphs. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
  17. Homophily influences ranking of minorities in social networks. Scientific Reports, 8:1–12, 2018. ISSN 2045-2322. doi: https://doi.org/10.1038/s41598-018-29405-7.
  18. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.
  19. Subgraph matching kernels for attributed graphs. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML’12, pp.  291–298, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
  20. Graph matching networks for learning the similarity of graph structured objects. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  3835–3845. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/li19d.html.
  21. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 35(1):857–876, 2023. doi: 10.1109/TKDE.2021.3090866.
  22. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’15, pp.  43–52, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336215. doi: 10.1145/2766462.2767755. URL https://doi.org/10.1145/2766462.2767755.
  23. Asymmetric transitivity preserving graph embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  1105–1114. ACM, 2016.
  24. Grlc: Graph representation learning with constraints. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–14, 2023. doi: 10.1109/TNNLS.2022.3230979.
  25. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014.
  26. Collective classification in network data. AI Magazine, 29(3):93, Sep. 2008. doi: 10.1609/aimag.v29i3.2157. URL https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2157.
  27. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15 Companion, pp.  243–246, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334730. doi: 10.1145/2740908.2742839. URL https://doi.org/10.1145/2740908.2742839.
  28. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
  29. Adversarial graph augmentation to improve graph contrastive learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=ioyq7NsR1KJ.
  30. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, 2015.
  31. Taubin, G. A Signal Processing Approach To Fair Surface Design. Association for Computing Machinery, New York, NY, USA, 1 edition, 2023. ISBN 9798400708978. URL https://doi.org/10.1145/3596711.3596724.
  32. Large-scale representation learning on graphs via bootstrapping. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=0UXT6PpRpW.
  33. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), pp.  839–846, 1998. doi: 10.1109/ICCV.1998.710815.
  34. Deep Graph Infomax. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rklz9iAcKQ.
  35. Graph attention networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJXMpikCZ.
  36. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowl. Inf. Syst., 14(3):347–375, mar 2008. ISSN 0219-1377. doi: 10.1007/s10115-007-0103-5. URL https://doi.org/10.1007/s10115-007-0103-5.
  37. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 35(4):4216–4235, 2023. doi: 10.1109/TKDE.2021.3131584.
  38. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, 2021. doi: 10.1109/TNNLS.2020.2978386.
  39. Progcl: Rethinking hard negative mining in graph contrastive learning. In International Conference on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:249282506.
  40. Self-supervised representation learning via latent graph prediction. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  24460–24477. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/xie22e.html.
  41. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2412–2429, 2023. doi: 10.1109/TPAMI.2022.3170559.
  42. Graph wavelet neural network. In International Conference on Learning Representations, 2019a.
  43. How powerful are graph neural networks? In International Conference on Learning Representations, 2019b. URL https://openreview.net/forum?id=ryGs6iA5Km.
  44. Deep graph kernels. KDD ’15, pp.  1365–1374, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336642. doi: 10.1145/2783258.2783417. URL https://doi.org/10.1145/2783258.2783417.
  45. Graph contrastive learning with augmentations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  5812–5823. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/3fe230348e9a12c13120749e3f9fa4cd-Paper.pdf.
  46. Graphsaint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=BJe8pkHFwS.
  47. Contrastive self-supervised learning for graph classification. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10824–10832, May 2021. doi: 10.1609/aaai.v35i12.17293. URL https://ojs.aaai.org/index.php/AAAI/article/view/17293.
  48. Transfer learning of graph neural networks with ego-graph information maximization. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021a. URL https://openreview.net/forum?id=CzVPfeqPOBu.
  49. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond, 2020. URL http://arxiv.org/abs/2006.04131.
  50. An Empirical Study of Graph Contrastive Learning. arXiv.org, September 2021b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now