Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision (2403.05064v1)

Published 8 Mar 2024 in cs.LG and cs.AI

Abstract: The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing to handle ubiquitous scenarios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due to the nature of the graph and the complexity of the neural architecture search process. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Specifically, we first design a disentangled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which are optimized simultaneously. Then, we estimate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-graph disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (129)
  1. Automated machine learning on graphs: A survey. arXiv preprint arXiv:2103.00742, 2021.
  2. Autograph: Automated graph neural network. In International Conference on Neural Information Processing, pages 189–201, 2020.
  3. Graph neural architecture search. In IJCAI, volume 20, pages 1403–1409, 2020.
  4. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2091–2100, 2021.
  5. How much do clinical trials cost. Nat Rev Drug Discov, 16(6):381–382, 2017.
  6. Artificial intelligence in drug discovery and development. Drug discovery today, 26(1):80, 2021.
  7. Why 90% of clinical drug development fails and how to improve it? Acta Pharmaceutica Sinica B, 2022.
  8. Graph neural networks for social recommendation. In The world wide web conference, pages 417–426, 2019.
  9. Disentangled graph convolutional networks. In International conference on machine learning, pages 4212–4221. PMLR, 2019.
  10. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286–20296, 2020.
  11. Disentangled graph neural networks for session-based recommendation. arXiv preprint arXiv:2201.03482, 2022.
  12. Graph-wise common latent factor extraction for unsupervised graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6420–6428, 2022.
  13. Decoupled self-supervised learning for non-homophilous graphs. arXiv preprint arXiv:2206.03601, 2022.
  14. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems, 34:21872–21884, 2021.
  15. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
  16. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.
  17. Fair darts: Eliminating unfair advantages in differentiable architecture search. In European conference on computer vision, pages 465–480. Springer, 2020.
  18. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12239–12248, 2021.
  19. Supernet in neural architecture search: A taxonomic survey. arXiv preprint arXiv:2204.03916, 2022.
  20. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  21. Graph attention networks. In International Conference on Learning Representations, 2018.
  22. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
  23. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019.
  24. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095–4104. PMLR, 2018.
  25. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.
  26. Single path one-shot neural architecture search with uniform sampling. In European conference on computer vision, pages 544–560. Springer, 2020.
  27. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
  28. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33:12486–12498, 2020.
  29. Neural architecture optimization. Advances in neural information processing systems, 31, 2018.
  30. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105–7114. PMLR, 2019.
  31. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In Tenth International Conference on Learning Representations, pages 1–36. OpenReview. net, 2022.
  32. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009–17021, 2020.
  33. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
  34. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457, 2022.
  35. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
  36. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pages 4602–4609, 2019.
  37. Graph differentiable architecture search with structure learning. Advances in Neural Information Processing Systems, 34, 2021.
  38. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4):771–783, 2003.
  39. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
  40. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786–797, 1991.
  41. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365–1374, 2015.
  42. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663, 2020.
  43. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243–246, 2015.
  44. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52, 2015.
  45. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020.
  46. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1–34, 2021.
  47. Neural architecture search for transformers: A survey. IEEE Access, 10:108374–108412, 2022.
  48. Autogl: A library for automated graph learning. In ICLR 2021 Workshop GTRL, 2021.
  49. Nas-bench-graph: Benchmarking graph neural architecture search. arXiv preprint arXiv:2206.09166, 2022.
  50. Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering, 34(1):249–270, 2020.
  51. Large graph models: A perspective. arXiv preprint arXiv:2308.14522, 2023.
  52. Learning to solve travelling salesman problem with hardness-adaptive curriculum. In Thirty-Fifth AAAI Conference on Artificial Intelligence, 2022.
  53. Revisiting transformation invariant geometric deep learning: Are initial representations all you need? arXiv preprint arXiv:2112.12345, 2021.
  54. Llm4dyg: Can large language models solve problems on dynamic graphs? arXiv preprint, 2023.
  55. Auto-gnn: Neural architecture search of graph neural networks. arXiv:1909.03184, 2019.
  56. Neural architecture search in graph neural networks. In Brazilian Conference on Intelligent Systems, pages 302–317, 2020.
  57. Genetic-gnn: Evolutionary architecture search for graph neural networks. Knowledge-Based Systems, page 108752, 2022.
  58. Deep and flexible graph neural architecture search. In International Conference on Machine Learning, pages 26362–26374. PMLR, 2022.
  59. Diffmg: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 279–288, 2021.
  60. Search to aggregate neighborhood for graph neural network. In 2021 IEEE 37th International Conference on Data Engineering, pages 552–563, 2021.
  61. One-shot graph neural architecture search with dynamic search space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8510–8517, 2021.
  62. Rethinking graph neural architecture search from message-passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6657–6666, 2021.
  63. Graph neural architecture search under distribution shifts. In International Conference on Machine Learning, pages 18083–18095. PMLR, 2022.
  64. Dynamic heterogeneous graph attention neural architecture search. In Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023.
  65. Multi-task graph neural architecture search with task-aware collaboration and curriculum. Advances in neural information processing systems, 2023.
  66. Self-supervised neural architecture search. arXiv preprint arXiv:2007.01500, 2020.
  67. Self-supervised neural architecture search for imbalanced datasets. arXiv preprint arXiv:2109.08580, 2021.
  68. Csnas: Contrastive self-supervised learning neural architecture search via sequential model-based optimization. IEEE Transactions on Artificial Intelligence, 3(4):609–624, 2021.
  69. Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12281–12291, 2021.
  70. Towards self-supervised and weight-preserving neural architecture search. arXiv preprint arXiv:2206.04125, 2022.
  71. Automated unsupervised graph representation learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
  72. Neural architecture search with random labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10907–10916, 2021.
  73. Neural architecture search without training. In International Conference on Machine Learning, pages 7588–7598. PMLR, 2021.
  74. Are labels necessary for neural architecture search? In European Conference on Computer Vision, pages 798–813. Springer, 2020.
  75. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering, 2022.
  76. Self-supervised representation learning via latent graph prediction. arXiv preprint arXiv:2202.08333, 2022.
  77. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
  78. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000, 2019.
  79. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 259–270, 2020.
  80. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pages 11548–11558. PMLR, 2021.
  81. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1150–1160, 2020.
  82. Generative subgraph contrast for self-supervised graph representation learning. arXiv preprint arXiv:2207.11996, 2022.
  83. Let invariant rationale discovery inspire graph contrastive learning. In International Conference on Machine Learning, pages 13052–13065. PMLR, 2022.
  84. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131, 2020.
  85. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812–5823, 2020.
  86. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116–4126. PMLR, 2020.
  87. Graphmae: Self-supervised masked graph autoencoders. arXiv preprint arXiv:2205.10803, 2022.
  88. Maskgae: Masked graph modeling meets graph autoencoders. arXiv preprint arXiv:2205.10053, 2022.
  89. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, pages 737–746, 2023.
  90. When does self-supervision help graph convolutional networks? In international conference on machine learning, pages 10871–10880. PMLR, 2020.
  91. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559–12571, 2020.
  92. Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006.10141, 2020.
  93. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
  94. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  95. Heterogeneous graph attention network. In The world wide web conference, pages 2022–2032, 2019.
  96. Community preserving network embedding. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  97. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121–12132. PMLR, 2021.
  98. Autogcl: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8892–8900, 2022.
  99. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2069–2080, 2021.
  100. Automated self-supervised learning for graphs. arXiv preprint arXiv:2106.05470, 2021.
  101. Multi-task self-supervised graph neural networks enable stronger task generalization. arXiv preprint arXiv:2210.02016, 2022.
  102. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  103. Disentangled representation learning, 2023.
  104. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems, 31, 2018.
  105. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99–108, 2018.
  106. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
  107. Emily L Denton et al. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30, 2017.
  108. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1415–1424, 2017.
  109. Mixup-augmented temporally debiased video grounding with content-location disentanglement. 2023.
  110. Disentangled representation learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  111. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34:26924–26936, 2021.
  112. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2021.
  113. Disentangled self-supervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 483–491, 2020.
  114. Learning disentangled representations for recommendation. Advances in neural information processing systems, 32, 2019.
  115. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering, 2021.
  116. Curriculum co-disentangled representation learning across multiple environments for social recommendation. 2023.
  117. Adaptive disentangled transformer for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3434–3445, 2023.
  118. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916–4923, 2020.
  119. Out-of-distribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference on Medical Artificial Intelligence, 2023.
  120. Spectral invariant learning for dynamic graphs under distribution shifts. In Advances in Neural Information Processing Systems, 2023.
  121. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems, 2022.
  122. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pages 145–158. Springer, 2011.
  123. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  124. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.
  125. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR, 2015.
  126. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  127. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  128. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  129. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets