Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Out-of-Distribution Generalized Dynamic Graph Neural Network with Disentangled Intervention and Invariance Promotion (2311.14255v2)

Published 24 Nov 2023 in cs.LG

Abstract: Dynamic graph neural networks (DyGNNs) have demonstrated powerful predictive abilities by exploiting graph structural and temporal dynamics. However, the existing DyGNNs fail to handle distribution shifts, which naturally exist in dynamic graphs, mainly because the patterns exploited by DyGNNs may be variant with respect to labels under distribution shifts. In this paper, we propose Disentangled Intervention-based Dynamic graph Attention networks with Invariance Promotion (I-DIDA) to handle spatio-temporal distribution shifts in dynamic graphs by discovering and utilizing invariant patterns, i.e., structures and features whose predictive abilities are stable across distribution shifts. Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns. By utilizing the disentangled patterns, we design a spatio-temporal intervention mechanism to create multiple interventional distributions and an environment inference module to infer the latent spatio-temporal environments, and minimize the variance of predictions among these intervened distributions and environments, so that our model can make predictions based on invariant patterns with stable predictive abilities under distribution shifts. Extensive experiments demonstrate the superiority of our method over state-of-the-art baselines under distribution shifts. Our work is the first study of spatio-temporal distribution shifts in dynamic graphs, to the best of our knowledge.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (146)
  1. Invariant risk minimization games. In International Conference on Machine Learning. PMLR, 145–155.
  2. Invariant risk minimization. arXiv preprint (2019).
  3. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
  4. HGWaveNet: A Hyperbolic Graph Neural Network for Temporal Link Prediction. In Proceedings of the ACM Web Conference 2023. 523–532.
  5. The architecture of complex weighted networks. Proceedings of the national academy of sciences 101, 11 (2004), 3747–3752.
  6. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798–1828.
  7. Higher-order organization of complex networks. Science 353, 6295 (2016), 163–166.
  8. Tanya Y Berger-Wolf and Jared Saia. 2006. A framework for analysis of dynamic social networks. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 523–528.
  9. Richard A Berk. 1983. An introduction to sample selection bias in sociological data. American sociological review (1983), 386–398.
  10. Size-invariant graph representations for graph classification extrapolations. In International Conference on Machine Learning. 837–851.
  11. Predicting the silent majority on graphs: Knowledge transferable graph neural network. In Proceedings of the ACM Web Conference 2023. 274–285.
  12. Survivorship bias in performance studies. The Review of Financial Studies 5, 4 (1992), 553–580.
  13. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems (2019).
  14. User cold-start recommendation via inductive heterogeneous graph neural network. ACM Transactions on Information Systems (TOIS) 41, 3 (2023), 1–27.
  15. Structural temporal graph neural networks for anomaly detection in dynamic graphs. In Proceedings of the 30th ACM international conference on Information & Knowledge Management. 3747–3756.
  16. Invariant rationalization. In International Conference on Machine Learning. PMLR, 1448–1458.
  17. Continuous-time dynamic graph learning via neural interaction processes. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 145–154.
  18. Learning to generalize in heterogeneous federated networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 159–168.
  19. Curriculum Disentangled Recommendation with Noisy Multi-feedback. Advances in Neural Information Processing Systems 34 (2021), 26924–26936.
  20. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems 29 (2016).
  21. Neural feature-aware recommendation with signed hypergraph convolutional network. ACM Transactions on Information Systems (TOIS) 39, 1 (2020), 1–22.
  22. Invariance Principle Meets Out-of-Distribution Generalization on Graphs. arXiv preprint (2022).
  23. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In EMNLP.
  24. James S Coleman. 1994. Foundations of social theory. Harvard university press.
  25. Dynamic Graph Representation Learning via Graph Transformer Networks. arXiv preprint arXiv:2111.10447 (2021).
  26. Dynamic knowledge graph based multi-event forecasting. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1585–1595.
  27. Emily L Denton et al. 2017. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems 30 (2017).
  28. A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs. (2021).
  29. Disentangled Spatiotemporal Graph Generative Models. In Thirty-Sixth AAAI Conference on Artificial Intelligence. AAAI Press, 6541–6549.
  30. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 402–411.
  31. Generalizing Graph Neural Networks on Out-Of-Distribution Graphs. arXiv preprint arXiv:2111.10657 (2021).
  32. Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
  33. WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series Tasks. arXiv preprint arXiv:2203.09978 (2022).
  34. Alleviating structural distribution shift in graph anomaly detection. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 357–365.
  35. Causal inference in statistics: A primer. John Wiley & Sons.
  36. Tracking the evolution of communities in dynamic social networks. In 2010 international conference on advances in social networks analysis and mining. IEEE, 176–183.
  37. Variational graph recurrent neural networks. Advances in neural information processing systems 32 (2019).
  38. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning. 8230–8248.
  39. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
  40. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems 31 (2018).
  41. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems 33 (2020), 22118–22133.
  42. Motif-Preserving Temporal Network Embedding.. In IJCAI. 1237–1243.
  43. Triadic closure pattern analysis and prediction in social networks. IEEE Transactions on Knowledge and Data Engineering 27, 12 (2015), 3374–3389.
  44. Kexin Huang and Marinka Zitnik. 2020. Graph meta learning via local subgraphs. Advances in Neural Information Processing Systems 33 (2020), 5862–5874.
  45. Position-enhanced and time-aware graph convolutional network for sequential recommendations. ACM Transactions on Information Systems (TOIS) 41, 1 (2023), 1–32.
  46. Coupled Graph ODE for Learning Interacting System Dynamics.. In KDD. 705–715.
  47. Community detection and co-author recommendation in co-author networks. International Journal of Machine Learning and Cybernetics 12, 2 (2021), 597–609.
  48. Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift. In International Conference on Learning Representations.
  49. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, Yoshua Bengio and Yann LeCun (Eds.).
  50. Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).
  51. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In 5th International Conference on Learning Representations. OpenReview.net.
  52. Temporal motifs in time-dependent networks. Journal of Statistical Mechanics: Theory and Experiment 2011, 11 (2011), P11005.
  53. Temporal motifs reveal homophily, gender-specific patterns, and group talk in call sequences. Proceedings of the National Academy of Sciences 110, 45 (2013), 18070–18075.
  54. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning. 5815–5826.
  55. Fates of Microscopic Social Ecosystems: Keep Alive or Dead?. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 668–676.
  56. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering (2021).
  57. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems 34 (2021), 21872–21884.
  58. Ood-gnn: Out-of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering (2022).
  59. Out-Of-Distribution Generalization on Graphs: A Survey. arXiv preprint (2022).
  60. Disentangled Graph Contrastive Learning With Independence Promotion. IEEE Transactions on Knowledge and Data Engineering (2022).
  61. Learning Invariant Graph Representations for Out-of-Distribution Generalization. In Thirty-Sixth Conference on Neural Information Processing Systems.
  62. Invariant Node Representation Learning under Distribution Shifts with Multiple Latent Environments. ACM Transactions on Information Systems (TOIS) (jun 2023). https://doi.org/10.1145/3604427 Just Accepted.
  63. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering 6, 12 (2022), 1353–1369.
  64. Preference-aware Graph Attention Networks for Cross-Domain Recommendations with Collaborative Knowledge Graph. ACM Transactions on Information Systems (TOIS) 41, 3 (2023), 1–26.
  65. Confidence may cheat: Self-training on graph neural networks under distribution shift. In Proceedings of the ACM Web Conference 2022. 1248–1258.
  66. Heterogeneous risk minimization. In International Conference on Machine Learning. PMLR, 6804–6814.
  67. Good-d: On unsupervised graph out-of-distribution detection. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 339–347.
  68. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4916–4923.
  69. DIVERSIFY to Generalize: Learning Generalized Representations for Time Series Classification. arXiv preprint (2021).
  70. Disentangled graph convolutional networks. In International conference on machine learning. PMLR, 4212–4221.
  71. Learning disentangled representations for recommendation. Advances in neural information processing systems 32 (2019).
  72. Disentangled self-supervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 483–491.
  73. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 99–108.
  74. Kr-gcn: Knowledge-aware reasoning with graph convolution network for explainable recommendation. ACM Transactions on Information Systems (TOIS) 41, 1 (2023), 1–27.
  75. Efficient Estimation of Word Representations in Vector Space. In 1st International Conference on Learning Representations, Yoshua Bengio and Yann LeCun (Eds.).
  76. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013).
  77. Representation Learning via Invariant Causal Mechanisms. In 9th International Conference on Learning Representations. OpenReview.net.
  78. Dynamic graph in a symbolic data framework: An account of the causal relation using COVID-19 reports and some reflections on the financial world. Chaos, Solitons & Fractals 153 (2021), 111440.
  79. Motifs in temporal networks. In Proceedings of the tenth ACM international conference on web search and data mining. 601–610.
  80. Evolvegcn: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5363–5370.
  81. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems (2019).
  82. Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress 19 (2000), 2.
  83. Dynamic graph convolutional network for long-term traffic flow prediction with reinforcement learning. Information Sciences 578 (2021), 401–416.
  84. Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting. Information Sciences 521 (2020), 277–290.
  85. Graph Neural Architecture Search Under Distribution Shifts. In International Conference on Machine Learning. 18083–18095.
  86. Temporal network embedding with high-order nonlinear information. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5436–5443.
  87. The Risks of Invariant Risk Minimization. In 9th International Conference on Learning Representations. OpenReview.net.
  88. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 (2020).
  89. Distributionally Robust Neural Networks. In International Conference on Learning Representations.
  90. Dysat: Deep neural representation learning on dynamic graphs via self-attention networks. In Proceedings of the 13th International Conference on Web Search and Data Mining. 519–527.
  91. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing. Springer, 362–373.
  92. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 (2021).
  93. Georg Simmel. 1950. The sociology of georg simmel. Vol. 92892. Simon and Schuster.
  94. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web. ACM, 243–246.
  95. Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey. IEEE Access (2021), 79143–79168.
  96. Hyperbolic variational graph neural network for modeling dynamic graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4375–4383.
  97. Learning to represent the evolution of dynamic graphs with recurrent models. In Proceedings of the ACM Web Conference 2019. 301–307.
  98. Dynamic Graph Evolution Learning for Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1589–1598.
  99. Cross-domain Collaboration Recommendation. In KDD’2012.
  100. ArnetMiner: Extraction and Mining of Academic Social Networks. In KDD’08. 990–998.
  101. A characterization of interventional distributions in semi-Markovian causal models. eScholarship, University of California.
  102. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1415–1424.
  103. Dyrep: Learning representations over dynamic graphs. In International conference on learning representations.
  104. Attention is all you need. Advances in neural information processing systems (2017).
  105. Graph Attention Networks. In International Conference on Learning Representations.
  106. Environment agnostic invariant risk minimization for classification of sequential datasets. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1615–1624.
  107. Hongwei Wang and Jure Leskovec. 2021. Combining graph convolutional neural networks and label propagation. ACM Transactions on Information Systems (TOIS) 40, 4 (2021), 1–27.
  108. A Tutorial on Domain Generalization. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 1236–1239.
  109. Microsoft academic graph: When experts are not enough. Quantitative Science Studies 1, 1 (2020), 396–413.
  110. Causal Representation Learning for Out-of-Distribution Recommendation. In Proceedings of the ACM Web Conference 2022. 3562–3571.
  111. Disentangled Representation Learning for Recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
  112. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME). 1–6.
  113. Community preserving network embedding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 31.
  114. Heterogeneous graph attention network. In The world wide web conference. 2022–2032.
  115. Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks. In 9th International Conference on Learning Representations. OpenReview.net.
  116. TEDIC: Neural modeling of behavioral patterns in dynamic social interaction networks. In Proceedings of the Web Conference 2021. 693–705.
  117. DisenCTR: Dynamic graph-based disentangled representation for click-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2314–2318.
  118. Imbalanced graph classification via graph-of-graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2067–2076.
  119. AdaGCL: Adaptive Subgraph Contrastive Learning to Generalize Large-scale Graph Training. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2046–2055.
  120. TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 5730–5746.
  121. Dynamic graph convolutional networks for entity linking. In Proceedings of The ACM Web Conference 2020. 1149–1159.
  122. Handling Distribution Shifts on Graphs: An Invariance Perspective. International Conference on Learning Representations (2022).
  123. Discovering Invariant Rationales for Graph Neural Networks. In The Tenth International Conference on Learning Representations. OpenReview.net.
  124. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems 32, 1 (2020), 4–24.
  125. Graph neural collaborative topic model for citation recommendation. ACM Transactions on Information Systems (TOIS) 40, 3 (2021), 1–30.
  126. Inductive representation learning on temporal graphs. In 8th International Conference on Learning Representations. OpenReview.net.
  127. Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1975–1985.
  128. Interpretable Research Interest Shift Detection with Temporal Heterogeneous Graphs. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 321–329.
  129. HGAT: Heterogeneous graph attention networks for semi-supervised short text classification. ACM Transactions on Information Systems (TOIS) 39, 3 (2021), 1–29.
  130. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems 33 (2020), 20286–20296.
  131. A Generic Learning Framework for Sequential Recommendation with Distribution Shifts. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval.
  132. Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time. In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  133. Improving Out-of-Distribution Robustness via Selective Augmentation. In Proceeding of the Thirty-ninth International Conference on Machine Learning.
  134. Hierarchical temporal convolutional networks for dynamic recommender systems. In The world wide web conference. 2236–2246.
  135. efraudcom: An e-commerce fraud detection system via competitive graph neural networks. ACM Transactions on Information Systems (TOIS) 40, 3 (2022), 1–29.
  136. DynGraphTrans: Dynamic Graph Embedding via Modified Universal Transformer Networks for Financial Transaction Data. In 2021 IEEE International Conference on Smart Data Services (SMDS). IEEE, 184–191.
  137. Disentangled dynamic graph deep generation. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM). SIAM, 738–746.
  138. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems.
  139. Learning to Solve Travelling Salesman Problem with Hardness-Adaptive Curriculum. In Thirty-Sixth AAAI Conference on Artificial Intelligence. AAAI Press, 9136–9144.
  140. Time-interval Aware Share Recommendation via Bi-directional Continuous Time Dynamic Graphs. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 822–831.
  141. Graph neural networks: A review of methods and applications. AI open 1 (2020), 57–81.
  142. Dynamic network embedding by modeling triadic closure process. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
  143. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems 34 (2021).
  144. Learnable Encoder-Decoder Architecture for Dynamic Graph: A Survey. arXiv preprint arXiv:2203.10480 (2022).
  145. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics 34, 13 (2018), i457–i466.
  146. Evolution of resilience in protein interactomes across the tree of life. Proceedings of the National Academy of Sciences 116, 10 (2019), 4426–4433.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Zeyang Zhang (28 papers)
  2. Xin Wang (1307 papers)
  3. Ziwei Zhang (40 papers)
  4. Haoyang Li (95 papers)
  5. Wenwu Zhu (104 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now