Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift (2211.02843v2)

Published 5 Nov 2022 in cs.LG

Abstract: The issue of distribution shifts is emerging as a critical concern in graph representation learning. From the perspective of invariant learning and stable learning, a recently well-established paradigm for out-of-distribution generalization, stable features of the graph are assumed to causally determine labels, while environmental features tend to be unstable and can lead to the two primary types of distribution shifts. The correlation shift is often caused by the spurious correlation between environmental features and labels that differs between the training and test data; the covariate shift often stems from the presence of new environmental features in test data. However, most strategies, such as invariant learning or graph augmentation, typically struggle with limited training environments or perturbed stable features, thus exposing limitations in handling the problem of covariate shift. To address this challenge, we propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA), to handle the covariate shift on graphs. Specifically, given the training data, AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process. Such a design equips the graph classification model with an enhanced capability to identify stable features in new environments, thereby effectively tackling the covariate shift in data. Extensive experiments with in-depth empirical analysis demonstrate the superiority of our approach. The implementation codes are publicly available at https://github.com/yongduosui/AIA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021.
  2. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2206.08452, 2022.
  3. Handling distribution shifts on graphs: An invariance perspective. In ICLR, 2022.
  4. Out-of-distribution generalization on graphs: A survey. arXiv preprint arXiv:2202.07987, 2022.
  5. Causal attention for interpretable and generalizable graph classification. In KDD, 2022.
  6. Graph neural networks are inherently good generalizers: Insights by bridging GNNs and MLPs. In ICLR, 2023.
  7. G-mixup: Graph data augmentation for graph classification. In ICML, pages 8230–8248, 2022.
  8. Learning substructure invariance for out-of-distribution molecular representations. In NeurIPS, 2022.
  9. Towards out-of-distribution sequential event prediction: A causal treatment. In NeurIPS, 2022.
  10. Invariant collaborative filtering to popularity distribution shift. In WWW, pages 1240–1251, 2023.
  11. Alleviating structural distribution shift in graph anomaly detection. In WSDM, pages 357–365, 2023.
  12. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, pages 7947–7958, 2022.
  13. A fine-grained analysis on distribution shift. In ICLR, 2022.
  14. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  15. Invariant rationalization. In ICML, pages 1448–1458, 2020.
  16. Deep stable learning for out-of-distribution generalization. In CVPR, pages 5372–5382, 2021.
  17. Discovering invariant rationales for graph neural networks. In ICLR, 2022.
  18. Graph rationalization with environment-based augmentations. In KDD, pages 1069–1078, 2022.
  19. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018.
  20. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
  21. Debiasing graph neural networks via learning disentangled causal substructure. In NeurIPS, 2022.
  22. Data augmentation for deep graph learning: A survey. arXiv preprint arXiv:2202.08235, 2022.
  23. Graph data augmentation for graph machine learning: A survey. arXiv preprint arXiv:2202.08871, 2022.
  24. Robust optimization as data augmentation for large-scale graphs. In CVPR, pages 60–69, 2022.
  25. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR, 2020.
  26. Mixup for node and graph classification. In WWW, pages 3663–3674, 2021.
  27. Graph contrastive learning with augmentations. In NeurIPS, 2020.
  28. Adversarial graph augmentation to improve graph contrastive learning. In NeurIPS, pages 15920–15933, 2021.
  29. Learning invariant graph representations for out-of-distribution generalization. In NeurIPS, 2022.
  30. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019.
  31. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
  32. Certifying some distributional robustness with principled adversarial training. In ICLR, 2018.
  33. Wasserstein generative adversarial networks. In ICML, pages 214–223, 2017.
  34. Generalizing to unseen domains via adversarial data augmentation. NeurIPS, 31, 2018.
  35. Generating images with perceptual similarity metrics based on deep networks. NIPS, 2016.
  36. Minimax statistical learning with wasserstein distances. NeurIPS, 31, 2018.
  37. Out-of-distribution generalization via risk extrapolation (rex). In ICML, pages 5815–5826, 2021.
  38. Interpretable and generalizable graph learning via stochastic attention mechanism. In ICML, pages 15524–15543, 2022.
  39. Ood-gnn: Out-of-distribution generalized graph neural network. IEEE TKDE, 2022.
  40. Generalizing graph neural networks on out-of-distribution graphs. arXiv preprint arXiv:2111.10657, 2021.
  41. Learning causally invariant representations for out-of-distribution generalization on graphs. In NeurIPS, 2022.
  42. Tradeoffs in data augmentation: An empirical study. In ICLR, 2021.
  43. Model-agnostic augmentation for accurate graph classification. In WWW, pages 1281–1291, 2022.
  44. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPSW, 2021.
  45. Invariant causal representation learning for out-of-distribution generalization. In ICLR, 2021.
  46. Shift-robust gnns: Overcoming the limitations of localized graph training data. NeurIPS, 34:27965–27977, 2021.
  47. Empowering graph representation learning with test-time graph transformation. In ICLR, 2023.
  48. Mind the label shift of augmentation-based graph ood generalization. In CVPR, 2023.
  49. Graphde: A generative framework for debiased learning and out-of-distribution detection on graphs. In NeurIPS, 2022.
  50. Energy-based out-of-distribution detection for graph neural networks. In ICLR, 2023.
  51. A data-centric framework to endow graph neural networks with out-of-distribution detection ability. In KDD, 2023.
  52. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
  53. Causally regularized learning with agnostic data selection bias. In ACM MM, pages 411–419, 2018.
  54. A theoretical analysis on independence-driven importance weighting for covariate-shift generalization. In ICML, pages 24803–24829. PMLR, 2022.
  55. Rethinking importance weighting for deep learning under distribution shift. NeurIPS, 33:11996–12007, 2020.
  56. Covariate-shift generalization via random sample weighting. AAAI, 2023.
  57. Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33(3):1065–1076, 1962.
  58. Monte carlo simulation in statistical physics. Computers in Physics, 7(2):156–157, 1993.
  59. Geometric deep learning on graphs and manifolds using mixture model cnns. In CVPR, pages 5115–5124, 2017.
  60. How powerful are graph neural networks? In ICLR, 2019.
  61. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  62. Simple and deep graph convolutional networks. In ICML, pages 1725–1735. PMLR, 2020.
  63. Attention-based graph neural network for semi-supervised learning. arXiv preprint arXiv:1803.03735, 2018.
  64. Tudataset: A collection of benchmark datasets for learning with graphs. In ICMLW, 2020.
  65. Heterogeneous risk minimization. In ICML, pages 6804–6814. PMLR, 2021.
  66. When is invariance useful in an out-of-distribution generalization problem? arXiv preprint arXiv:2008.01883, 2020.
  67. Invariant risk minimization games. In ICML, pages 145–155. PMLR, 2020.
  68. Toward learning robust and invariant representations with alignment regularization and data augmentation. In KDD, pages 1846–1856, 2022.
  69. Does invariant risk minimization capture invariance? In International Conference on Artificial Intelligence and Statistics, pages 4069–4077. PMLR, 2021.
  70. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309–1342, 2018.
  71. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  72. Generalizing to unseen domains: A survey on domain generalization. IEEE TKDE, 2022.
  73. Learning to extrapolate knowledge: Transductive few-shot out-of-graph link prediction. NeurIPS, 33:546–560, 2020.
  74. Domain generalization using causal matching. In ICML, pages 7313–7324. PMLR, 2021.
  75. Towards recognizing unseen categories in unseen domains. In ECCV, pages 466–483. Springer, 2020.
  76. Domain generalization via invariant feature representation. In ICML, pages 10–18. PMLR, 2013.
  77. Graph domain adaptation via theory-grounded spectral regularization. In ICLR, 2023.
  78. Sizeshiftreg: a regularization method for improving size-generalization in graph neural networks. In NeurIPS, 2022.
  79. From local structures to size generalization in graph neural networks. In ICML, pages 11975–11986. PMLR, 2021.
  80. Dynamic graph neural networks under spatio-temporal distribution shift. In NeurIPS, 2022.
  81. Cooperative explanations of graph neural networks. In WSDM, pages 616–624, 2023.
  82. On regularization for explaining graph neural networks: An information theory perspective. 2022.
  83. Inductive lottery ticket learning for graph neural networks. Journal of Computer Science and Technology, 2023.
  84. A unified lottery ticket hypothesis for graph neural networks. In ICML, pages 1695–1706. PMLR, 2021.
  85. Exploring lottery ticket hypothesis in media recommender systems. International Journal of Intelligent Systems, 37(5):3006–3024, 2022.
  86. Addressing heterophily in graph anomaly detection: A perspective of graph spectrum. In WWW, pages 1528–1538, 2023.
  87. Rumor detection with self-supervised learning on texts and social graph. Frontiers Comput. Sci., 17(4):174611, 2023.
  88. A survey on deep graph generation: Methods and applications. arXiv preprint arXiv:2203.06714, 2022.
Citations (28)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now