Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Attend Who is Weak: Enhancing Graph Condensation via Cross-Free Adversarial Training (2311.15772v1)

Published 27 Nov 2023 in cs.LG

Abstract: In this paper, we study the \textit{graph condensation} problem by compressing the large, complex graph into a concise, synthetic representation that preserves the most essential and discriminative information of structure and features. We seminally propose the concept of Shock Absorber (a type of perturbation) that enhances the robustness and stability of the original graphs against changes in an adversarial training fashion. Concretely, (I) we forcibly match the gradients between pre-selected graph neural networks (GNNs) trained on a synthetic, simplified graph and the original training graph at regularly spaced intervals. (II) Before each update synthetic graph point, a Shock Absorber serves as a gradient attacker to maximize the distance between the synthetic dataset and the original graph by selectively perturbing the parts that are underrepresented or insufficiently informative. We iteratively repeat the above two processes (I and II) in an adversarial training fashion to maintain the highly-informative context without losing correlation with the original dataset. More importantly, our shock absorber and the synthesized graph parallelly share the backward process in a free training manner. Compared to the original adversarial training, it introduces almost no additional time overhead. We validate our framework across 8 datasets (3 graph and 5 node classification datasets) and achieve prominent results: for example, on Cora, Citeseer and Ogbn-Arxiv, we can gain nearly 1.13% to 5.03% improvements compare with SOTA models. Moreover, our algorithm adds only about 0.2% to 2.2% additional time overhead over Flicker, Citeseer and Ogbn-Arxiv. Compared to the general adversarial training, our approach improves time efficiency by nearly 4-fold.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Binary graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9492–9501.
  2. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572.
  3. Dataset Distillation by Matching Training Trajectories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, 10708–10717.
  4. Fastgcn: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247.
  5. Fast gradient attack on network embedding. arXiv preprint arXiv:1809.02797.
  6. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, 1695–1706. PMLR.
  7. Fusing 2D and 3D molecular graphs as unambiguous molecular descriptors for conformational and chiral stereoisomers. Briefings in Bioinformatics, 24(1).
  8. A comprehensive study on large-scale graph training: Benchmarking and rethinking. arXiv preprint arXiv:2210.07494.
  9. Benchmarking Graph Neural Networks. CoRR, abs/2003.00982.
  10. Provable and practical approximations for the degree distribution using sublinear graph samples. In Proceedings of the 2018 World Wide Web Conference, 449–458.
  11. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media.
  12. Graph u-nets. In international conference on machine learning, 2083–2092. PMLR.
  13. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings.
  14. A deep graph neural network-based mechanism for social recommendations. IEEE Transactions on Industrial Informatics, 17(4): 2776–2783.
  15. Graph neural network: Current state of Art, challenges and applications. Materials Today: Proceedings, 46: 10927–10932.
  16. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 1024–1034.
  17. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, 173–182.
  18. HetETA: Heterogeneous information network embedding for estimating time of arrival. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2444–2454.
  19. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020.
  20. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2): 494–514.
  21. Delving into effective gradient matching for dataset condensation. arXiv preprint arXiv:2208.00311.
  22. Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey. arXiv preprint arXiv:2303.14483.
  23. Condensing graphs via one-step gradient matching. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 720–730.
  24. Graph Condensation for Graph Neural Networks. In The Tenth International Conference on Learning Representations, ICLR 2022.
  25. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, 11102–11118. PMLR.
  26. Semi-Supervised Classification with Graph Convolutional Networks. In 5th International Conference on Learning Representations, ICLR 2017.
  27. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 60–69.
  28. Self-attention graph pooling. In International conference on machine learning, 3734–3743. PMLR.
  29. Dataset condensation with contrastive signals. In International Conference on Machine Learning, 12352–12364. PMLR.
  30. Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739.
  31. Dataset distillation via factorization. arXiv preprint arXiv:2210.16774.
  32. Towards Deep Learning Models Resistant to Adversarial Attacks. In 6th International Conference on Learning Representations, ICLR 2018.
  33. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2574–2582.
  34. TUDataset: A collection of benchmark datasets for learning with graphs. CoRR, abs/2007.08663.
  35. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663.
  36. Dataset Meta-Learning from Kernel Ridge-Regression. In 9th International Conference on Learning Representations, ICLR 2021.
  37. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34: 5186–5198.
  38. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277.
  39. ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, 5470–5477.
  40. Structure-Aware Hierarchical Graph Pooling using Information Bottleneck. In 2021 International Joint Conference on Neural Networks (IJCNN), 1–8. IEEE.
  41. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In 6th International Conference on Learning Representations, ICLR 2018.
  42. Adversarial training for free! Advances in Neural Information Processing Systems, 32.
  43. Inductive Lottery Ticket Learning for Graph Neural Networks.
  44. Degree-quant: Quantization-aware training for graph neural networks. arXiv preprint arXiv:2008.05000.
  45. Graphless: Toward serverless graph processing. In 2019 18th International Symposium on Parallel and Distributed Computing (ISPDC), 66–73. IEEE.
  46. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv:1705.07204.
  47. Graph Attention Networks. In 6th International Conference on Learning Representations, ICLR 2018.
  48. Bi-gcn: Binary graph convolutional network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1561–1570.
  49. Searching Lottery Tickets in Graph Neural Networks: A Dual Perspective. In The Eleventh International Conference on Learning Representations.
  50. Dataset distillation. arXiv preprint arXiv:1811.10959.
  51. Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, 165–174.
  52. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3): 279–287.
  53. Welling, M. 2009. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, 1121–1128.
  54. Simplifying graph convolutional networks. In International conference on machine learning, 6861–6871. PMLR.
  55. Structural entropy guided graph hierarchical pooling. In International Conference on Machine Learning, 24017–24030. PMLR.
  56. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5): 1–37.
  57. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1): 4–24.
  58. How Powerful are Graph Neural Networks? In 7th International Conference on Learning Representations, ICLR 2019.
  59. MoleRec: Combinatorial Drug Recommendation with Substructure-Aware Molecular Representation Learning. In Proceedings of the ACM Web Conference 2023, 4075–4085.
  60. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31.
  61. L2-gcn: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2127–2135.
  62. Graph embedding on biomedical networks: methods, applications and evaluations. Bioinformatics, 36(4): 1241–1251.
  63. GraphSAINT: Graph Sampling Based Inductive Learning Method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  64. Graph-less neural networks: Teaching old mlps new tricks via distillation. arXiv preprint arXiv:2110.08727.
  65. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, 12674–12685. PMLR.
  66. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 6514–6523.
  67. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929.
  68. Graph neural networks: A review of methods and applications. AI open, 1: 57–81.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now