Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CaT: Balanced Continual Graph Learning with Graph Condensation (2309.09455v2)

Published 18 Sep 2023 in cs.LG

Abstract: Continual graph learning (CGL) is purposed to continuously update a graph model with graph data being fed in a streaming manner. Since the model easily forgets previously learned knowledge when training with new-coming data, the catastrophic forgetting problem has been the major focus in CGL. Recent replay-based methods intend to solve this problem by updating the model using both (1) the entire new-coming data and (2) a sampling-based memory bank that stores replayed graphs to approximate the distribution of historical data. After updating the model, a new replayed graph sampled from the incoming graph will be added to the existing memory bank. Despite these methods are intuitive and effective for the CGL, two issues are identified in this paper. Firstly, most sampling-based methods struggle to fully capture the historical distribution when the storage budget is tight. Secondly, a significant data imbalance exists in terms of the scales of the complex new-coming graph data and the lightweight memory bank, resulting in unbalanced training. To solve these issues, a Condense and Train (CaT) framework is proposed in this paper. Prior to each model update, the new-coming graph is condensed to a small yet informative synthesised replayed graph, which is then stored in a Condensed Graph Memory with historical replay graphs. In the continual learning phase, a Training in Memory scheme is used to update the model directly with the Condensed Graph Memory rather than the whole new-coming graph, which alleviates the data imbalance problem. Extensive experiments conducted on four benchmark datasets successfully demonstrate superior performances of the proposed CaT framework in terms of effectiveness and efficiency. The code has been released on https://github.com/superallen13/CaT-CGL.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in ECCV, 2018.
  2. G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J. Zhu, “Generalizing dataset distillation via deep generative prior,” in CVPR, 2023.
  3. Y. Chen, M. Welling, and A. J. Smola, “Super-samples from kernel herding,” in UAI, 2010.
  4. A. A. Daruna, M. Gupta, M. Sridharan, and S. Chernova, “Continual learning of knowledge graph embeddings,” IEEE Robotics Autom. Lett., 2021.
  5. X. Gao, T. Chen, Y. Zang, W. Zhang, Q. V. H. Nguyen, K. Zheng, and H. Yin, “Graph condensation for inductive node representation learning,” CoRR, vol. abs/2307.15967, 2023.
  6. J. Gu, K. Wang, W. Jiang, and Y. You, “Summarizing stream data for memory-restricted online continual learning,” CoRR, vol. abs/2305.16645, 2023.
  7. W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NeurIPS, 2017.
  8. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” in NeurIPS, 2020.
  9. W. Jin, X. Tang, H. Jiang, Z. Li, D. Zhang, J. Tang, and B. Yin, “Condensing graphs via one-step gradient matching,” in KDD, 2022.
  10. W. Jin, L. Zhao, S. Zhang, Y. Liu, J. Tang, and N. Shah, “Graph condensation for graph neural networks,” in ICLR, 2022.
  11. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, 2017.
  12. J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” CoRR, vol. abs/1612.00796, 2016.
  13. J. Ko, S. Kang, and K. Shin, “Begin: Extensive benchmark scenarios and an easy-to-use framework for graph continual learning,” CoRR, vol. abs/2211.14568, 2022.
  14. Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in ICLR, 2016.
  15. Z. Li and D. Hoiem, “Learning without forgetting,” TPAMI, 2018.
  16. H. Liu, Y. Yang, and X. Wang, “Overcoming catastrophic forgetting in graph neural networks,” in AAAI, 2021.
  17. M. Liu, S. Li, X. Chen, and L. Song, “Graph condensation via receptive field distribution matching,” CoRR, vol. abs/2206.13697, 2022.
  18. D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in NeurIPS, 2017.
  19. W. Masarczyk and I. Tautkute, “Reducing catastrophic forgetting with learning on synthetic data,” in CVPR Workshop, 2020.
  20. A. McCallum, K. Nigam, J. Rennie, and K. Seymore, “Automating the construction of internet portals with machine learning,” Inf. Retr., 2000.
  21. R. Qiu, H. Yin, Z. Huang, and T. Chen, “GAG: global attributed graph neural network for streaming session-based recommendation,” in SIGIR, 2020.
  22. A. Rosasco, A. Carta, A. Cossu, V. Lomonaco, and D. Bacciu, “Distilled replay: Overcoming forgetting through synthetic samples,” in Continual Semi-Supervised Learning - First International Workshop, CSSL, 2021.
  23. M. Sangermano, A. Carta, A. Cossu, and D. Bacciu, “Sample condensation in online continual learning,” in IJCNN, 2022.
  24. O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” in ICLR, 2018.
  25. L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” JMLR, 2008.
  26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017.
  27. P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in ICLR, 2018.
  28. K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and Y. You, “CAFE: learning to condense dataset by aligning features,” in CVPR, 2022.
  29. L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,” CoRR, vol. abs/2302.00487, 2023.
  30. T. Wang, J. Zhu, A. Torralba, and A. A. Efros, “Dataset distillation,” CoRR, vol. abs/1811.10959, 2018.
  31. Z. Wang and J. Ye, “Querying discriminative and representative samples for batch mode active learning,” in KDD, 2013.
  32. F. Wiewel and B. Yang, “Condensed composite memory continual learning,” in IJCNN, 2021.
  33. F. Wu, A. H. S. Jr., T. Zhang, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” in ICML, 2019.
  34. K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in ICLR, 2019.
  35. Y. Xu, Y. Zhang, W. Guo, H. Guo, R. Tang, and M. Coates, “Graphsail: Graph structure aware incremental learning for recommender systems,” in CIKM, 2020.
  36. X. Zhang, D. Song, and D. Tao, “Cglb: Benchmark tasks for continual graph learning,” in NeurIPS Systems Datasets and Benchmarks Track, 2022.
  37. ——, “Sparsified subgraph memory for continual graph representation learning,” in ICDM, 2022.
  38. ——, “Hierarchical prototype networks for continual graph representation learning,” TPAMI, 2023.
  39. ——, “Sufficient subgraph embedding memory for continual graph representation learning,” 2023. [Online]. Available: https://openreview.net/forum?id=SJjvXfape5U
  40. B. Zhao and H. Bilen, “Dataset condensation with differentiable siamese augmentation,” in ICML, 2021.
  41. ——, “Dataset condensation with distribution matching,” in WACV, 2023.
  42. B. Zhao, K. R. Mopuri, and H. Bilen, “Dataset condensation with gradient matching,” in ICLR, 2021.
  43. D. Zhou, Q. Wang, Z. Qi, H. Ye, D. Zhan, and Z. Liu, “Deep class-incremental learning: A survey,” CoRR, vol. abs/2302.03648, 2023.
  44. F. Zhou and C. Cao, “Overcoming catastrophic forgetting in graph neural networks with experience replay,” in AAAI, 2021.
Citations (29)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - superallen13/CaT-CGL (16 stars)