Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework (2403.01079v1)
Abstract: We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability.
- Liu Xin, F.: Survey on graph neural network acceleration: An algorithmic perspective. arXiv preprint arXiv: 2202.04822 (2022).
- Zhou Hongkuan, F.: Accelerating large scale real-time GNN inference using channel pruning. arXiv preprint arXiv: 2105.04528 (2021).
- Zhang Shichang, F.: Graph-less neural networks: Teaching old mlps new tricks via distillation. arXiv preprint arXiv:2110.08727 (2021).
- Yang Chenxiao, F.: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks. arXiv preprint arXiv:2210.13014 (2022).
- Xu Keyulu, F.: How powerful are graph neural networks?. arXiv preprint arXiv:1810.00826 (2018).
- Li Guohao, F.: Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739 (2020).
- Yun Seongjun, F.: Graph transformer networks. Advances in neural information processing systems 32 (2019).
- Dwivedi Vijay Prakash, F.: A generalization of transformer networks to graphs (2020).
- Cheng Yang, F.: Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework (2021a).
- Chen Jie, F.: SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP. arXiv preprint arXiv:2210.09609 (2022).
- Hu Yang, F.: Graph-mlp: Node classification without message passing in graph. arXiv preprint arXiv:2106.04051 (2021).
- Wang Haorui, F.: Equivariant and stable positional encoding for more powerful graph neural networks. arXiv preprint arXiv:2203.00199 (2022).
- Chen Yuzhao, F.: On self-distilling graph neural network. arXiv preprint arXiv:2011.02255 (2020).
- Ba Jimmy,F.: Do deep nets really need to be deep?. Advances in neural information processing systems 27 (2014).
- Tang Raphael, F.: Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136 (2019).
- Xu Bingbing, F.: Graph convolutional networks using heat kernel for semi-supervised learning. arXiv preprint arXiv:2007.16002 (2020).
- Joshi Chaitanya K., F.: On representation knowledge distillation for graph neural networks. IEEE Transactions on Neural Networks and Learning Systems (2022).
- Hinton Geoffrey, F.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.