Overcoming Catastrophic Forgetting in Graph Neural Networks

Abstract: Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving~(TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art. Code is publicly available at \url{https://github.com/hhliu79/TWP}.

• The paper introduces TWP, a module designed to preserve crucial graph topology and parameters for enhanced continual learning.
• It integrates the TWP module into various GNN architectures such as GCNs, GATs, and GINs, demonstrating improved metrics like AP and AF.
• The approach significantly reduces catastrophic forgetting in GNNs, benefiting applications in social network analysis and biological data modeling.

Overcoming Catastrophic Forgetting in Graph Neural Networks

The paper, "Overcoming Catastrophic Forgetting in Graph Neural Networks," by Huihui Liu, Yiding Yang, and Xinchao Wang, addresses a prevalent issue in machine learning models, particularly applicable to Graph Neural Networks (GNNs): catastrophic forgetting during the course of continual learning. In this context, catastrophic forgetting describes the degradation in performance when a neural network, trained sequentially on different tasks, fails to retain the knowledge of previously learned tasks upon learning new ones.

Traditionally, significant efforts have been focused on tackling this issue in the domain of Convolutional Neural Networks (CNNs), which process data defined on grid spaces such as images. However, these strategies neglect the unique challenges posed in non-grid structured data such as graphs, which is where GNNs come into play. A pivotal advancement made in this study is the introduction of a methodology specifically designed to address catastrophic forgetting within the domain of GNNs.

Central to overcoming this challenge is the proposal of a module named Topology-aware Weight Preserving (TWP). TWP is tailored to be a plug-and-play component that can apply to any GNN architecture to enhance its continual learning capabilities without being confined to specific models. The novelty of TWP lies in its focus on both stabilizing parameters pivotal for the downstream task performance (a common approach in CNNs) and explicitly maintaining the local structures — the topological interactions among the nodes — of the input graphs. This dual-focus offers significant advantages over traditional parameter-slowing methodologies which may neglect the essential topological considerations inherent in GNNs.

The experimental validations presented in the paper reinforce the efficacy of the TWP module. Evaluations spanned across various GNN backbones including Graph Attention Networks (GATs), Graph Convolutional Networks (GCNs), and Graph Isomorphism Networks (GINs). The datasets tested varied from node classification to graph classification across a spectrum of domains, further affirming the versatility of the approach. The results highlighted that the integration of TWP led to superior performance relative to existing state-of-the-art methods, documented by improvements in metrics such as Average Performance (AP) and Average Forgetting (AF).

Critically, this paper ties the advancement into broader contexts of practical utility and theoretical exploration. Practically, improvements in continual learning for GNNs can greatly benefit fields reliant on sequential task updating such as social network analysis, biological pathway modeling, and citation networks. Theoretically, this work opens avenues for further exploration into integrating additional graph properties into continual learning paradigms, possibly bridging GNN architectures with newly proposed ones that are more intrinsic to capturing sequential task dependencies.

Future explorations might extend the principles outlined by the TWP module to other paradigms of neural computation, or further refine the understanding of topological influences within different types of GNNs. Additionally, measuring the balance and trade-offs between parameter preservation and model plasticity might offer insights that could refine continual learning strategies both theoretically and practically.

Overall, the paper underscores a substantial stride in addressing catastrophic forgetting for GNNs, with the nuances of topological considerations providing key insights that propel the field towards more efficient and resilient learning systems amidst evolving task requirements. The contribution made by Liu and colleagues is a notable one, supporting the expansion of GNN capabilities in various complex applications beyond conventional grid-based data.