Self-Supervised Learning of Graph Neural Networks: A Unified Review (2102.10757v5)

Abstract: Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. In either category, we provide a unified framework for methods as well as how these methods differ in each component under the framework. Our unified treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms. We also summarize different SSL settings and the corresponding datasets used in each setting. To facilitate methodological development and empirical comparison, we develop a standardized testbed for SSL in GNNs, including implementations of common baseline methods, datasets, and evaluation metrics.

• The paper introduces a unified framework that categorizes SSL for GNNs into contrastive and predictive methods.
• It demonstrates that contrastive techniques capture graph dependencies with mutual information estimators while predictive methods enhance model learning via self-generated tasks.
• The paper establishes a standardized testbed with baseline implementations for reproducible evaluations across diverse graph benchmark datasets.

Self-Supervised Learning of Graph Neural Networks: A Unified Review

The paper, "Self-Supervised Learning of Graph Neural Networks: A Unified Review," provides a comprehensive survey of self-supervised learning (SSL) in the context of graph neural networks (GNNs). The authors systematically categorize and review the existing methodologies, conceptualizing them into two principal types: contrastive methods and predictive methods. They propose a unified framework for SSL methods in GNNs that synthesizes the various approaches into a cohesive structure, offering insights into existing models and paving the way for future developments in the field.

Framework and Approaches

1. Contrastive Learning: The paper classifies contrastive methods based on their mechanisms to maximize mutual information (MI) between representations of different views of the same graph. These methods are further divided into two categories: node-level and graph-level representations. Contrastive methods are grounded in discriminative objectives that aim to differentiate positive instance pairs from negative ones. The focus is on maximizing MI estimates using various statistical bounds like the InfoNCE, Jensen-Shannon Estimators, or Donsker-Varadhan representation.
2. Predictive Learning: In contrast, predictive methods focus on self-generated labels from the graph data itself. These methods are structured around reconstructive or generative objectives, such as autoencoders for graph reconstruction and downstream predictive tasks like property prediction or invariance regularization. These are categorized into non-probabilistic and variational graph autoencoders, and autoregressive reconstruction models. The predictive methods, unlike their contrastive counterparts, do not require explicitly mining negative samples but instead rely on auxiliary tasks to improve feature learning.

Key Contributions

• Unified Framework: By providing a unified framework for understanding SSL in GNNs, this paper aids in identifying patterns and commonalities among various SSL methods. This synthesis enables more effective comparisons and the identification of gaps in the current research landscape.
• Standardized Testbed: The authors present a standardized testbed, including implementations of baseline methods, to facilitate the empirical comparison of SSL techniques across multiple GNN tasks. This testbed aims to streamline the evaluation process for new methodologies and supports reproducibility in SSL research.
• Analysis of Datasets: The paper evaluates SSL methods in the context of common graph datasets, differentiating between graph-level and node-level tasks and providing critical insights into how these datasets are utilized. This analysis helps to contextualize the application of SSL methods in real-world scenarios and highlights dataset-specific considerations.

Numerical Results and Implications

The survey emphasizes the performance of SSL methods on benchmark datasets in both inductive (e.g., molecular and social network graphs) and transductive tasks (e.g., large citation networks). The results underscore the effectiveness of contrastive methods in capturing complex dependencies within the graph structures and their superior performance in various graph-based ML tasks. The incorporation of SSL within GNNs enhances the robustness and accuracy of models, especially in scenarios with limited labeled data.

Future Directions

The paper outlines several potential directions for future research in SSL for GNNs. Key areas of interest include the exploration of novel contrastive objectives, refinement of mutual information bounds, and the integration of domain-specific priors to enhance predictive tasks. Further, the development of scalable SSL algorithms capable of handling large-scale graph data remains a critical challenge. By iterating on these methods, the paper suggests that researchers could significantly advance the theoretical and practical capabilities of GNNs.

Overall, this paper acts as a significant resource, not only providing an exhaustive survey of current SSL technologies for GNNs but also setting up a foundation for future work that will continue to expand and refine these ambitious methodologies.