Graph Contrastive Learning with Augmentations (2010.13902v3)

Abstract: Generalizable, transferrable, and robust representation learning on graph-structured data remains a challenge for current graph neural networks (GNNs). Unlike what has been developed for convolutional neural networks (CNNs) for image data, self-supervised learning and pre-training are less explored for GNNs. In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We first design four types of graph augmentations to incorporate various priors. We then systematically study the impact of various combinations of graph augmentations on multiple datasets, in four different settings: semi-supervised, unsupervised, and transfer learning as well as adversarial attacks. The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, our GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods. We also investigate the impact of parameterized graph augmentation extents and patterns, and observe further performance gains in preliminary experiments. Our codes are available at https://github.com/Shen-Lab/GraphCL.

Graph Contrastive Learning with Augmentations

The paper "Graph Contrastive Learning with Augmentations" presents a novel framework, GraphCL, to advance the unsupervised learning of representations on graph-structured data. This research is motivated by the observation that existing self-supervised and pre-training techniques for graph neural networks (GNNs) lag behind those developed for convolutional neural networks (CNNs). The authors propose a contrastive learning approach enhanced with innovative graph-specific augmentations to tackle the challenges of data heterogeneity and limited labeled data in graphs.

Key Contributions

Graph Data Augmentations: The cornerstone of GraphCL is the introduction of four types of graph data augmentations:

1. Node Dropping: Randomly discarding vertices and their edges, thereby simulating a scenario where some nodes are missing while preserving the semantic meaning of the graph.
2. Edge Perturbation: Modifying the graph's connectivity by randomly adding or removing edges, reflecting the robustness of the graph's semantics to such changes.
3. Attribute Masking: Masking node attributes to simulate the loss of partial attribute information, encouraging the model to infer missing data from the context.
4. Subgraph Sampling: Extracting subgraphs based on random walks, leveraging local structural information to infer global semantics.

Framework for Graph Contrastive Learning: The GraphCL framework involves applying these augmentations to obtain two correlated views of the same graph. A shared GNN-based encoder processes these views, and a non-linear projection head maps them to a latent space. The framework maximizes agreement between these augmented views using a contrastive loss, specifically the normalized temperature-scaled cross-entropy loss (NT-Xent).

Theoretical Insights: The authors provide a theoretical foundation showing that GraphCL essentially maximizes a lower bound of the mutual information between different views of the graph. They also establish connections between GraphCL and other contrastive learning methods, demonstrating its generality and unifying various techniques under one framework.

Experimental Evaluation

The effectiveness of GraphCL is systematically evaluated across multiple datasets and settings, including semi-supervised learning, unsupervised representation learning, and transfer learning.

1. Semi-Supervised Learning: GraphCL consistently outperforms traditional pre-training methods like adjacency information reconstruction (GAE) and local-global representation consistency (Infomax). The improvements are significant, with accuracy gains of up to 7.11% for certain datasets, highlighting the robustness and generalizability of the learned representations.
2. Unsupervised Representation Learning: In this setting, GraphCL is compared with state-of-the-art graph kernel methods and unsupervised graph representation techniques. GraphCL achieves superior results in most cases, particularly for larger graphs, thus validating its effectiveness in capturing meaningful graph representations in an unsupervised manner.
3. Transfer Learning: The transferability of GraphCL is evaluated on molecular and protein datasets. GraphCL outperforms previous best pre-training schemes on 5 out of 9 datasets, demonstrating its potential to generalize across diverse domains.
4. Adversarial Robustness: The paper extends the evaluation to adversarial robustness, showing that GNNs pre-trained with GraphCL exhibit enhanced resistance to various adversarial attacks compared to models trained from scratch.

Practical and Theoretical Implications

The findings of this paper have several important implications:

• Practical Impact: The proposed graph-specific augmentations and contrastive learning framework can be directly applied to enhance the performance of GNNs in various real-world applications such as drug discovery, social network analysis, and recommendation systems.
• Theoretical Contributions: The connection between GraphCL and mutual information maximization provides a deeper understanding of how contrastive learning can be effectively applied to graph-structured data. This insight opens avenues for further research in designing better augmentations and improving the theoretical underpinnings of contrastive learning in graphs.

Future Directions

Future developments may explore several promising directions:

• Optimizing Augmentation Strategies: While the paper shows that combining different augmentations generally works well, further research could optimize the types, extents, and patterns of augmentations for specific graph datasets and tasks.
• Scalability: As graph datasets grow larger and more complex, efficient algorithms for graph augmentations and contrastive learning will be crucial. Future work could focus on scaling GraphCL to handle these challenges.
• Application-Specific Tailoring: Tailoring GraphCL to specific domains, such as biological networks or industrial graphs, could enhance its utility by incorporating domain-specific knowledge into the augmentation and contrastive learning processes.

In conclusion, the paper "Graph Contrastive Learning with Augmentations" provides a robust framework for improving the unsupervised and semi-supervised learning of graph representations. The innovative augmentation strategies and the rigorous evaluation underscore the potential of contrastive learning to enhance the capabilities of GNNs in both general and specific applications.