Large-Scale Representation Learning on Graphs via Bootstrapping (2102.06514v3)

Abstract: Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs. However, to achieve state-of-the-art performance, methods often need large numbers of negative examples and rely on complex augmentations. This can be prohibitively expensive, especially for large graphs. To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input. BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design. BGRL outperforms or matches prior methods on several established benchmarks, while achieving a 2-10x reduction in memory costs. Furthermore, we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-of-the-art performance and improving over supervised baselines where representations are shaped only through label information. In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph Benchmark - Large Scale Challenge at KDD Cup 2021, on a graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the scalability and effectiveness of our approach.

• The paper presents BGRL, a bootstrapping approach for self-supervised graph learning that eliminates the need for negative examples.
• It employs dual encoders and simple graph augmentations to achieve linear scalability and state-of-the-art performance on diverse benchmark tasks.
• BGRL demonstrates memory efficiency and excels on large-scale datasets, notably winning the OGB-LSC challenge at KDD Cup 2021.

Overview of "Large-Scale Representation Learning on Graphs via Bootstrapping"

Introduction

The paper "Large-Scale Representation Learning on Graphs via Bootstrapping" presents Bootstrapped Graph Latents (BGRL), a novel self-supervised learning technique aimed at enhancing graph representation learning without the need for labeled data. This approach is particularly notable for its deviation from contrastive methods, opting instead for bootstrapping strategies that simplify augmentations and eliminate reliance on negative examples. BGRL is designed to be scalable, efficiently handling large graph structures with hundreds of millions of nodes.

Methodology

BGRL employs two encoders—an online encoder and a target encoder—to generate node representations through augmented graph views. The online encoder predicts the representations of the target encoder, which is updated using an exponential moving average of the online encoder’s parameters. This setup negates the need for contrasting negative examples, distinguishing BGRL in scalability and efficiency aspects compared to contrastive methods, which traditionally demand extensive computational resources for negative example generation and comparison.

Key Contributions

The paper highlights several critical contributions of BGRL:

1. Scalability: BGRL performs optimally with simple graph augmentations, scaling linearly in time and space with respect to input size, unlike traditional contrastive methods, which scale quadratically.
2. Memory Efficiency: The technique maintains competitive performance with reduced memory usage (2-10x less than leading contrastive methods), an advantage demonstrated across standard benchmarks.
3. Performance: It achieves state-of-the-art results in semi-supervised regimes, leveraging large-scale unlabeled data on graph structures. Notably, BGRL was a winning solution in the Open Graph Benchmark - Large Scale Challenge at KDD Cup 2021 for a graph dataset significantly larger than previously available benchmarks.

Experimental Results

The paper provides a detailed experimental analysis across several datasets, including both small-scale and extremely large-scale graphs:

• Benchmark Performance: BGRL outperforms existing methods in several transductive and inductive benchmark tasks, underscoring its efficacy without excessive use of memory.
• Scalability: On the ogbn-arXiv and Protein-Protein Interaction datasets, BGRL demonstrates superior scalability, without compromising on performance even with simpler augmentation strategies.
• Extreme Scale: In the MAG240M dataset from OGB-LSC, BGRL showcases impressive scalability and state-of-the-art performance, making judicious use of unlabeled data for improved representation learning.

Implications and Future Work

BGRL’s approach represents a significant step forward in scalable, efficient self-supervised graph representation learning. Its memory-efficient strategy and avoidance of complex augmentation techniques hold promise for practical applications across vast datasets, particularly in scenarios with abundant unlabeled data. Theoretical exploration of the bootstrapping dynamics could further optimize BGRL, enhancing its non-collapsing behavior and general applicability.

Future research can focus on refining its application to various graph-based tasks, potentially incorporating deeper insights into the dynamics of bootstrapping strategies in self-supervised learning. Additionally, integration with diverse graph neural network architectures could pave the way for heightened adaptability and performance enhancement in heterogeneous network structures.