GraphSAINT: Graph Sampling Based Inductive Learning Method (1907.04931v4)

Abstract: Graph Convolutional Networks (GCNs) are powerful models for learning representations of attributed graphs. To scale GCNs to large graphs, state-of-the-art methods use various layer sampling techniques to alleviate the "neighbor explosion" problem during minibatch training. We propose GraphSAINT, a graph sampling based inductive learning method that improves training efficiency and accuracy in a fundamentally different way. By changing perspective, GraphSAINT constructs minibatches by sampling the training graph, rather than the nodes or edges across GCN layers. Each iteration, a complete GCN is built from the properly sampled subgraph. Thus, we ensure fixed number of well-connected nodes in all layers. We further propose normalization technique to eliminate bias, and sampling algorithms for variance reduction. Importantly, we can decouple the sampling from the forward and backward propagation, and extend GraphSAINT with many architecture variants (e.g., graph attention, jumping connection). GraphSAINT demonstrates superior performance in both accuracy and training time on five large graphs, and achieves new state-of-the-art F1 scores for PPI (0.995) and Reddit (0.970).

• The paper introduces a graph sampling-based inductive learning method that mitigates the neighbor explosion problem in deep GCNs.
• The approach employs normalization techniques, variance reduction, and decoupled sampling to construct efficient subgraphs during training.
• Empirical results demonstrate state-of-the-art performance with F1 scores of 0.995 on PPI and 0.970 on Reddit, confirming its effectiveness.

Overview of Graph Sampling Based Inductive Learning Method for Training GCNs

The presented paper introduces a novel approach to enhancing the efficiency and accuracy of Graph Convolutional Networks (GCNs) specifically designed for large graphs. Current state-of-the-art methods suffer from the "neighbor explosion" issue during minibatch training as the GCNs become deeper. The authors propose a fundamentally different method focusing on a graph sampling-based inductive learning technique, denoted as {} within the document.

Methodology

The method operates by sampling subgraphs from the entire training graph and constructing a full GCN for these subgraphs in each iteration, thus addressing the neighbor explosion problem effectively. This approach ensures a fixed number of well-connected nodes across all GCN layers. Several key components drive the effectiveness of the proposed model:

1. Normalization Techniques: The approach involves the development of normalization techniques to eliminate biases introduced by the non-identical node sampling probabilities.
2. Variance Reduction: The proposal includes the design of sampling algorithms targeted at reducing variance, which is crucial for improving training quality.
3. Decoupling Sampling: An essential feature of the graph sampling-based approach is its ability to decouple sampling from forward and backward propagation within the neural network.

This paper also highlights the flexibility of the method: the approach can integrate with various GCN architectures, such as those incorporating graph attention and jumping connections.

Empirical Results

The model's efficacy is demonstrated by experimental results, showcasing superior performance over existing alternatives in terms of both training accuracy and time efficiency, particularly on datasets with five large graphs. Notably, the proposed model achieved an F1 score of 0.995 on PPI and 0.970 on Reddit, establishing new state-of-the-art benchmarks.

Theoretical and Practical Implications

Theoretically, this model provides a solution to the scalability issue inherent in training large and deep GCNs. Practically, it paves the way for more efficient usage of computational resources during the training process. As a result of tackling the neighbor explosion problem, the method not only accelerates the training process but also enhances the model's predictive performance.

Future Directions

Given the decoupled nature of the sampling process, potential future developments could include:

• Distributed Computing: Exploration into distributed computing setups where subgraph sampling and training occur independently across multiple processors, reducing communication costs.
• System Co-optimization: Aligning the learning algorithm closely with hardware platforms to further optimize performance, especially on large-scale heterogeneous computing infrastructures.

The paper's contributions open several avenues for exploration in adapting GCNs to broader applications, potentially impacting the development of future graph-based learning models. This research provides not only a methodological advancement but also practical insight into managing computational complexity in machine learning on graphs.