Self-Attention Graph Pooling (1904.08082v4)

Abstract: Advanced methods of applying deep learning to structured data such as graphs have been proposed in recent years. In particular, studies have focused on generalizing convolutional neural networks to graph data, which includes redefining the convolution and the downsampling (pooling) operations for graphs. The method of generalizing the convolution operation to graphs has been proven to improve performance and is widely used. However, the method of applying downsampling to graphs is still difficult to perform and has room for improvement. In this paper, we propose a graph pooling method based on self-attention. Self-attention using graph convolution allows our pooling method to consider both node features and graph topology. To ensure a fair comparison, the same training procedures and model architectures were used for the existing pooling methods and our method. The experimental results demonstrate that our method achieves superior graph classification performance on the benchmark datasets using a reasonable number of parameters.

• The paper introduces SAGPool, a self-attention-based pooling method that jointly considers node features and graph topology to overcome traditional limitations.
• The paper utilizes graph convolution to compute attention scores, maintaining a constant parameter count regardless of graph size for enhanced scalability.
• The paper demonstrates improved classification performance on benchmarks like D&D and PROTEINS, outperforming existing methods and proving its efficacy.

Self-Attention Graph Pooling

The research paper "Self-Attention Graph Pooling" by Junhyun Lee, Inyeop Lee, and Jaewoo Kang introduces a novel graph pooling method called Self-Attention Graph Pooling (SAGPool) within the domain of Graph Neural Networks (GNNs). This method addresses the challenges of applying pooling operations in non-Euclidean domains, ensuring a comprehensive consideration of both node features and graph topology.

Context and Motivation

The extension of Convolutional Neural Networks (CNNs) to graph-based data has attracted significant attention due to its potential to improve structured data recognition. While techniques for graph convolution are relatively mature and have proven efficacy, graph pooling methods lag in performance and reliability. Traditional graph pooling methods often overlook the intricate relationship between node features and graph structure, resulting in suboptimal representations.

Methodology

SAGPool leverages self-attention mechanisms to determine which nodes to retain or discard during pooling. This method employs graph convolution to calculate attention scores, enabling a joint consideration of node features and graph topology. Notably, the architecture maintains a reasonable number of parameters regardless of the graph size, which contrasts with some existing pooling methods that suffer from parameter size dependency on the number of graph nodes.

The self-attention mask in SAGPool is computed using graph convolutional layers, with the attention scores determining the nodes selected for the next layer. This approach inherently integrates node features and graph structure into the pooling operation. Furthermore, the method allows for various adaptations, such as considering multi-hop connections and averaging attention scores from multiple GNNs.

Experimental Results

The authors conducted extensive experiments on several benchmark datasets, including D&D, PROTEINS, NCI1, NCI109, and FRANKENSTEIN, using both global and hierarchical pooling architectures.

Empirical results show:

• Global Pooling: SAGPool outperformed established methods like Set2Set and SortPool across multiple datasets, with notable improvements on D&D (76.19%) and PROTEINS (70.04%).
• Hierarchical Pooling: Compared to DiffPool and gPool, SAGPool consistently achieved higher accuracy. For instance, it recorded 76.45% on D&D and 71.86% on PROTEINS.

Analysis and Implications

The primary advantage of SAGPool lies in its unified consideration of node features and topology, facilitated by the attention mechanism. This approach allows for more nuanced and informative pooling operations, directly contributing to improved classification performance. The consistent parameter count, independent of graph size, further underscores its practical utility, making it scalable and efficient.

From a theoretical standpoint, integrating self-attention mechanisms with graph convolutions innovatively adapts successful techniques from other domains (e.g., NLP) to graph-based structures, potentially inspiring further interdisciplinary methodology adaptations.

Future Directions

Several avenues remain open for future exploration:

1. Learnable Pooling Ratios: Introducing adaptability in pooling ratios might yield optimized graph representations tailored to specific structures.
2. Multi-mask Aggregation: Investigating the impact of using multiple attention masks per pooling layer could lead to richer representations and improved performance in complex graph scenarios.
3. Sparse vs Dense Implementations: Further optimization could explore balancing between sparse and dense representations to enhance computational efficiency while maintaining or improving accuracy.

Conclusion

The introduction of SAGPool marks a critical step forward in hierarchical graph representation learning, addressing significant limitations of previous methods. By combining self-attention with graph convolution, SAGPool delivers superior performance on graph classification tasks while maintaining efficiency, thus presenting a compelling case for its broader adoption and further development within the GNN community.

This work not only pushes the boundaries of what is achievable with graph pooling but also lays the groundwork for future innovations in machine learning applications involving structured data.