Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (1606.09375v3)

Abstract: In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.

• The paper proposes a spectral graph-based method to generalize CNNs to irregular graph data while ensuring filter locality and computational efficiency.
• It utilizes Chebyshev polynomial approximations to design strictly localized filters with linear complexity and effective pooling via binary tree structure.
• Experimental results on MNIST and 20NEWS confirm that the approach achieves competitive accuracy and superior GPU scalability compared to traditional CNNs.

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Introduction

The paper addresses the generalization of convolutional neural networks (CNNs) from regular grids to graph-structured data, encompassing domains like social networks, brain connectomes, and text documents. Traditional CNNs are effective on structured data such as images and videos; however, extending them to graphs entails defining localized filters that accommodate the irregular nature of graphs.

Main Contributions

The authors make several noteworthy contributions:

1. Spectral Formulation: They propose a spectral graph-based formulation using graph signal processing (GSP) techniques. This builds on existing GSP methods to create a theoretical foundation for graph CNNs.
2. Strictly Localized Filters: The proposed filters are strictly localized, providing computational efficiency and scalability, avoiding the use of the expensive Fourier basis.
3. Low Computational Complexity: The method achieves linear computational complexity relative to the filter support size and the number of edges. For sparse real-world graphs, this results in significant computational savings.
4. Efficient Pooling: The pooling strategy arranges vertices in a binary tree structure, enabling efficient pooling akin to 1D signal pooling.
5. Experimental Validation: The proposed method was validated through experiments on MNIST and 20NEWS, demonstrating its capacity to learn local, stationary, and compositional features on graph-structured data while maintaining computational efficiency.

Proposed Technique

Generalizing CNNs to graphs involves:

• Localized Convolutional Filters: Using spectral graph theory, convolution operators are designed in the spectral domain to maintain locality and stationarity.
• Graph Coarsening: A multilevel clustering approach is applied to group similar vertices, preserving geometric structures.
• Fast Pooling: Reorganizing vertices into a binary tree allows graph pooling operations to be performed efficiently.

Learning Fast Localized Spectral Filters

The filters are parameterized using polynomial functions, ensuring locality and reducing computational and learning complexity. The Chebyshev polynomial expansion is used to approximate the filters, significantly reducing the operations required for filtering signals, enabling efficient implementations on parallel architectures like GPUs.

Graph Coarsening and Pooling

Graph coarsening clusters vertices to create a hierarchy of graphs with decreasing resolution. The Graclus algorithm is employed for its efficiency and ability to reduce graph sizes substantively at each level. Pooling rearranges vertices post-coarsening to facilitate efficient operations comparable to 1D signal pooling.

Experimental Results

MNIST

An 8-NN graph was constructed from the 2D grid of MNIST images. The proposed method achieved comparable accuracy to classical CNNs, demonstrating its ability to adapt to grid-like structures. Specifically, a graph CNN reached an accuracy of 99.14%, closely matching the classical CNN.

20NEWS

For the 20NEWS dataset, a 16-NN graph based on word2vec embeddings was utilized. The graph CNN outperformed fully connected networks, and performed competitively with Naive Bayes, reflecting its robustness in handling text data structured as graphs.

Computational Efficiency

The proposed graph CNN demonstrated superior scalability, with computational complexity linear with respect to input size. It featured marked improvements in GPU acceleration compared to CPU, similar to classical CNNs.

Implications and Future Directions

The proposed framework highlights the potential of graph-based CNNs to extend deep learning methods to a broad range of non-Euclidean structures. This paves the way for further advancements in fields such as social network analysis, bioinformatics, and natural language processing.

Future research directions include integrating advanced GSP tools into the framework and exploring application-specific graph constructions to enhance model performance. Another compelling avenue involves jointly learning the graph structure and the CNN parameters, leveraging methods for adaptive graph learning.

In summary, this paper establishes a robust and efficient methodology for extending CNNs to graph-structured data, opening new possibilities for the application of deep learning across diverse scientific and engineering domains.