Deep Convolutional Networks on Graph-Structured Data (1506.05163v1)

Abstract: Deep Learning's recent successes have mostly relied on Convolutional Networks, which exploit fundamental statistical properties of images, sounds and video data: the local stationarity and multi-scale compositional structure, that allows expressing long range interactions in terms of shorter, localized interactions. However, there exist other important examples, such as text documents or bioinformatic data, that may lack some or all of these strong statistical regularities. In this paper we consider the general question of how to construct deep architectures with small learning complexity on general non-Euclidean domains, which are typically unknown and need to be estimated from the data. In particular, we develop an extension of Spectral Networks which incorporates a Graph Estimation procedure, that we test on large-scale classification problems, matching or improving over Dropout Networks with far less parameters to estimate.

• The paper introduces spectral networks that enable convolutional operations on graph-structured data using the Graph Fourier Transform.
• It adapts pooling through hierarchical graph clustering, ensuring learned filters remain localized in the spectral domain.
• Empirical results demonstrate competitive performance with fewer parameters, while outlining future directions for computational efficiency and enhanced graph estimation.

Deep Convolutional Networks on Graph-Structured Data

The paper "Deep Convolutional Networks on Graph-Structured Data" by Henaff, Bruna, and LeCun introduces a significant extension of convolutional networks (ConvNets) tailored for non-Euclidean domains. While ConvNets have achieved substantial success in analyzing Euclidean data such as images, audio, and video, their utility diminishes when applied to data that do not conform to grid-like structures. The primary contribution of this paper is the development of spectral networks capable of handling graph-structured data, specifically targeting applications where the underlying data relationships form complex graphs.

Main Contributions

Generalization of Spectral Networks

The authors propose a generalization of spectral networks, which are ConvNets operating in the spectral domain of graphs. This involves leveraging the Graph Fourier Transform (GFT) founded on the graph Laplacian to perform convolutions. For an input signal $x$ defined on a graph $G$ with $N$ nodes, the graph convolution with filters $g$ is represented as $x \ast_G g = U^T (Ux \odot Ug)$, where $U$ is the unitary matrix of the graph Laplacian’s eigenvectors, and $\odot$ denotes pointwise multiplication.

One of the essential strategies introduced is smoothing in the frequency domain to ensure that learned filters are localized. This mirrors the ConvNet approach of using spatially compact filters, but the localization is expressed through restrictions on the spectral multipliers.

Graph Pooling Mechanism

Similar to the pooling operations in ConvNets that reduce spatial resolution, the authors adapt pooling operations for graphs using hierarchical graph clustering. This multi-resolution spectral clustering constructs neighborhoods at different scales, enabling localized information aggregation, akin to spatial down-sampling in traditional ConvNets.

Graph Estimation

A key challenge addressed in this work is that many applications do not provide predefined graph structures. Thus, the authors introduce methods to estimate the graph:

• Unsupervised Graph Estimation: Employs measures like Euclidean distance or mutual information between features, transforming these into a Gaussian diffusion kernel.
• Supervised Graph Estimation: Utilizes the outputs from fully connected networks to establish feature similarity, thereby refining graph structures to better suit specific tasks.

Results and Implications

Empirical Evaluations

The paper experimentally evaluates spectral networks across various datasets and domains:

• Reuters Text Categorization: Spectral networks using supervised graph estimation matched the performance of fully connected networks but with significantly fewer parameters.
• Merck Molecular Activity Challenge: On this computational biology dataset, the supervised graph estimation enabled the spectral network to achieve comparable results to a more complex fully connected network, again with markedly fewer parameters.
• ImageNet Object Recognition: Although this included a predefined grid structure (making it an outlier), spectral networks achieved competitive results to traditional ConvNets, validating the efficacy of spectral methods even on standard image data.

Computational Efficiency

A notable discussion within the paper pertains to computational complexity. Evaluating spectral networks necessitates costly $O(N^2)$ matrix multiplications for each forward and backward pass when applying the Graph Fourier Transform. In contrast, classical ConvNets benefit from optimized $O(N \log N)$ Fourier Transforms. The paper suggests potential future research into approximate eigen-basis computations to better handle high-dimensional graphs efficiently.

Future Directions

The theoretical and practical contributions set a solid foundation for future research in several directions:

• Enhanced Graph Estimation Techniques: Improving the robustness and precision of both supervised and unsupervised graph estimation methods remains a fertile ground for exploration.
• Relaxed Weight Sharing Constraints: Introducing variances in weight sharing, potentially through a commutation error approach, may enhance the versatility of spectral networks.
• Optimization of Computational Efficiency: Techniques to reduce the computational footprint, such as Givens decompositions for approximations analogous to the Fast Fourier Transform, could broaden the applicability of spectral networks.

In summary, this paper systematically extends the scope of convolutional networks to encompass graph-structured data, providing rigorous frameworks for graph convolutions, pooling, and estimation. The results manifest potential across diverse high-dimensional datasets with varying structures, making a compelling case for the inclusion of graph-based spectral methods in the toolkit of deep learning architectures.