Variational Graph Auto-Encoders (1611.07308v1)

Abstract: We introduce the variational graph auto-encoder (VGAE), a framework for unsupervised learning on graph-structured data based on the variational auto-encoder (VAE). This model makes use of latent variables and is capable of learning interpretable latent representations for undirected graphs. We demonstrate this model using a graph convolutional network (GCN) encoder and a simple inner product decoder. Our model achieves competitive results on a link prediction task in citation networks. In contrast to most existing models for unsupervised learning on graph-structured data and link prediction, our model can naturally incorporate node features, which significantly improves predictive performance on a number of benchmark datasets.

• The paper introduces a variational graph autoencoder that merges GCNs with VAE techniques to capture graph structure and perform link prediction.
• The methodology employs a two-layer GCN encoder and an inner-product decoder to compute latent representations from node features and adjacency data.
• The research demonstrates significant improvements in link prediction on citation networks like Cora, Citeseer, and Pubmed with high AUC and AP scores.

Variational Graph Auto-Encoders

The paper "Variational Graph Auto-Encoders" by Thomas N. Kipf and Max Welling presents a method for unsupervised learning on graph-structured data using the Variational Graph Auto-Encoder (VGAE). This model merges principles from variational auto-encoders (VAE) with graph learning techniques, leveraging Graph Convolutional Networks (GCNs) to create latent variable models for undirected graphs. The primary application explored is link prediction in citation networks, showcasing the model's capacity to learn meaningful and interpretable graph embeddings.

Methodology

The VGAE framework employs a two-layer GCN as an encoder, capturing the graph's structural nuances and node features. The encoder outputs parameters for a Gaussian distribution, from which latent variables are sampled. These latent representations are then used by a simple inner product decoder to reconstruct the graph's adjacency matrix, effectively predicting the presence or absence of edges.

Inference Model

The inference model for VGAE is specified as a probabilistic encoder: $$q(\mathbf{Z}\,|\,\mathbf{X},\mathbf{A}) = \prod_{i=1}^N q(\mathbf{z}_i\,|\,\mathbf{X},\mathbf{A})$$ with each $q(\mathbf{z}_i\,|\,\mathbf{X},\mathbf{A})$ being a Gaussian distribution parameterized by the output of the GCN: $$\mathcal{N}(\mathbf{z}_i\,|\, \boldsymbol{\mu}_i, \mathrm{diag}(\boldsymbol{\sigma}_i^2))$$ The mean $\boldsymbol{\mu}_i$ and variance $\boldsymbol{\sigma}_i$ are derived from the graph convolution network acting on node features $\mathbf{X}$ and adjacency matrix $\mathbf{A}$.

Generative Model

The generative model used in VGAE involves reconstructing the adjacency matrix using the inner product of latent variables: $$p\left(\mathbf{A}\,|\,\mathbf{Z}\right) = \prod_{i=1}^N \prod_{j=1}^N p\left(A_{ij}\,|\,\mathbf{z}_i, \mathbf{z}_j\right)$$ where the conditional probability of an edge's presence is given by a sigmoid function: $$p(A_{ij} = 1\,|\, \mathbf{z}_i, \mathbf{z}_j) = \sigma(\mathbf{z}_i^\top \mathbf{z}_j)$$

Learning

The variational lower bound $\mathcal{L}$ is maximized to learn the model parameters, involving the reconstruction log-likelihood of the observed adjacency matrix and a Kullback-Leibler (KL) divergence term: $$\mathcal{L} = \mathbb{E}_{q(\mathbf{Z}|\mathbf{X}, \mathbf{A})} \bigl[\log p(\mathbf{A}\,|\,\mathbf{Z})\bigr] - \mathrm{KL}[q(\mathbf{Z}\,|\,\mathbf{X}, \mathbf{A})\,\|\,p(\mathbf{Z})]$$ A Gaussian prior is adopted for the latent variables, and the model is trained using full-batch gradient descent and the reparameterization trick.

Experiments and Results

The VGAE and its non-probabilistic counterpart, the Graph Auto-Encoder (GAE), are evaluated on link prediction tasks using several citation network datasets: Cora, Citeseer, and Pubmed. The datasets are modified by removing edges to form validation and test sets, ensuring the model's ability to correctly classify both edges and non-edges.

The performance metrics include the area under the ROC curve (AUC) and average precision (AP). The VGAE and GAE demonstrated superior performance compared to traditional methods such as spectral clustering (SC) and DeepWalk (DW). Specifically, incorporating node features led to significant performance improvements, with VGAE achieving top scores on the Cora and Citeseer datasets:

• Cora: VGAE achieved an AUC of $91.4\%$ and AP of $92.6\%$
• Citeseer: VGAE attained an AUC of $90.8\%$ and AP of $92.0\%$
• Pubmed: GAE achieved the highest scores, with an AUC and AP of $96.4\%$ and $96.5\%$, respectively.

Discussion and Future Directions

The results confirm that both VGAE and GAE can effectively encode graph structure and node features into latent spaces useful for predictive tasks. The authors acknowledge the limitations of the Gaussian prior and propose future investigations into more suitable priors and flexible generative models. Additionally, incorporating stochastic gradient descent could enhance scalability.

Implications

The implications of this research are significant for the development of unsupervised learning models in graph-structured domains. The proposed VGAE model offers a robust framework that can potentially be adapted for other tasks like node classification and graph generation. Future work might explore integrating more complex node features and leveraging alternative generative mechanisms to further enhance model performance and applicability across diverse datasets.

This paper contributes a methodological advancement in the field of graph neural networks, particularly for tasks requiring the capture of complex relational structures within data.