InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization (1908.01000v3)

Abstract: This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios. Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks. Traditional graph kernel based methods are simple, yet effective for obtaining fixed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs. There are also some recent methods based on LLMs (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives. Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations. We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures. Furthermore, we further propose InfoGraph*, an extension of InfoGraph for semi-supervised scenarios. InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods. As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task. Experimental results on the tasks of graph classification and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.

InfoGraph: Unsupervised and Semi-Supervised Graph-Level Representation Learning via Mutual Information Maximization

Introduction

Graph representation learning is pivotal for a spectrum of applications that span from molecular property prediction in drug discovery to analyzing social network community structures. The paper "InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization" introduces InfoGraph, a method for learning graph-level representations in both unsupervised and semi-supervised contexts. Conventional graph kernel methods, while straightforward and effective, suffer from poor generalization due to their reliance on hand-crafted designs. Recent advancements explore LLMs for graphs but remain limited as they often only consider particular graph substructures. This work is positioned at the intersection of unsupervised representation learning and graph neural networks (GNNs), positing mutual information maximization as a robust mechanism for capturing graph-level representations.

Methodology

Unsupervised Learning: The core of InfoGraph is the unsupervised model that employs mutual information maximization to learn graph-level representations. The model achieves this by maximizing the mutual information between the representation of an entire graph and the representations of its substructures at varying scales. Specifically, the method leverages a graph neural network (GNN) to produce these hierarchical substructure representations. The final graph-level features are obtained by aggregating these representations through a concatenated vector.

Mathematically, the formulation involves a mutual information estimator, parameterized by a neural network, which effectively distinguishes between positive samples (graph and its substructures) and negative samples (randomly paired graph and substructure representations from different graphs). This approach is inspired by Deep Infomax (DIM), adapted for graph data to effectively learn latent representations.

Semi-Supervised Learning: Extending the unsupervised model, InfoGraph* introduces semi-supervised learning by utilizing a student-teacher framework. Here, the student model is guided by a supervised task, while the teacher model uses an unsupervised objective on unlabeled data. The crux is maximizing mutual information between intermediate representations learned by the two models, ensuring that the student model benefits from the rich representations learned by the teacher from unlabeled data. This setup is significantly beneficial as it helps mitigate the common issue of "negative transfer" by preserving the semantic integrity of representations favored by the supervised task.

Experimental Results

Graph Classification Tasks: InfoGraph was evaluated across multiple benchmark graph datasets, including MUTAG, PTC, REDDIT-BINARY, REDDIT-MULTI-5K, IMDB-BINARY, and IMDB-MULTI. The experimental results revealed that InfoGraph outperforms state-of-the-art graph kernels and other unsupervised graph representation learning methods. Specifically, InfoGraph achieved superior classification accuracy on four out of six datasets, demonstrating the effectiveness of mutual information maximization in capturing meaningful graph representations.

Semi-Supervised Learning Tasks: The effectiveness of InfoGraph* was evaluated using the QM9 dataset for molecular property prediction. The semi-supervised approach was shown to outperform both fully supervised models and the Mean Teacher method, across a multitude of targets. Notably, InfoGraph* consistently provided lower error rates, underscoring the advantage of utilizing mutual information maximization to transfer knowledge from unlabeled data.

Implications and Future Directions

Theoretical Implications: The paper highlights the potential of mutual information maximization for graph-level representation learning. By leveraging hierarchical substructure information, InfoGraph captures comprehensive graph representations that are resilient to the limitations of traditional and other unsupervised methods. This provides a new direction for unsupervised representation learning on graph-structured data.

Practical Applications: Practically, the method has far-reaching implications. In domains such as computational chemistry and social network analysis, where labeled data can be scarce or expensive to obtain, methods like InfoGraph* can significantly enhance model performance by effectively utilizing unlabeled data. This has potential applications in drug discovery, where molecular properties need to be predicted efficiently with limited labeled samples.

Future Developments: The research paves the way for future work exploring enhanced semi-supervised and unsupervised frameworks specifically designed for graph data. For instance, incorporating attention mechanisms within InfoGraph could further refine the way substructure information contributes to the final graph-level representation. Additionally, investigating the scalability and performance of InfoGraph on larger and more complex datasets will be crucial for broadening its applicability.

Conclusion

This paper presents a robust framework for unsupervised and semi-supervised graph representation learning through mutual information maximization. InfoGraph and its semi-supervised counterpart InfoGraph* are shown to be competitive with state-of-the-art methods, validating the efficacy of the approach. This contribution extends our capabilities to work with graph-structured data, offering substantial improvements in both theoretical understanding and practical performance across various applications.