GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks (2302.08043v3)

Abstract: Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks(GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily rely on a large amount of task-specific supervision. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-train model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt.

• The paper introduces a unified subgraph similarity learning framework that bridges pre-training and downstream tasks for graph neural networks.
• It leverages a task-specific learnable prompt to guide the aggregation process, resulting in improved efficiency and accuracy, especially in few-shot learning scenarios.
• Experimental evaluations on multiple datasets demonstrate that GraphPrompt consistently outperforms state-of-the-art GNN models and pre-training methods.

Overview of "GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks"

The paper "GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks" addresses the challenge of bridging the gap between pre-training and downstream tasks in graph neural networks (GNNs) through the introduction of a novel framework termed GraphPrompt. This approach is motivated by the need for more efficient utilization of pre-trained models in downstream tasks, such as node and graph classification, without the large computational and data requirements associated with traditional pre-training and fine-tuning paradigms.

GraphPrompt proposes a unified framework that reformulates both pre-training and downstream tasks into a common template centered around subgraph similarity learning. This template is pivotal in aligning the objectives of pre-training with those of the downstream tasks, thus facilitating more effective transfer of pre-trained knowledge. The central innovation of GraphPrompt lies in its use of a learnable prompt, which acts as a mechanism to guide downstream tasks in leveraging relevant subgraph information from the pre-trained GNN model. By focusing on graph topology and subgraph structures, the authors successfully unify node and graph classification tasks into a coherent framework that reduces the inconsistency typically observed in step transitions between pre-training objectives and downstream goals.

Technical Contributions

The paper introduces the following key components:

1. Unified Subgraph Similarity Learning: The framework converts various tasks into a subgraph similarity computation problem, enabling consistency across pre-training and downstream applications. Pre-training is based on a self-supervised link prediction task that can be generalized to other tasks, while node and graph classification tasks are mapped to classifying subgraph representations based on similarity to class prototypes.
2. Learnable Prompt for Task-Specific Aggregation: The deployment of a task-specific learnable prompt distinguishes GraphPrompt from conventional language-based prompts. This prompt influences the $ReadOut$ operation, allowing different forms of aggregation suitable for different downstream tasks. This results in improved efficiency and accuracy, especially in few-shot learning scenarios.
3. Few-Shot Capability: The proposed framework is designed to operate effectively in few-shot learning setups, significantly reducing the amount of labeled data required to achieve competitive performance. This attribute is particularly relevant in domains where labeled graph data is scarce.

Experimental Evaluation

GraphPrompt was empirically evaluated on five publicly available datasets: Flickr, PROTEINS, COX2, ENZYMES, and BZR, covering both node and graph classification tasks. The experimental results demonstrate that GraphPrompt consistently outperforms state-of-the-art GNN models (e.g., GCN, GraphSAGE, GAT, GIN) and pre-training methods (e.g., DGI, InfoGraph, GraphCL). Notably, in few-shot learning scenarios, GraphPrompt exhibits superior performance, highlighting its efficacy in leveraging pre-trained knowledge efficiently.

Implications and Future Directions

GraphPrompt presents a critical advancement in the field of graph neural networks by addressing the oft-cited issue of pre-training versus downstream task alignment. Its ability to unify these processes while introducing task-specific prompts offers a promising avenue for further research and application, particularly in few-shot contexts.

Moving forward, the integration of more complex prompt structures or adaptive prompt mechanisms could enhance the flexibility and adaptability of the model to dynamically changing tasks. Additionally, exploring GraphPrompt's applicability beyond the scope of traditional graph tasks—such as in dynamic graph environments or heterogeneous graph structures—could further enhance its utility and expand its application domain.

In summary, by efficiently unifying and streamlining the pre-training and prompting phases for graph neural networks, GraphPrompt sets the stage for more robust and versatile applications of GNNs in diverse and data-constrained environments.