Graph Generative Pre-trained Transformer (2501.01073v2)

Abstract: Graph generation is a critical task in numerous domains, including molecular design and social network analysis, due to its ability to model complex relationships and structured data. While most modern graph generative models utilize adjacency matrix representations, this work revisits an alternative approach that represents graphs as sequences of node set and edge set. We advocate for this approach due to its efficient encoding of graphs and propose a novel representation. Based on this representation, we introduce the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns graph structures via next-token prediction. To further exploit G2PT's capabilities as a general-purpose foundation model, we explore fine-tuning strategies for two downstream applications: goal-oriented generation and graph property prediction. We conduct extensive experiments across multiple datasets. Results indicate that G2PT achieves superior generative performance on both generic graph and molecule datasets. Furthermore, G2PT exhibits strong adaptability and versatility in downstream tasks from molecular design to property prediction. Code available at https://github.com/tufts-ml/G2PT,

• The paper introduces a novel sequence-based graph representation that transforms graph generation into a next-token prediction task using a transformer decoder.
• It demonstrates strong performance on molecular datasets like QM9, MOSES, and GuacaMol, achieving high validity and uniqueness metrics.
• The work highlights G2PT's adaptability through fine-tuning for tasks including graph property prediction and goal-oriented molecular design using reinforcement learning.

Graph Generative Pre-trained Transformer

The paper "Graph Generative Pre-trained Transformer" introduces an innovative approach in the domain of graph generation through the proposal of the Graph Generative Pre-trained Transformer (G2PT). This model leverages an auto-regressive framework to efficiently learn graph structures by transforming them into sequences. The paper's central assertion is that representing graphs as sequences of node and edge definitions, as opposed to traditional adjacency matrix-based representations, offers computational and performance advantages in graph generative tasks.

The G2PT approach stands out for several key methodological innovations. Firstly, the authors propose a sequence-based representation for graphs that eschews the dense and often cumbersome adjacency matrix for a more streamlined format. This format is composed of node and edge definitions that introduce efficiency in data encoding. Transforming the graph generation problem into a next-token prediction task, the G2PT employs a transformer decoder architecture to predict graph structures in an auto-regressive manner.

The training strategy employs extensive experiments across varied datasets, notably molecular datasets like QM9, MOSES, and GuacaMol, alongside generic graph data such as Planar and Tree structures. The G2PT model achieves commendable performance, often surpassing state-of-the-art models in key metrics. For instance, in molecular datasets, G2PT's validity and uniqueness scores were notably high, reflecting its strong generative capabilities.

Further, the paper articulates the adaptability of G2PT for downstream tasks through fine-tuning techniques aimed at applications like goal-oriented generation and graph property prediction. This is particularly beneficial in fields like drug discovery, where graph generative models are employed to evoke molecules with desired chemical properties. Fine-tuning is carried out using strategies like rejection sampling and reinforcement learning, showcasing the model's ability to adjust to task-specific requirements.

Importantly, the authors engage with the scaling behavior of G2PT, examining how adjustments in model size and training data affect performance, a critical insight for scaling such models appropriately in real-world applications. This provides valuable guidance on how G2PT might be expanded or compressed to fit specific computational constraints or task demands.

One of the primary implications of this work lies in its potential to transform graph generation tasks across a wide array of applications, from molecular design to social network analysis. By moving away from the computationally burdensome adjacency matrix approach, G2PT allows for more agile and scalable graph generative solutions. Furthermore, its adaptability through fine-tuning reinforces its utility across varied domains where specific graph properties are desired.

Future research directions following this work might explore more universal strategies for graph sequence ordering, as G2PT is sensitive to the order in which graph components are generated. Additionally, integrating domain-specific knowledge into the generation process could further enhance the efficacy and adaptability of the model.

In summary, the G2PT offers a compelling advancement in graph generation techniques, marrying the strengths of transformer-based models with an innovative sequence representation, and thus opening new possibilities for efficient and versatile graph generative modeling.