Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations (2202.02514v3)

Abstract: Generating graph-structured data requires learning the underlying distribution of graphs. Yet, this is a challenging problem, and the previous graph generative methods either fail to capture the permutation-invariance property of graphs or cannot sufficiently model the complex dependency between nodes and edges, which is crucial for generating real-world graphs such as molecules. To overcome such limitations, we propose a novel score-based generative model for graphs with a continuous-time framework. Specifically, we propose a new graph diffusion process that models the joint distribution of the nodes and edges through a system of stochastic differential equations (SDEs). Then, we derive novel score matching objectives tailored for the proposed diffusion process to estimate the gradient of the joint log-density with respect to each component, and introduce a new solver for the system of SDEs to efficiently sample from the reverse diffusion process. We validate our graph generation method on diverse datasets, on which it either achieves significantly superior or competitive performance to the baselines. Further analysis shows that our method is able to generate molecules that lie close to the training distribution yet do not violate the chemical valency rule, demonstrating the effectiveness of the system of SDEs in modeling the node-edge relationships. Our code is available at https://github.com/harryjo97/GDSS.

Score-based Generative Modeling of Graphs via the System of SDEs

The paper under discussion presents an advanced framework for the generation of graph-structured data, employing score-based generative modeling integrated with a system of stochastic differential equations (SDEs). This work aims to address the challenges inherent in graph generation, particularly the difficulty of capturing the permutation-invariance and complex dependencies between nodes and edges that are vital for modeling real-world graphs, such as molecular graphs.

Summary of Contributions

1. Graph Diffusion Process: The authors devise a novel graph generation framework featuring a continuous-time diffusion process that models both node features and adjacency matrices simultaneously. This is achieved through a system of stochastic differential equations (GDSS), effectively enabling the modeling and generation of graphs in a permutation-invariant manner that previous autoregressive and one-shot models struggled with.
2. Score Matching Objectives: The paper introduces new score matching objectives designed specifically for this diffusion process. These training objectives estimate the gradients of the joint log-density with respect to each component of the graph, thus allowing for effective training of the model.
3. Efficient Sampling Mechanism: A new integrator is presented to solve the system of reverse-time SDEs efficiently, facilitating the generation of samples from learned distributions.
4. Empirical Validation: Comprehensive experiments demonstrate the superiority of this approach over existing models on several datasets, covering synthetic and real-world graph generation tasks. Strong numerical results indicate that GDSS either significantly outperforms or is competitive with state-of-the-art methods.

Numerical Results and Observations

The results underscore GDSS's ability to model complex dependencies effectively. The generated samples on synthetic and real-world datasets showed that GDSS effectively learns the underlying distributions of graphs. Particularly, the method achieved competitive performance in molecule generation tasks, a domain where understanding node-edge dependencies is crucial. The validity of generated molecular structures without requiring correction indicates that the model adheres closely to chemical valency rules.

Practical and Theoretical Implications

Practically, GDSS offers a powerful tool for applications in drug design and neural architecture search, where complex graph structures are prevalent. Theoretically, the integration of score-based generative modeling with a graph diffusion process suggests new directions for handling data with inherent structural relationships.

Potential Impact and Future Research

This approach paves the way for future research in improving diffusion processes for structured data. Additional studies might explore how these methods could integrate with latent variable models or be applied to dynamic graphs. Moreover, potential developments could involve experimenting with different types of SDEs tailored to particular graph characteristics, thus broadening the applicability and effectiveness of the GDSS framework in various domains.

Overall, this paper presents a significant advancement in generative modeling for graph structures, which are essential for a broad array of scientific and industrial applications. The novel use of score-based frameworks alongside a system of SDEs suggests promising avenues for both current applications and future research within artificial intelligence and data modeling disciplines.