D-VAE: A Variational Autoencoder for Directed Acyclic Graphs (1904.11088v4)

Abstract: Graph structured data are abundant in the real world. Among different graph types, directed acyclic graphs (DAGs) are of particular interest to machine learning researchers, as many machine learning models are realized as computations on DAGs, including neural networks and Bayesian networks. In this paper, we study deep generative models for DAGs, and propose a novel DAG variational autoencoder (D-VAE). To encode DAGs into the latent space, we leverage graph neural networks. We propose an asynchronous message passing scheme that allows encoding the computations on DAGs, rather than using existing simultaneous message passing schemes to encode local graph structures. We demonstrate the effectiveness of our proposed DVAE through two tasks: neural architecture search and Bayesian network structure learning. Experiments show that our model not only generates novel and valid DAGs, but also produces a smooth latent space that facilitates searching for DAGs with better performance through Bayesian optimization.

• The paper presents D-VAE—a variational autoencoder that injectively encodes DAG computations to optimize complex computational tasks.
• It introduces an asynchronous message passing scheme that respects dependency order, allowing accurate mapping of DAG structures into latent space.
• Empirical results demonstrate near-perfect reconstruction and enhanced predictive performance in neural architecture search and Bayesian network learning.

D-VAE: A Variational Autoencoder for Directed Acyclic Graphs

This paper introduces a novel deep generative model for directed acyclic graphs (DAGs) known as DAG variational autoencoder (D-VAE), specifically designed to optimize DAG structures that represent computational tasks. DAGs play a crucial role in various machine learning models and systems such as neural networks, Bayesian networks, and electronic circuit design. Traditional methods of optimizing DAG structures present challenges primarily due to their discrete nature, which complicates the application of black-box optimization techniques that usually work in continuous spaces. The proposed D-VAE enables the inclusion of all DAGs into a continuous latent space, facilitating the optimization process through established continuous optimization techniques like Bayesian optimization.

Core Methodological Innovations

The primary innovation of D-VAE lies in its asynchronous message passing scheme, a departure from the synchronous scheme used in standard graph neural networks. This asynchronous model respects the computation dependencies embedded in DAG structures, effectuating a direct mapping of DAG computations rather than merely encoding local graph structures. The asynchronous scheme ensures that nodes can only pass messages once all predecessor computations have been completed, adhering to their inherent sequential nature.

Through the encoder component, D-VAE guarantees injective encoding of computations—two DAGs representing identical computations result in identical encodings. This injective mapping is critical as it ensures the latent space accurately reflects computational equivalence, thus aiding performance-driven searches in the latent space. Leveraging the powerful approximation capabilities of neural networks, D-VAE uses specific aggregation and update functions modeled by neural networks to injectively encode computations on DAG.

Key Results and Implications

The empirical evaluation demonstrates D-VAE's efficacy across various tasks, primarily focusing on neural architecture search and Bayesian network structure learning. It exhibits robust reconstructive and generative capabilities evidenced by near-perfect reconstruction accuracy and the ability to decode novel and valid DAG structures. The predictive performance metrics highlight that D-VAE's latent representations correlate strongly with network performance indicators, facilitating effective Bayesian optimization.

The paper's experimental results underline the superiority of D-VAE over baseline methods (including string-based, adjacency-matrix-based, and synchronous message-passing models) by showcasing better predictive performance in terms of root mean square error (RMSE) and Pearson correlation coefficient regarding DAG evaluation metrics. D-VAE's superior predictive ability illustrates the importance of encoding computation rather than simply structural characteristics.

Furthermore, visualization of the latent space via methods such as great circle interpolation reveals smooth transitions between neural architectures in the latent space, underscoring D-VAE's capability to support optimization algorithms focused on discovering high-performance DAGs. The smoothness permits efficient navigation of the latent space using global optimization techniques, thus addressing the challenges posed by discrete optimization scenarios.

Future Directions

The paper points toward promising future research avenues involving the expansion of D-VAE's framework to more complex DAG applications, including large-scale circuit designs and other computation-driven DAG systems. Additionally, incorporating vertex semantics to reflect functional similarities between nodes in the latent space could significantly enhance performance prediction accuracy. The findings advocate for continued exploration of generative models for discrete optimization challenges, emphasizing the practical benefits of translating complex discrete structures into manageable continuous spaces.