Papers
Topics
Authors
Recent
Search
2000 character limit reached

Graph Diffusion Policy Optimization

Published 26 Feb 2024 in cs.LG, cs.AI, and cs.CE | (2402.16302v2)

Abstract: Recent research has made significant progress in optimizing diffusion models for downstream objectives, which is an important pursuit in fields such as graph generation for drug design. However, directly applying these models to graph presents challenges, resulting in suboptimal performance. This paper introduces graph diffusion policy optimization (GDPO), a novel approach to optimize graph diffusion models for arbitrary (e.g., non-differentiable) objectives using reinforcement learning. GDPO is based on an eager policy gradient tailored for graph diffusion models, developed through meticulous analysis and promising improved performance. Experimental results show that GDPO achieves state-of-the-art performance in various graph generation tasks with complex and diverse objectives. Code is available at https://github.com/sail-sg/GDPO.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Efficient graphlet counting for large networks. In 2015 IEEE international conference on data mining, pp.  1–10. IEEE, 2015.
  2. Structured denoising diffusion models in discrete state-spaces. ArXiv, abs/2107.03006, 2021.
  3. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862, 2022.
  4. Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of Cheminformatics, 7, 2015.
  5. Quantifying the chemical beauty of drugs. Nature chemistry, 4 2:90–8, 2012.
  6. Training diffusion models with reinforcement learning. ArXiv, abs/2305.13301, 2023.
  7. Memory-assisted reinforcement learning for diverse molecular de novo design. Journal of Cheminformatics, 12, 2020.
  8. Structure and reaction based evaluation of synthetic accessibility. Journal of Computer-Aided Molecular Design, 21:311–325, 2007.
  9. Generative code modeling with graphs. ArXiv, abs/1805.08490, 2018.
  10. Molgan: An implicit generative model for small molecular graphs. ArXiv, abs/1805.11973, 2018.
  11. Generative models should at least be able to design molecules that dock well: A new benchmark. Journal of Chemical Information and Modeling, 63:3238 – 3247, 2020.
  12. Directly fine-tuning diffusion models on differentiable rewards. ArXiv, abs/2309.17400, 2023.
  13. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233, 2021.
  14. Autodock vina 1.2. 0: New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, 61(8):3891–3898, 2021.
  15. Limo: Latent inceptionism for targeted molecule generation. Proceedings of machine learning research, 162:5777–5792, 2022.
  16. On the evolution of random graphs. Transactions of the American Mathematical Society, 286:257–257, 1984.
  17. Optimizing ddpm sampling with shortcut fine-tuning. In International Conference on Machine Learning, 2023.
  18. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. ArXiv, abs/2305.16381, 2023.
  19. Handbook of Markov decision processes: methods and applications, volume 40. Springer Science & Business Media, 2012.
  20. Freeman, L. The development of social network analysis. A Study in the Sociology of Science, 1(687):159–167, 2004.
  21. A kernel two-sample test. J. Mach. Learn. Res., 13:723–773, 2012.
  22. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6):1291–1307, 2012.
  23. Graphite: Iterative generative modeling of graphs. In International Conference on Machine Learning, 2018.
  24. A systematic survey on deep generative models for graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:5370–5390, 2020.
  25. On eulerian and hamiltonian graphs and line graphs. Canadian Mathematical Bulletin, 8:701 – 709, 1965.
  26. Semi-implicit graph variational auto-encoders. ArXiv, abs/1908.07078, 2019.
  27. Denoising diffusion probabilistic models. ArXiv, abs/2006.11239, 2020.
  28. Stochastic blockmodels: First steps. Social Networks, 5:109–137, 1983.
  29. Zinc: A free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 52:1757 – 1768, 2012.
  30. Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors. Scientific Reports, 10, 2020.
  31. Hierarchical generation of molecular graphs using structural motifs. In International Conference on Machine Learning, 2020a.
  32. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, 2020b.
  33. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, 2022.
  34. Elucidating the design space of diffusion-based generative models. ArXiv, abs/2206.00364, 2022.
  35. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30:595–608, 2016.
  36. A machine learning-based method to improve docking scoring functions and its application to drug repurposing. Journal of chemical information and modeling, 51 2:408–19, 2011.
  37. Variational graph auto-encoders. ArXiv, abs/1611.07308, 2016.
  38. Aligning text-to-image models using human feedback. ArXiv, abs/2302.12192, 2023.
  39. Mgcvae: Multi-objective inverse design via molecular graph conditional variational autoencoder. Journal of chemical information and modeling, 2022.
  40. Exploring chemical space with score-based out-of-distribution generation. ArXiv, abs/2206.07632, 2022.
  41. Efficient graph generation with graph recurrent attention networks. Advances in neural information processing systems, 32, 2019a.
  42. Efficient graph generation with graph recurrent attention networks. In Neural Information Processing Systems, 2019b.
  43. Graph normalizing flows. ArXiv, abs/1905.13177, 2019.
  44. Constrained graph variational autoencoders for molecule design. In Neural Information Processing Systems, 2018.
  45. Graphdf: A discrete flow model for molecular graph generation. In International Conference on Machine Learning, 2021.
  46. Spectre : Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In International Conference on Machine Learning, 2022.
  47. Agl-score: Algebraic graph learning score for protein-ligand binding scoring, ranking, docking, and screening. Journal of chemical information and modeling, 2019.
  48. Reinforcement learning for bandit neural machine translation with simulated human feedback. ArXiv, abs/1707.07402, 2017.
  49. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, 2020.
  50. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9, 2017.
  51. Software for molecular docking: a review. Biophysical Reviews, 9:91 – 102, 2017.
  52. Molecular sets (moses): A benchmarking platform for molecular generation models. Frontiers in Pharmacology, 11, 2018.
  53. Aligning text-to-image diffusion models with reward backpropagation. ArXiv, abs/2310.03739, 2023.
  54. Conditional constrained graph variational autoencoders for molecule design. 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp.  729–736, 2020.
  55. Graphaf: a flow-based autoregressive model for molecular graph generation. ArXiv, abs/2001.09382, 2020.
  56. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, 2018.
  57. Network clustering coefficient without degree-correlation biases. Physical Review E, 71(5):057101, 2005.
  58. Deep unsupervised learning using nonequilibrium thermodynamics. ArXiv, abs/1503.03585, 2015.
  59. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011.13456, 2020.
  60. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 9:1054–1054, 1998.
  61. Autodock vina: Improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of Computational Chemistry, 31, 2009.
  62. Digress: Discrete denoising diffusion for graph generation. ArXiv, abs/2209.14734, 2022.
  63. Graphgan: Graph representation learning with generative adversarial nets. ArXiv, abs/1711.08267, 2017.
  64. Exploring randomly wired neural networks for image recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  1284–1293, 2019.
  65. Mars: Markov molecular sampling for multi-objective drug discovery. ArXiv, abs/2103.10432, 2021.
  66. Conditional structure generation through graph variational generative adversarial nets. In Neural Information Processing Systems, 2019.
  67. Hit and lead discovery with explorative rl and fragment-based molecule generation. ArXiv, abs/2110.01219, 2021.
  68. Graph convolutional policy network for goal-directed molecular graph generation. In Neural Information Processing Systems, 2018a.
  69. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International Conference on Machine Learning, 2018b.
  70. A survey on graph diffusion models: Generative ai in science for molecule, protein and material. ArXiv, abs/2304.01565, 2023.
  71. Optimization of molecules via deep reinforcement learning. Scientific Reports, 9, 2018.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up