Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Automated 3D Pre-Training for Molecular Property Prediction (2306.07812v2)

Published 13 Jun 2023 in q-bio.QM, cs.AI, and cs.LG

Abstract: Molecular property prediction is an important problem in drug discovery and materials science. As geometric structures have been demonstrated necessary for molecular property prediction, 3D information has been combined with various graph learning methods to boost prediction performance. However, obtaining the geometric structure of molecules is not feasible in many real-world applications due to the high computational cost. In this work, we propose a novel 3D pre-training framework (dubbed 3D PGT), which pre-trains a model on 3D molecular graphs, and then fine-tunes it on molecular graphs without 3D structures. Based on fact that bond length, bond angle, and dihedral angle are three basic geometric descriptors corresponding to a complete molecular 3D conformer, we first develop a multi-task generative pre-train framework based on these three attributes. Next, to automatically fuse these three generative tasks, we design a surrogate metric using the \textit{total energy} to search for weight distribution of the three pretext task since total energy corresponding to the quality of 3D conformer.Extensive experiments on 2D molecular graphs are conducted to demonstrate the accuracy, efficiency and generalization ability of the proposed 3D PGT compared to various pre-training baselines.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Simon Axelrod and Rafael Gomez-Bombarelli. 2020. Molecular machine learning with conformer ensembles. arXiv preprint arXiv:2012.08452 (2020).
  2. Simon Axelrod and Rafael Gomez-Bombarelli. 2022. GEOM, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data 9, 1 (2022), 1–14.
  3. Open catalyst 2020 (OC20) dataset and community challenges. ACS Catalysis 11, 10 (2021), 6059–6072.
  4. A simple framework for contrastive learning of visual representations. In ICML. PMLR, 1597–1607.
  5. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems 33 (2020), 13260–13271.
  6. A Crum-Brown and Thomas R Fraser. 1865. The connection of chemical constitution and physiological action. Trans R Soc Edinb 25, 1968-1969 (1865), 257.
  7. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  8. Machine learning prediction errors better than DFT accuracy. arXiv preprint arXiv:1702.05532 (2017).
  9. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. PMLR, 1126–1135.
  10. Daan Frenkel and Berend Smit. 2001. Understanding molecular simulation: from algorithms to applications. Vol. 1. Elsevier.
  11. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems 34 (2021), 6790–6802.
  12. Neural message passing for quantum chemistry. In International conference on machine learning. PMLR, 1263–1272.
  13. Corwin Hansch and Toshio Fujita. 1964. p-σ𝜎\sigmaitalic_σ-π𝜋\piitalic_π Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. Journal of the American Chemical Society 86, 8 (1964), 1616–1626.
  14. Masked autoencoders are scalable vision learners. In CVPR. 16000–16009.
  15. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430 (2021).
  16. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 (2019).
  17. Gpt-gnn: Generative pre-training of graph neural networks. In KDD. 1857–1867.
  18. Global self-attention as a replacement for graph convolution. In KDD. 655–665.
  19. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016).
  20. Pure transformers are powerful graph learners. arXiv preprint arXiv:2207.02505 (2022).
  21. A novel Method for calculation of Molecular energies and charge Distributions by thermodynamic formalization. Scientific Reports 9, 1 (2019), 1–12.
  22. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123 (2020).
  23. Greg Landrum et al. 2016. Rdkit: Open-source cheminformatics software. 2016. URL http://www. rdkit. org/, https://github. com/rdkit/rdkit 149, 150 (2016), 650.
  24. David R Lide. 2004. CRC handbook of chemistry and physics. Vol. 85. CRC press.
  25. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).
  26. GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactions. (2022).
  27. Pre-training Molecular Graph Representation with 3D Geometry. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning.
  28. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013 (2021).
  29. One transformer can understand both 2d & 3d molecular data. arXiv preprint arXiv:2210.01765 (2022).
  30. GPS++: An Optimised Hybrid MPNN/Transformer for Molecular Property Prediction. arXiv preprint arXiv:2212.02229 (2022).
  31. Compendium of chemical terminology. Vol. 1669. Blackwell Science Oxford.
  32. Maho Nakata and Tomomi Shimazaki. 2017. PubChemQC project: a large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling 57, 6 (2017), 1300–1308.
  33. Artificial intelligence techniques for bioinformatics. Applied bioinformatics 1 (2002), 191–222.
  34. GRPE: Relative Positional Encoding for Graph Transformer. In ICLR2022 Machine Learning for Drug Discovery.
  35. Robert G Parr. 1983. Density functional theory. Annual Review of Physical Chemistry 34, 1 (1983), 631–656.
  36. Fabian Pedregosa. 2016. Hyperparameter optimization with approximate gradient. In International conference on machine learning. PMLR, 737–746.
  37. Gcc: Graph contrastive coding for graph neural network pre-training. In KDD. 1150–1160.
  38. Improving language understanding by generative pre-training. (2018).
  39. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data 1, 1 (2014), 1–7.
  40. Recipe for a General, Powerful, Scalable Graph Transformer. arXiv preprint arXiv:2205.12454 (2022).
  41. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems 33 (2020), 12559–12571.
  42. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning. PMLR, 9377–9388.
  43. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning. PMLR, 20479–20502.
  44. James JP Stewart. 2007. Stewart computational chemistry. http://openmopac. net/ (2007).
  45. A deep learning approach to antibiotic discovery. Cell 180, 4 (2020), 688–702.
  46. Does GNN Pretraining Help Molecular Representation?. In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, et al. (Eds.).
  47. A graph neural network-based interpretable framework reveals a novel DNA fragility–associated chromatin structural unit. Genome Biology 24, 1 (2023), 90.
  48. End-to-end learning on 3d protein structure for interface prediction. Advances in Neural Information Processing Systems 32 (2019).
  49. The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysis. arXiv preprint arXiv:2206.08917 (2022).
  50. Graph Property Prediction on Open Graph Benchmark: A Winning Solution by Graph Neural Architecture Search. arXiv preprint arXiv:2207.06027 (2022).
  51. An ensemble of VisNet, Transformer-M, and pretraining models for molecular property prediction in OGB 2022. arXiv preprint arXiv:2211.12791 (2022).
  52. ViSNet: a scalable and accurate geometric deep learning potential for molecular dynamics simulation. arXiv preprint arXiv:2210.16518 (2022).
  53. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems 34 (2021), 13266–13279.
  54. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).
  55. Learning neural generative dynamics for molecular conformation generation. arXiv preprint arXiv:2102.10240 (2021).
  56. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems 34 (2021), 28877–28888.
  57. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133 (2022).
  58. Uni-Mol: A Universal 3D Molecular Representation Learning Framework. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=6K2RM6wVqKu
  59. Unified 2d and 3d pre-training of molecular representations. In KDD. 2626–2636.
  60. Spherical Channels for Modeling Atomic Interactions. arXiv preprint arXiv:2206.14331 (2022).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Xu Wang (319 papers)
  2. Huan Zhao (109 papers)
  3. Weiwei Tu (9 papers)
  4. Quanming Yao (102 papers)
Citations (33)
View on Semantic Scholar