Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generalizing Denoising to Non-Equilibrium Structures Improves Equivariant Force Fields (2403.09549v3)

Published 14 Mar 2024 in cs.LG, cs.AI, physics.comp-ph, and physics.chem-ph

Abstract: Understanding the interactions of atoms such as forces in 3D atomistic systems is fundamental to many applications like molecular dynamics and catalyst design. However, simulating these interactions requires compute-intensive ab initio calculations and thus results in limited data for training neural networks. In this paper, we propose to use denoising non-equilibrium structures (DeNS) as an auxiliary task to better leverage training data and improve performance. For training with DeNS, we first corrupt a 3D structure by adding noise to its 3D coordinates and then predict the noise. Different from previous works on denoising, which are limited to equilibrium structures, the proposed method generalizes denoising to a much larger set of non-equilibrium structures. The main difference is that a non-equilibrium structure does not correspond to local energy minima and has non-zero forces, and therefore it can have many possible atomic positions compared to an equilibrium structure. This makes denoising non-equilibrium structures an ill-posed problem since the target of denoising is not uniquely defined. Our key insight is to additionally encode the forces of the original non-equilibrium structure to specify which non-equilibrium structure we are denoising. Concretely, given a corrupted non-equilibrium structure and the forces of the original one, we predict the non-equilibrium structure satisfying the input forces instead of any arbitrary structures. Since DeNS requires encoding forces, DeNS favors equivariant networks, which can easily incorporate forces and other higher-order tensors in node embeddings. We study the effectiveness of training equivariant networks with DeNS on OC20, OC22 and MD17 datasets and demonstrate that DeNS can achieve new state-of-the-art results on OC20 and OC22 and significantly improve training efficiency on MD17.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  2. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1), May 2022. doi: 10.1038/s41467-022-29939-5. URL https://doi.org/10.1038/s41467-022-29939-5.
  3. Geometric and physical quantities improve e(3) equivariant message passing. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=_xwr8gOBeV1.
  4. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  5. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 10.1021/acscatal.0c04525.
  6. Machine learning of accurate energy-conserving molecular force fields. Science Advances, 3(5):e1603015, 2017. doi: 10.1126/sciadv.1603015. URL https://www.science.org/doi/abs/10.1126/sciadv.1603015.
  7. Towards exact molecular dynamics simulations with machine-learned force fields. Nature Communications, 9(1), sep 2018. doi: 10.1038/s41467-018-06169-2.
  8. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
  9. BERT: Pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805, 2019.
  10. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  11. May the force be with you: Unified force-centric pre-training for 3d molecular conformations. arXiv preprint arXiv:2308.14759, 2023a.
  12. Fractional denoising for 3D molecular pre-training. In International Conference on Machine Learning (ICML), 2023b.
  13. Se(3)-transformers: 3d roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  14. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICLR), 2020.
  15. GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets. Transactions on Machine Learning Research (TMLR), 2022.
  16. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453, 2022.
  17. e3nn/e3nn: 2022-04-13, April 2022. URL https://doi.org/10.5281/zenodo.6459381.
  18. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
  19. Simple GNN regularisation for 3d molecular property prediction and beyond. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=1wVvweK3oIb.
  20. Improved adsorption energetics within density-functional theory using revised perdew-burke-ernzerhof functionals. Phys. Rev. B, 1999.
  21. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  22. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016.
  23. Energy-motivated equivariant pretraining for 3d molecular graphs. arXiv preprint arXiv:2207.08824, 2022.
  24. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=1YLJDvSx6J4.
  25. Clebsch–gordan nets: a fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems 32, pp.  10117–10126, 2018.
  26. AdsorbML: Accelerating adsorption energy calculations with machine learning. arXiv preprint arXiv:2211.16486, 2022.
  27. Equivariant graph attention networks for molecular property prediction. arXiv preprint arXiv:2202.09891, 2022.
  28. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=KwmPfARgOTD.
  29. EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations. arxiv preprint arxiv:2306.12059, 2023.
  30. Molecular geometry pretraining with SE(3)-invariant denoising distance matching. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=CjTHVo1dvR.
  31. Relevance of rotationally equivariant convolutions for predicting molecular properties. arxiv preprint arxiv:2008.08461, 2020.
  32. Learning local equivariant representations for large-scale atomistic dynamics. arxiv preprint arxiv:2204.05249, 2022.
  33. Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic size. arXiv preprint arXiv:2304.10061, 2023.
  34. Pubchemqc project: A large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57 6:1300–1308, 2017.
  35. Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs. In International Conference on Machine Learning (ICML), 2023.
  36. A recipe for cracking the quantum scaling limit with machine learned electron densities. Machine Learning: Science and Technology, 4(1):015027, feb 2023. doi: 10.1088/2632-2153/acb314. URL https://dx.doi.org/10.1088/2632-2153/acb314.
  37. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
  38. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp.  1–6. IEEE, 2018.
  39. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52(11):2864–2875, 2012. doi: 10.1021/ci300415d. URL https://doi.org/10.1021/ci300415d. PMID: 23088335.
  40. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning (ICML), 2020.
  41. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML), 2021.
  42. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  43. Quantum-chemical insights from deep tensor neural networks. Nature Communications, 8(1), jan 2017. doi: 10.1038/ncomms13890.
  44. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning (ICML), 2021.
  45. Ani-1: A data set of 20m off-equilibrium dft calculations for organic molecules. arXiv preprint arXiv:1708.04987, 2017.
  46. Less is more: sampling chemical space with active learning. The Journal of chemical physics, 2018.
  47. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems (NeurIPS), 2019.
  48. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
  49. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
  50. UL2: Unifying language learning paradigms. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=6ruVLB727MC.
  51. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=zNHzqZ9wrRB.
  52. Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds. arxiv preprint arXiv:1802.08219, 2018.
  53. Geometric prediction: Moving beyond scalars. arXiv preprint arXiv:2006.14163, 2020.
  54. The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysis. arXiv preprint arXiv:2206.08917, 2022.
  55. SE(3)-equivariant prediction of molecular wavefunctions and electronic densities. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https://openreview.net/forum?id=auGY2UQfhSu.
  56. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  57. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), 2008.
  58. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 2010.
  59. Denoise pretraining on nonequilibrium molecules for accurate and transferable neural potentials. Journal of Chemical Theory and Computation, 2023.
  60. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems 32, pp.  10402–10413, 2018.
  61. Pre-training via denoising for molecular property prediction. In International Conference on Learning Representations (ICLR), 2023.
  62. Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics. Phys. Rev. Lett., 120:143001, Apr 2018. doi: 10.1103/PhysRevLett.120.143001. URL https://link.aps.org/doi/10.1103/PhysRevLett.120.143001.
  63. Spherical channels for modeling atomic interactions. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets