Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 64 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 30 tok/s Pro
GPT-5 High 35 tok/s Pro
GPT-4o 77 tok/s Pro
Kimi K2 174 tok/s Pro
GPT OSS 120B 457 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

DPA-2: a large atomic model as a multi-task learner (2312.15492v2)

Published 24 Dec 2023 in physics.chem-ph, cond-mat.mtrl-sci, and physics.comp-ph

Abstract: The rapid advancements in AI are catalyzing transformative changes in atomic modeling, simulation, and design. AI-driven potential energy models have demonstrated the capability to conduct large-scale, long-duration simulations with the accuracy of ab initio electronic structure methods. However, the model generation process remains a bottleneck for large-scale applications. We propose a shift towards a model-centric ecosystem, wherein a large atomic model (LAM), pre-trained across multiple disciplines, can be efficiently fine-tuned and distilled for various downstream tasks, thereby establishing a new framework for molecular modeling. In this study, we introduce the DPA-2 architecture as a prototype for LAMs. Pre-trained on a diverse array of chemical and materials systems using a multi-task approach, DPA-2 demonstrates superior generalization capabilities across multiple downstream tasks compared to the traditional single-task pre-training and fine-tuning methodologies. Our approach sets the stage for the development and broad application of LAMs in molecular and materials simulation research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (110)
  1. Inhomogeneous electron gas. Physical review, 136(3B):B864, 1964.
  2. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133, 1965.
  3. Combined free-energy calculation and machine learning methods for understanding ligand unbinding kinetics. Journal of chemical theory and computation, 18(4):2543–2555, 2022.
  4. Qdπ𝜋\piitalic_π: A quantum deep potential interaction model for drug discovery. Journal of Chemical Theory and Computation, 19(4):1261–1275, 2023.
  5. Machine learning a general-purpose interatomic potential for silicon. Physical Review X, 8(4):041048, 2018.
  6. A general-purpose machine-learning force field for bulk and nanostructured phosphorus. Nature communications, 11(1):1–11, 2020.
  7. Deep potentials for materials science. Materials Futures, 1(2):022601, 2022.
  8. Machine learning for atomic simulation and activity prediction in heterogeneous catalysis: current status and future. ACS Catalysis, 10(22):13213–13226, 2020.
  9. Ammonia decomposition on lithium imide surfaces: A new paradigm in heterogeneous catalysis. ChemRxiv, 2022.
  10. Unified approach for molecular dynamics and density-functional theory. Physical review letters, 55(22):2471, 1985.
  11. Dp-gen: A concurrent learning platform for the generation of reliable deep learning based potential energy models. Computer Physics Communications, 253:107206, 2020.
  12. Less is more: Sampling chemical space with active learning. The Journal of chemical physics, 148(24):241733, 2018.
  13. Active learning in gaussian process interpolation of potential energy surfaces. The Journal of chemical physics, 149(17):174114, 2018.
  14. Accurate deep potential model for the al–cu–mg alloy in the full concentration space. Chinese Physics B, 30(5):050706, 2021.
  15. A universal graph deep learning interatomic potential for the periodic table. arXiv preprint arXiv:2202.02450, 2022.
  16. Towards universal neural network potential for material discovery applicable to arbitrary combination of 45 elements. Nature Communications, 13(1):2991, 2022.
  17. Unified graph neural network force-field for the periodic table: solid state applications. Digital Discovery, 2(2):346–355, 2023.
  18. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  19. Gpip: Geometry-enhanced pre-training on interatomic potentials. arXiv preprint arXiv:2309.15718, 2023.
  20. May the force be with you: Unified force-centric pre-training for 3d molecular conformations. arXiv preprint arXiv:2308.14759, 2023.
  21. Leveraging multitask learning to improve the transferability of machine learned force fields. ChemRxiv, 2023.
  22. Denoise pre-training on non-equilibriummolecules for accurate and transferable neuralpotentials. arXiv preprint arXiv:2303.02216, 2023.
  23. Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6129–6138, 2017.
  24. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  25. From molecules to materials: Pre-training large generalizable models for atomic property prediction. arXiv preprint arXiv:2310.16802, 2023.
  26. Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning. Nature communications, 10(1):1–8, 2019.
  27. Transfer learning using attentions across atomic systems with graph neural networks (taag). The Journal of Chemical Physics, 156(18):184702, 2022.
  28. Deep learning. MIT press, 2016.
  29. Mark Tuckerman. Statistical Mechanics: Theory and Molecular Simulation. OUP Oxford, 2010.
  30. How do graph networks generalize to large and diverse molecular systems? arXiv preprint arXiv:2204.02782, 2022.
  31. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. arXiv preprint arXiv:2306.12059, 2023.
  32. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1–11, 2022.
  33. Learning local equivariant representations for large-scale atomistic dynamics. arXiv preprint arXiv:2204.05249, 2022.
  34. Generalized neural-network representation of high-dimensional potential-energy surfaces. Physical review letters, 98(14):146401, 2007.
  35. Ani-1: an extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4):3192–3203, 2017.
  36. Quantum-chemical insights from deep tensor neural networks. Nature communications, 8(1):1–8, 2017.
  37. wacsf—weighted atom-centered symmetry functions as descriptors in machine learning potentials. The Journal of chemical physics, 148(24), 2018.
  38. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Physical review letters, 120(14):143001, 2018.
  39. End-to-end symmetry preserving inter-atomic potential energy model for finite and extended systems. Advances in Neural Information Processing Systems, 31, 2018.
  40. Deepmd-kit v2: A software package for deep potential models. The Journal of Chemical Physics, 159:054801, 2023.
  41. Dpa-1: Pretraining of attention-based deep potential model for molecular simulation. arXiv preprint arXiv:2208.08236, 2022.
  42. Embedded atom neural network potentials: Efficient and accurate machine learning with a physically inspired representation. The journal of physical chemistry letters, 10(17):4962–4967, 2019.
  43. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters, 104(13):136403, 2010.
  44. On representing chemical environments. Physical Review B, 87(18):184115, 2013.
  45. Fast and accurate modeling of molecular atomization energies with machine learning. Physical review letters, 108(5):058301, 2012.
  46. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.
  47. Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials. Journal of Computational Physics, 285:316–330, 2015.
  48. Alexander V Shapeev. Moment tensor potentials: A class of systematically improvable interatomic potentials. Multiscale Modeling & Simulation, 14(3):1153–1173, 2016.
  49. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.
  50. Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. arXiv preprint arXiv:2101.03164, 2021.
  51. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
  52. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.14115, 2020.
  53. Equivariant message passing for the prediction of tensorial properties and molecular spectra. arXiv preprint arXiv:2102.03150, 2021.
  54. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790–6802, 2021.
  55. Rotation invariant graph neural networks using spin convolutions. arXiv preprint arXiv:2106.09575, 2021.
  56. Spherical channels for modeling atomic interactions. Advances in Neural Information Processing Systems, 35:8054–8067, 2022.
  57. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11990, 2022.
  58. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  59. Network representation learning: A survey. IEEE transactions on Big Data, 6(1):3–28, 2018.
  60. Improving language understanding by generative pre-training. 2018.
  61. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  62. Uni-mol: A universal 3d molecular representation learning framework. ChemRxiv, 2022.
  63. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2626–2636, 2022.
  64. Pre-training via denoisingfor molecular property prediction. arXiv preprint arXiv:2206.00133, 2022.
  65. Highly accurate quantum chemical propertyprediction with uni-mol+. arXiv preprint arXiv:2303.16982, 2023.
  66. Fractional denoising for 3d molecular pre-training. arXiv preprint arXiv:2307.10683, 2023.
  67. 3d equivariant molecular graph pretraining. arXiv preprint arXiv:2207.08824, 2022.
  68. D. Beaini and et.al. Towards foundational models for molecularlearning on large-scale multi-task datasets. arXiv:2310.04292, 2023.
  69. K.L.K. Lee and et.al. Towards foundation models for materials science: The open matsci ml toolkit. arXiv:2310.07864, 2023.
  70. Synthetic pre-training for neural-networkinteratomic potentials. arXiv preprint arXiv:2307.15714, 2023.
  71. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 11(10):6059–6072, 2021.
  72. Dp compress: A model compression scheme for generating efficient deep potential models. Journal of chemical theory and computation, 18(9):5559–5567, 2022.
  73. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Computational materials science, 6(1):15–50, 1996.
  74. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Physical review B, 54(16):11169, 1996.
  75. Gaussian 16, 2016.
  76. Systematically improvable optimized atomic basis sets for ab initio calculations. Journal of Physics: Condensed Matter, 22(44):445501, 2010.
  77. Large-scale ab initio simulations based on systematically improvable atomic basis. Computational Materials Science, 112:503–517, 2016.
  78. Gemnet-oc: developing graph neural networks for large and diverse molecular simulation datasets. arXiv preprint arXiv:2204.02782, 2022.
  79. The individual and collective effects of exact exchange and dispersion interactions on the ab initio structure of liquid water. The Journal of chemical physics, 141(8), 2014.
  80. Deep potential generation scheme and simulation protocol for the li10gep2s12-type superionic conductors. The Journal of Chemical Physics, 154(9):094703, 2021.
  81. First principles study of the li10gep2s12 lithium super ionic conductor material. Chemistry of Materials, 24(1):15–17, 2012.
  82. Ionic correlations and failure of nernst-einstein relation in solid-state electrolytes. Physical Review Materials, 1(2):025402, 2017.
  83. Single-crystal x-ray structure analysis of the superionic conductor li 10 gep 2 s 12. Physical Chemistry Chemical Physics, 15(28):11620–11622, 2013.
  84. Universal interatomic potential for perovskite oxides. Physical Review B, 108(18):L180104, 2023.
  85. Phase diagram and properties of pb (in1/2nb1/2) o3–pb (mg1/3nb2/3) o3–pbtio3 polycrystalline ceramics. Journal of the European Ceramic Society, 32(2):433–439, 2012.
  86. Soft phonon modes and diffuse scattering in pb (in1/2nb1/2) o3-pb (mg1/3nb2/3) o3-pbtio3 relaxor ferroelectrics. Journal of Materiomics, 4(4):345–352, 2018.
  87. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  88. Generalized gradient approximation made simple. Physical review letters, 77(18):3865, 1996.
  89. Cp2k: An electronic structure and molecular dynamics software package-quickstep: Efficient and accurate electronic structure calculations. The Journal of Chemical Physics, 152(19):194103, 2020.
  90. A consistent and accurate ab initio parametrization of density functional dispersion correction (dft-d) for the 94 elements h-pu. The Journal of chemical physics, 132(15), 2010.
  91. The ebi rdf platform: linked open data for the life sciences. Bioinformatics, 30(9):1338–1339, 2014.
  92. The chembl bioactivity database: an update. Nucleic acids research, 42(D1):D1083–D1090, 2014.
  93. Mychembl: a virtual platform for distributing cheminformatics tools and open data. Challenges, 5(2):334–337, 2014.
  94. Machine-learning-based interatomic potentials for group iib to via semiconductors: A comparative study of universal and independent models. arXiv preprint arXiv:2311.11305, 2023.
  95. Phase diagram of a deep potential water model. Physical review letters, 126(23):236001, 2021.
  96. A generalizable machine learning potential of ag–au nanoalloys and its application to surface reconstruction, segregation and diffusion. Modelling and Simulation in Materials Science and Engineering, 30(2):025003, 2021.
  97. Virtual exploration of the small-molecule chemical universe below 160 daltons. Angewandte Chemie International Edition, 44(10):1504–1508, 2005.
  98. Virtual exploration of the chemical universe up to 11 atoms of c, n, o, f: assembly of 26.4 million structures (110.9 million stereoisomers) and analysis for new ring systems, stereochemistry, physicochemical properties, compound classes, and drug discovery. Journal of chemical information and modeling, 47(2):342–353, 2007.
  99. 970 million druglike small molecules for virtual screening in the chemical universe database gdb-13. Journal of the American Chemical Society, 131(25):8732–8733, 2009.
  100. Transition1x-a dataset for building generalizable reactive machine learning potentials. Scientific Data, 9(1):779, 2022.
  101. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(11):2864–2875, 2012.
  102. Modeling the high-pressure solid and liquid phases of tin from deep potentials with ab initio accuracy. Physical Review Materials, 7(5):053603, 2023.
  103. Specialising neural network potentials for accurate properties and application to the mechanical response of titanium. npj Computational Materials, 7(1):206, 2021.
  104. Classical and machine learning interatomic potentials for bcc vanadium. Physical Review Materials, 6(11):113603, 2022.
  105. A tungsten deep neural-network potential for simulating mechanical property degradation under fusion service environment. Nuclear Fusion, 2022.
  106. Exploring the chemical space of linear alkane pyrolysis via deep potential generator. Energy & fuels, 35(1):762–769, 2020.
  107. Deep learning of accurate force field of ferroelectric hfo 2. Physical Review B, 103(2):024108, 2021.
  108. Accurate force field of two-dimensional ferroelectrics from deep learning. Physical Review B, 104(17):174107, 2021.
  109. A deep potential model with long-range electrostatic interactions. The Journal of Chemical Physics, 156(12), 2022.
  110. Modeling liquid water by climbing up jacob’s ladder in density functional theory facilitated by using deep neural network potentials. The Journal of Physical Chemistry B, 125(41):11444–11456, 2021.
Citations (12)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now