Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Distribution Learning for Molecular Regression (2407.20475v1)

Published 30 Jul 2024 in cs.LG and q-bio.QM

Abstract: Using "soft" targets to improve model performance has been shown to be effective in classification settings, but the usage of soft targets for regression is a much less studied topic in machine learning. The existing literature on the usage of soft targets for regression fails to properly assess the method's limitations, and empirical evaluation is quite limited. In this work, we assess the strengths and drawbacks of existing methods when applied to molecular property regression tasks. Our assessment outlines key biases present in existing methods and proposes methods to address them, evaluated through careful ablation studies. We leverage these insights to propose Distributional Mixture of Experts (DMoE): A model-independent, and data-independent method for regression which trains a model to predict probability distributions of its targets. Our proposed loss function combines the cross entropy between predicted and target distributions and the L1 distance between their expected values to produce a loss function that is robust to the outlined biases. We evaluate the performance of DMoE on different molecular property prediction datasets -- Open Catalyst (OC20), MD17, and QM9 -- across different backbone model architectures -- SchNet, GemNet, and Graphormer. Our results demonstrate that the proposed method is a promising alternative to classical regression for molecular property prediction tasks, showing improvements over baselines on all datasets and architectures.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. arXiv preprint arXiv:2101.03164, 2021.
  2. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02905, 2021.
  3. Graphnorm: A principled approach to accelerating graph neural network training, 2020. URL https://arxiv.org/abs/2009.03294.
  4. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 10.1021/acscatal.0c04525.
  5. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.
  6. Graph convolutional neural networks with node transition probability-based message passing and DropNode regularization. Expert Systems with Applications, 174:114711, jul 2021. doi: 10.1016/j.eswa.2021.114711. URL https://doi.org/10.1016%2Fj.eswa.2021.114711.
  7. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In NeurIPS, 2015.
  8. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.14115, 2020a.
  9. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020b.
  10. GemNet: Universal directional graph neural networks for molecules. arXiv preprint arXiv:2106.08903, 2021.
  11. How do graph networks generalize to large and diverse molecular systems? arXiv preprint arXiv:2204.02782, 2022.
  12. Neural Message Passing for Quantum Chemistry. In ICML, 2017.
  13. Simple gnn regularisation for 3d molecular property prediction and beyond. In ICLR, 2022.
  14. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225–1234. PMLR, 2016.
  15. Improving regression performance with distributional losses. In International Conference on Machine Learning, pages 2157–2166. PMLR, 2018.
  16. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pages 2796–2804. PMLR, 2018.
  17. Deepergcn: All you need to train deeper gcns, 2020. URL https://arxiv.org/abs/2006.07739.
  18. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021.
  19. Regularizing neural networks by penalizing confident output distributions. ArXiv, abs/1701.06548, 2017.
  20. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
  21. Dropedge: Towards deep graph convolutional networks on node classification, 2019. URL https://arxiv.org/abs/1907.10903.
  22. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, 52(11):2864–2875, 2012. doi: 10.1021/ci300415d. URL https://doi.org/10.1021/ci300415d. PMID: 23088335.
  23. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
  24. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377–9388. PMLR, 2021.
  25. Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint arXiv:2203.04810, 2022.
  26. Rotation invariant graph neural networks using spin convolutions. arXiv preprint arXiv:2106.09575, 2021.
  27. Towards training billion parameter graph neural networks for atomic simulations. arXiv preprint arXiv:2203.09697, 2022.
  28. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  29. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018.
  30. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34, 2021.
  31. Molecular mechanics-driven graph neural network with multiplex graph for molecular structures. arXiv preprint arXiv:2011.07457, 2020.
  32. Pairnorm: Tackling oversmoothing in gnns, 2019. URL https://arxiv.org/abs/1909.12223.
  33. Towards deeper graph neural networks with differentiable group normalization, 2020. URL https://arxiv.org/abs/2006.06972.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now